Testing gradual traversal of entire CI-BER collection (100mil files). Tuning the processing work flows such that they do not fail or overwhelm any of the services involved.

Starting trying this last week. So far errors have related to extreme text file sizes. I expect other errors related to special case files. I also expect certain services in the work flow to fall over due to load, so these limitations will be discovered.