...
Here the each source parses the data then waits for the subprocess that bins the data until finished. When the binning finishes, the the next source parser starts. No timeout between parsers.
- gltg-dev
- resources
- nebula, proxy 2cpu 4ram
- nebula, postgres (4 CPU, 8G RAM)
- times by source
- greon .5h
- iwqis 1h
- usgs ~1.5h (estimate as USGS is having some latency issues) 1h 24m failed presumably do to http timeout
- No errors0
- resources
- gltg
- resources
- sd stack, proxy 4cpu 6ram
- nebula, postgres (4 CPU, 8G RAM)
- No errors
- times by source (3 days of data parsed)
- greon 5m6m
- iwqis 45m40m
- usgs 3h 10m
- however, geostreams not responding 3h laterfor long time
- resources
- ilnlrs-dev
- major problem - authentication of cache client timing out
- added a try with 10minute wait if timed out and try again, then another try ans wait 20minutes - still timed out and crashed the parser
- this was with the usgs parser only
- checked ilnlrs-geodashboard-dev - high cpu usage up to %198 long after parser crashed
- major problem - authentication of cache client timing out
GLM Parsing Time
GLM Zooplankton/Phytoplankton ingestion timing for Production server (141.142.211.239):
- Update Statistics: About an hour for all sensors. (api/sensors/update/).
- Binning by season
- Ran the following endpoint for all 3 parameters simultaneously:
- /api/cache/season/parameter-name
- Took about 9 hours for all to complete.
- Ran the following endpoint for all 3 parameters simultaneously:
- Number of Datapoints:
- Zooplankton-biomass: 3004
- Zooplankton-biovolume: 3004
- Phytoplankton-biovolume: 1930