...
These should be migrated to https://opensource.ncsa.illinois.edu/bitbucket/projects/GEOD/repos/pygeotemporal-parsers/browse on update
Legacy Data Files
https://uofi.app.box.com/folder/131386931931
Overview of GLTG parsers
- nebula instance: gltg-parsers2.ncsa.illinois.edu
- user = parsers (no password)
- root directory = /home/parsers
- directory structure
- 4 directories for 4 systems
- 3 parsers for 3 sources for each system
- 4 directories for 4 systems
- run parsers
- each system has a shell script that runs all three sources sequentially
- for each source all data is parsed first, then a subprocess runs the binning with a wait on subprocess until finished. When done the next source parser starts. No timeout between source parsers.
- cronjobs
- get greon data from gltg.ncsa.illinois.edu:/var/opt/CampbellSci/loggernet_ordered
- runs in as marcuss user (this can be changed but for now it is due to permissions on loggernet on gltg)
/home/marcuss/get_greon_data.sh
uses rsync to pull data to /home/marcuss/data/greon/ then copies to/home/parsers/greon-data/
- parsers
- 4 lines for 4 systems
- get greon data from gltg.ncsa.illinois.edu:/var/opt/CampbellSci/loggernet_ordered
...
- gltg-dev
- resources
- nebula, proxy 2cpu 4ram
- nebula, postgres (4 CPU, 8G RAM)
- times by source
- greon .5h
- iwqis 1husgs 1h 24m failed presumably do to http timeout
- usgs
- resources
- gltg
- resources
- sd stack, proxy 4cpu 6ram
- nebula, postgres (4 CPU, 8G RAM)
- times by source
- greon 6m
- iwqis 40m
- usgs 3h 10m
- however, geostreams not responding for long time after parsers ended
- resources
- ilnlrs
- setup
- 10m sleep between greon and iwqis, 20m sleep between iwqis and usgs
- binning
- 2 workers (not 4)
- USGS uses 60s sleep after a bin finishes
- times
- greon 25m (can be improved
- iwqis 40m
- usgs 54m
- setup
GLM Parsing Time
GLM Zooplankton/Phytoplankton ingestion timing for Production server (141.142.211.239):
...