This page provides information on GLTG parsers' status and properties, as well as, discussions related to optimizing development.

 

Discussions

  • Parser Classifications
    • For looping through data, there are (at least) two categories that require different structures
      • Catagories
        • One site at a time - frequently this is possible for sources where the data is pulled from an api
          • In this case, the site names are known in advance.
        • Data with multiple sites - tends to be downloaded data in text files of various types.
          • In this case, the site names are read from the file
      • Does it make more sense to first separate mixed sources and use the one-site-at-a-time structure?
  • Array sizes
    • For both API and file parsing, memory use need to be controlled
      • As off july 2016, the usgs parser is the only source that required parsing by a time increment (adds a lot of checking and date management).
  • Parameters
    • How to select needed measurements
      • 7-25-16 have been blocking unneeded values, which are stored in a list in config for the particular parser
        • want to switch to listing all needing values from source
      • Think the best way is to sort using original names (keys) and then map to geodashboard parameter id format

Parser Details

SourceRow Last Updated onSource

Stations

Stations Status on Production Server

Name of parameters from source

Problem SensorsQuestions/Issues
ALL PARSING      
  • 160823 - caching during large parsing causes posting to clowder to fail
    • disable parsing in clowder/custon.conf is the current solution
USGS160921 

03339000, 03343820, 03381495, 05446500, 05447500

  • 07381600

    • parsing works

    • discharge filling works

'00060', '99133', '00095', '00400', '00010', '00300', '00301','63680','99133', '62361', '32295'

  • 161108
    • 07056515
      • no data showing
      • fixed
        • issue was that discharge(_cd) was not being mapped properly - the length of the usgs property names 4263_00060 4263_00060_cd
    • 05432927
      • no data showing
      • fixed - same as above
    • 07055814
      • like above
    • 07144100, 07
      • data not showing in detail view
    • 05599490
      • data not showing in detail view
  • 170113
    • 05480603
      • usgs data only until 141012
    • 05480968
      • usgs data only until 131103
    • 05432927
      • usgs data only until 161013
  • when gap filling, starts searching for gaps since last gap_fill datapoint. Takes time if last gap was early in dataset
  • 161103

    • Traceback (most recent call last):

        File "usgs_calc_load.py", line 222, in <module>

        main(site)

        File "usgs_calc_load.py", line 91, in main

          print "stream_json_calc_load['end_time'][:10] = " + str(stream_json_calc_load["end_time"][:10])

      TypeError: 'NoneType' object has no attribute '__getitem__'

      Traceback (most recent call last):

        File "usgs_calc_cumul_load.py", line 197, in <module>

          main(site)

        File "usgs_calc_cumul_load.py", line 94, in main

          streams_start_time = stream_json_load["start_time"][:10]

GREON160830 GREON-01,02,03,04,05,06,07
  • 01 data until 160615
  • 02 data until 160629
  • 03 up to date
  • 04 up to date
  • 05 up to date
  • 06 data until 151021
  • 07 up to date
 
  • 161103 - Looks like parameters missing before july 2016
    • GREON-03
      • Nitrate as N
      • Water Temperature
    • GREON-05
  • 161103
    • coordinates where flip-flopped at some point
      • reparsed both
  • 160811 - having issues parsing GREON-06 on dev
    • have noticed some discrepancies in number of column headers and data fields - requires investigation
    • 1610 - refactored greon parser so that each row of data has it's own header row (previously just used row with all parameters for all rows of data)

 

Water Quality Portal       
NOAA       
LTMR      
  • Still runs all sensors in one script 160822
EPA Pollutant Loading       
Tennessee  around 3900 sensors   
  • Total of data causes front end to be very slow
    • smaller set of 417 sensors seems to run ok
  • did TIMESTAMP slip through?
Fox River       
Illinois EPA 

IEPA Ambient Water Quality Monitoring Network (AWQMN)

Acquired from STORET

AK-02, D-32, DA-06, DS-07, E-26
  • 20161107
    • AK-02, D-32, DA-06, DS-07, E-26, o-02, O-07, O-08, O-10, O-11, O-20, O-30, O-31
  • 20161108
    • DS-06, DS-07, DT-01, DT-06, DT-09, DT-22, DT-35, DT-38, DTD-02, DTG-02, DTK-04, E-06, E-18, E-29, H-01, HBD-04, HCC-07, HCCC-02, O-02, O-07, O-08, O-10, O-11, O-20, O-30, O-31, OC-04, OD-06, OD-07, OI-07, OI-08, OJ-08, OL-02, ON-01, OQ-01
 
  • Column Headers
    • Characteristic Name:

      • Kjeldahl nitrogen

      • Phosphorus

      • Ammonia-nitrogen

    • Sample Fraction
      • Total
      • Dissolved
 Add measurement threshold data?

API Parser Structure

 

File Parser Structure

 

 

 

 

  • No labels