Page History

...

1. Get data from source (unique)
   1. Parsing begins by getting the data from the source.  Two types of data are needed:
      1. data that describes the site such as geocodes, name,  and source. 
      2. measurements
   2. The format and retrieval method varies from source to source
      1. Some source formats
         1. API for a single station (USGS,NOAA)
         2. API for mixed stations (Water Quality Portal)
         3. Files stored to server with loggetnet (GREON)
         4. csv download (LRTM)
2. Parse data to sensor (unique and general)
   1. Up till now, this has been a unique process for each source; however, this portion should be broken into 2 portions
      1. reformat data into a standard that can be input into general parser (unique)
      2. parse data to sensor json (general)
3. Parse data to stream(s) (unique abd general)
   1. Similar to parse to sensor, with the main difference being that sources can have multiple stream for different reasons.  
      1. For example:
         1. GREON uses 2 streams -  one for water quality data and one for environmental data
         2. USGS uses 5 streams -  water quality measurements, gap filled nitrate, gap filled discharge, load, and cumulative load
      2. two different conventions have been used, and need to be standardized
         1. GREON names the streams differently: GREON-07_MD or GREON-07_WQ
         2. USGS puts a data_type key in properties with possible values: source_data, fill_nitrate, fill_discharge, calc_load, and calc_cumul_load
            1. Probably should be discussed and decided.
   2. Currently, each source has it's own implementation for all, like sensors, it should be broken into unique and general portions
      1. get stream data from source including determining number of streams needed and parsing data to standard format (unique)
      2. parse data to stream json (general)
4. Parse data to datapoints