pyGeodashboard

Development of pyGeodashboard started on 2016-02-04.  It is a library that contains the basic functions needed for parsing sensors, streams, and datapoints to the geostreaming API.

• Step 1: Outline Parser Functions
  • Create a outline that describes the process of parsing with focus on separating reusable portions of a parser from those that are particular to a specific data source.

Outline Parser Functions

Functions will be described as unique= code particular to the source is needed and general= should be able to run as part of every source

1. Get data from source (unique)
   1. Parsing begins by getting the data from the source.  Two types of data are needed:
      1. data that describes the site such as geocodes, name,  and source. 
      2. measurements
   2. The format and retrieval method varies from source to source
      1. Some source formats
         1. API for a single station (USGS,NOAA)
         2. API for mixed stations (Water Quality Portal)
         3. Files stored to server with loggetnet (GREON)
         4. csv download (LRTM)
2. Parse data to sensor (unique and general)
   1. Up till now, this has been a unique process for each source; however, this portion should be broken into 2 portions
      1. reformat data into a standard that can be input into general parser (unique)
      2. parse data to sensor json (general)
3. Parse data to stream(s) (unique abd general)
   1. Similar to parse to sensor, with the main difference being that sources can have multiple stream for different reasons.  
      1. For example:
         1. GREON uses 2 streams -  one for water quality data and one for environmental data
         2. USGS uses 5 streams -  water quality measurements, gap filled nitrate, gap filled discharge, load, and cumulative load
      2. two different conventions have been used, and need to be standardized
         1. GREON names the streams differently: GREON-07_MD or GREON-07_WQ
         2. USGS puts a data_type key in properties with possible values: source_data, fill_nitrate, fill_discharge, calc_load, and calc_cumul_load
            1. Probably should be discussed and decided.
   2. Currently, each source has it's own implementation for all, like sensors, it should be broken into unique and general portions
      1. get stream data from source including determining number of streams needed and parsing data to standard format (unique)
      2. parse data to stream json (general)
4. Parse data to datapoints