Overall Process Pipeline
- download the raw meteorology data
- automate the downloading process
- creating master meteorological data file
- preprocessing raw meteorology data for eddypro
- timestamp checks
- adding missing timestamps
- value missing checks
- unit checks
- soil heat flux calculation
- absolute humidity calculation
- shortwave radiation calculation
- test for 1 week data
- test for full data (2020)
- test for new data (2021, 3 month timeframe)
- precipitation data calculation - change units and timestamp
- need some more testing for precip data
- precip data QA/QC check.
- preprocessing raw meteorology data for eddypro
- formatting the raw met data
- copy and rename master met data file
- get required columns and corresponding labels
- choose correct fields for soil temp, shf, air temp and moisture
- shf done
- air temp done
- soil temp field need confirmation on code changes done
- moisture field changes with crop type/filename - Bethany working on table to match keys
- wind speed and wind direction needs clarification - these fields change with year
- get precipitation data
- need to do precip data for creating met file section first
- check for duplicate naming of labels (Eg: _1 _1 _2 )
- choose correct fields for soil temp, shf, air temp and moisture
- get correct metrics
- insert metrics as first row
- timestamp format check
- convert temp fields from Celsius to kelvin
- convert numeric fields to numerical values
- check for non-numeric values and change to -9999.0
- create function to check columns required for eddypro
- check if conversion from string to integer is needed in format.py.
- create the dynamic metadata
- dynamic metadata shared
- automate eddyPro launching in the pipeline
- check the output from eddypro
- use the eddyPro output in pyFluxPro (L1)
- use eddypro output in pyfluxpro L2 to generate graphs
- formatting the EddyPro output to AmeriFlux variables
- L1
- L2
- automation of running pypluxpro L1 and L2 process
Automation File Structure
- downloaded raw meteorology data
- /raw_metdata/
- created raw meteorology data
- formatted row meteorology data (input for eddypro)
- output from eddypro
Creating a Master Meteorological Data File
- need to obtain the data file (check permission for accessing the data, 2021-08-10 meeting)
- connecting to , file-server.igb.illinois.edu
- acquiring ofTools excel tool
Processing Raw Flux Data in EddyPro
- had a demo in 2021-08-10 meeting
Formatting Meteorological Data for Eddypro
- this could be automated by using python
- NCSA will work on coding and will ask verification of the output to Carl's group
Assembling Dynamic Metadata
Running Eddy Pro
Common Errors and Issues
- can eddy pro run without gui? (headless)
- if not, does it have project file or something so it can be modified automatically?
PyFluxPro: QA/QC, Gap Filling, and Partitioning
Installing
Creating the Database
first demo at Sep 15 2021