Overall Process Pipeline
- download the raw meteorology data
- automate the downloading process
- creating master meteorological data file
- preprocessing raw meteorology data for eddypro
- timestamp checks
- adding missing timestamps
- value missing checks
- unit checks
- soil heat flux calculation
- absolute humidity calculation
- shortwave radiation calculation
- test for 1 week data
- test for full data
- test for new data (3 month timeframe)
- precipitation data calculation (pending as data not available/shared)
- preprocessing raw meteorology data for eddypro
- formatting the raw met data
- create the dynamic metadata
- automate eddyPro launching in the pipeline
- check the output from eddypro
- use the eddyPro output in pyFluxPro (L1)
Automation File Structure
- downloaded raw meteorology data
- /raw_metdata/
- created raw meteorology data
- formatted row meteorology data (input for eddypro)
- output from eddypro
Creating a Master Meteorological Data File
- need to obtain the data file (check permission for accessing the data, 2021-08-10 meeting)
- connecting to , file-server.igb.illinois.edu
- acquiring ofTools excel tool
Processing Raw Flux Data in EddyPro
- had a demo in 2021-08-10 meeting
Formatting Meteorological Data for Eddypro
- this could be automated by using python
- NCSA will work on coding and will ask verification of the output to Carl's group
Assembling Dynamic Metadata
Running Eddy Pro
Common Errors and Issues
- can eddy pro run without gui? (headless)
- if not, does it have project file or something so it can be modified automatically?
PyFluxPro: QA/QC, Gap Filling, and Partitioning
Installing
Creating the Database
first demo at Sep 15 2021