Overall Process Pipeline

• download the raw meteorology data
  • automate the downloading process
• creating master meteorological data file
  • preprocessing raw meteorology data for eddypro
    • timestamp checks
    • adding missing timestamps
    • value missing checks
    • unit checks
    • soil heat flux calculation
    • absolute humidity calculation
    • shortwave radiation calculation
    • test for 1 week data
    • test for full data (2020)
    • test for new data (2021, 3 month timeframe)
    • precipitation data calculation - change units and timestamp
      • need some more testing for precip data
    • precip data QA/QC check.
• formatting the raw met data
  • copy and rename master met data file
  • get required columns and corresponding labels
    • choose correct fields for soil temp, shf, air temp and moisture
      • shf done
      • air temp done 
      • soil temp field need confirmation on code changes done
      • moisture field changes with crop type/filename - Bethany working on table to match keys
      • wind speed and wind direction needs clarification - these fields change with year
    • get precipitation data 
      • need to do precip data for creating met file section first
    • check for duplicate naming of labels (Eg: _1 _1 _2 )
  • get correct metrics
  • insert metrics as first row
  • timestamp format check
  • convert temp fields from Celsius to kelvin
  • convert numeric fields to numerical values
  • check for non-numeric values and change to -9999.0
  • create function to check columns required for eddypro
  • check if conversion from string to integer is needed in format.py.
• create the dynamic metadata
  • dynamic metadata shared
• automate eddyPro launching in the pipeline
  • check the output from eddypro
• use the eddyPro output in pyFluxPro (L1)
• use eddypro output in pyfluxpro L2 to generate graphs
• formatting the EddyPro output to AmeriFlux variables
  • L1
  • L2
• automation of running pypluxpro L1 and L2 process

Automation File Structure

• downloaded raw meteorology data
  • /raw_metdata/
• created raw meteorology data
• formatted row meteorology data (input for eddypro)
• output from eddypro

Creating a Master Meteorological Data File 

• need to obtain the data file (check permission for accessing the data, 2021-08-10 meeting)  
• connecting to , file-server.igb.illinois.edu
• acquiring ofTools excel tool

Processing Raw Flux Data in EddyPro

• had a demo in 2021-08-10 meeting

Formatting Meteorological Data for Eddypro

• this could be automated by using python 
• NCSA will work on coding and will ask verification of the output to Carl's group 

Assembling Dynamic Metadata

Running Eddy Pro

Common Errors and Issues

• can eddy pro run without gui? (headless)
• if not, does it have project file or something so it can be modified automatically?

PyFluxPro: QA/QC, Gap Filling, and Partitioning

Installing

Creating the Database

first demo at Sep 15 2021

Formatting the PyFluxPro Control Files

Primary

Secondary

Other

Running a Flux Database through PyFluxPro

L1-L3: Processing Single Years

L4-L6: Processing a Full, or Muti-Year, Dataset (Incomplete)