Automation Procedures

Overall Process Pipeline

• download the raw meteorology data
  • automate the downloading process
• creating master meteorological data file
  • preprocessing raw meteorology data for eddypro
    • timestamp checks
    • adding missing timestamps
    • value missing checks
    • unit checks
    • soil heat flux calculation
    • absolute humidity calculation
    • shortwave radiation calculation
    • test for 1 week data
    • test for full data
    • precipitation data calculation (pending as data not available/shared)
• formatting the raw met data
• create the dynamic metadata
• automate eddyPro launching in the pipeline
  • check the output from eddypro
• use the eddyPro output in pyFluxPro (L1)

Automation File Structure

• downloaded raw meteorology data
• created raw meteorology data
• formatted row meteorology data (input for eddypro)
• output from eddypro

Creating a Master Meteorological Data File 

• need to obtain the data file (check permission for accessing the data, 2021-08-10 meeting)  
• connecting to , file-server.igb.illinois.edu
• acquiring ofTools excel tool

Processing Raw Flux Data in EddyPro

• had a demo in 2021-08-10 meeting

Formatting Meteorological Data for Eddypro

• this could be automated by using python 
• NCSA will work on coding and will ask verification of the output to Carl's group 

Assembling Dynamic Metadata

Running Eddy Pro

Common Errors and Issues

• can eddy pro run without gui? (headless)
• if not, does it have project file or something so it can be modified automatically?

PyFluxPro: QA/QC, Gap Filling, and Partitioning

Installing

Creating the Database

first demo at Sep 15 2021

Formatting the PyFluxPro Control Files

Primary

Secondary

Other

Running a Flux Database through PyFluxPro

L1-L3: Processing Single Years

L4-L6: Processing a Full, or Muti-Year, Dataset (Incomplete)