Overall Process Pipeline

  • download the raw meteorology data
    • automate the downloading process
  • creating master meteorological data file
    • preprocessing raw meteorology data for eddypro
      • timestamp checks
      • adding missing timestamps
      • value missing checks
      • unit checks
      • soil heat flux calculation
      • absolute humidity calculation
      • shortwave radiation calculation
      • test for 1 week data
      • test for full data (2020)
      • test for new data (2021, 3 month timeframe)
      • precipitation data calculation - change units and timestamp
        • need some more testing for precip data
      • precip data QA/QC check.
  • formatting the raw met data
    • copy and rename master met data file
    • get required columns and corresponding labels
      • choose correct fields for soil temp, shf, air temp and moisture
        • shf done
        • air temp done 
        • soil temp field need confirmation on code changes done
        • moisture field changes with crop type/filename - Bethany working on table to match keys
        • wind speed and wind direction needs clarification - these fields change with year
      • get precipitation data 
        • need to do precip data for creating met file section first
      • check for duplicate naming of labels (Eg: _1 _1 _2 )
    • get correct metrics
    • insert metrics as first row
    • timestamp format check
    • convert temp fields from Celsius to kelvin
    • convert numeric fields to numerical values
    • check for non-numeric values and change to -9999.0
    • create function to check columns required for eddypro
    • check if conversion from string to integer is needed in format.py.
  • create the dynamic metadata
    • dynamic metadata shared
  • automate eddyPro launching in the pipeline
    • check the output from eddypro
  • use the eddyPro output in pyFluxPro (L1)
  • use eddypro output in pyfluxpro L2 to generate graphs
  • formatting the EddyPro output to AmeriFlux variables
    • L1
    • L2
  • automation of running pypluxpro L1 and L2 process


Automation File Structure

  • downloaded raw meteorology data
    • /raw_metdata/
  • created raw meteorology data
  • formatted row meteorology data (input for eddypro)
  • output from eddypro

Creating a Master Meteorological Data File 

  • need to obtain the data file (check permission for accessing the data, 2021-08-10 meeting)  
  • connecting to , file-server.igb.illinois.edu
  • acquiring ofTools excel tool

Processing Raw Flux Data in EddyPro

  • had a demo in 2021-08-10 meeting

Formatting Meteorological Data for Eddypro

  • this could be automated by using python 
  • NCSA will work on coding and will ask verification of the output to Carl's group 

Assembling Dynamic Metadata

Running Eddy Pro

Common Errors and Issues

  • can eddy pro run without gui? (headless)
  • if not, does it have project file or something so it can be modified automatically?

PyFluxPro: QA/QC, Gap Filling, and Partitioning

Installing

Creating the Database

first demo at Sep 15 2021

Formatting the PyFluxPro Control Files

Primary

Secondary

Other

Running a Flux Database through PyFluxPro

L1-L3: Processing Single Years

L4-L6: Processing a Full, or Muti-Year, Dataset (Incomplete)


  • No labels