Priority List on 2022-02-25

Yong Wook Kim 

Minu Mathew 

Rob Kooper 

Priority Task list  :

1. Add missing variables in L1 formatting
   1. we will only use the ameriflux related one
   2. amerfilux only L1 and mainstem L1, combine these 
   3. manually change variables names to run pyfluxpro L1
      1. there are three variables
      2. maybe we can automate this by creating the changed variables first than convert it later using another mapping
      3. maybe the above process is less priority
2. Make L1 formatting more efficient
   1. there are several L1 files
3. Erroring variables : {AmeriFlux variable name -> PyFluxPro-friendly variable name} mapping below
   LE->Fe
   LE_SSITC_TEST->Fe_EP_QC
   LW_IN->Fld
   LW_OUT->Flu
4. L2 formatting for AmeriFlux 
5. minor formatting issues 
   1. pyfluxpro version changes
   2. erroring variables name to plot as well
   3. empty data frames because of the deletion (no data, empty line)
6. Excel Formatting for the AmeriFlux 
7. Run whole pipeline for a certain site probably using whole year
8. Run PyFluxPro and others to generated the final output for AmeriFlux
9. How do we handle VWC, VWC1, automatically.
10. Submit output to AmeriFlux site (has been submitted, waiting for the result)
11. Separate each steps and manually run the steps based on the user's need
12. Check L1.txt variable names if exist in full_output and met_data_30 (less priority)

    1. Case : Checking if Variables in L1.txt actually exists in full_output and Met_data_30 — input excel sheet to pyfluxpro.

       variable name XYZ in L1_ameriflux. but this is not there in full-output and met-data-30.. pyfluxpro throws a warning. it runs. but we will catch this when submitting to ameriflux. 

13. Create unit test
14. L2 checks for Variables section : https://github.com/OzFlux/PyFluxPro/wiki/Level-2#the-variables-section
    1. Currently we check for DependencyCheck, ExcludeDates and RangeCheck.
       1. PyFluxPro supports other checks as well, such as, DiurnalCheck, ExcludeHours, CorrectWindDirection, LinearCheck. But these are also not used by CABBI.
       2. We create L2 from the template L2 given by Cabbi. Since Cabiii didnt give L2 with the other extra checks, our code does not include these extra sections.
       3. But If you give L2.txt with these extra sections added manually, pyfluxpro will handle it. 

Low Priority

1. Data validation. 
   1. user input validation
   2. data input validation
2. QA / QC 
3. Logging
4. Code clean up
   1. check string comparison in whole code
5. L2   variables section can have as multiple checks . For Example : RangeCheck, DependencyCheck, ExcludeDates, DiurnalCheck, ExcludeHours, CorrectWindDirection, LinearCheck etc. The actual number of checks will depend on the pyfluxpro version.
   1.  Currently we check for only RangeCheck, DependencyCheck, ExcludeDates. The other checks are not supported in our code.
   2. The problem with checking for these many checks are :
      1. Variables may or may not have these checks : meaning a variable can have 0, 1,2 ,3 , 10 types of checks if necessary. No way to know ahead what checks are present
      2. These checks can appear in any order
      3. We do not know what is the ending index of each checks. 
   3. validation for variables
   4. check in any order
   5. indexing by sorting or something
   6. it d
6. read/write module changes in utils
7. Google form? (It is for them to make some instant changes on their setting like sensors and others. Then pipeline automatically grab the information from the form and apply necessary changes)