Priority Task list :
Add missing variables in L1 formattingwe will only use the ameriflux related oneamerfilux only L1 and mainstem L1, combine thesemanually change variables names to run pyfluxpro L1there are three variablesmaybe we can automate this by creating the changed variables first than convert it later using another mappingmaybe the above process is less priority
Make L1 formatting more efficientthere are several L1 files
Erroring variables : {AmeriFlux variable name -> PyFluxPro-friendly variable name} mapping belowLE->FeLE_SSITC_TEST->Fe_EP_QCLW_IN->FldLW_OUT->FluL2 formatting for AmeriFluxminor formatting issuespyfluxpro version changeserroring variables name to plot as wellempty data frames because of the deletion (no data, empty line)
Excel Formatting for the AmeriFluxRun whole pipeline for a certain site probably using whole yearRun PyFluxPro and others to generated the final output for AmeriFluxHow do we handle VWC, VWC1, automatically.- Submit output to AmeriFlux site (has been submitted, waiting for the result)
- Separate each steps and manually run the steps based on the user's need
- Check L1.txt variable names if exist in full_output and met_data_30 (less priority)
Case : Checking if Variables in L1.txt actually exists in full_output and Met_data_30 — input excel sheet to pyfluxpro.
variable name XYZ in L1_ameriflux. but this is not there in full-output and met-data-30.. pyfluxpro throws a warning. it runs. but we will catch this when submitting to ameriflux.
Create unit testL2 checks for Variables section : https://github.com/OzFlux/PyFluxPro/wiki/Level-2#the-variables-section- Currently we check for DependencyCheck, ExcludeDates and RangeCheck.
- PyFluxPro supports other checks as well, such as, DiurnalCheck, ExcludeHours, CorrectWindDirection, LinearCheck. But these are also not used by CABBI.
- We create L2 from the template L2 given by Cabbi. Since Cabiii didnt give L2 with the other extra checks, our code does not include these extra sections.
- But If you give L2.txt with these extra sections added manually, pyfluxpro will handle it.
- Currently we check for DependencyCheck, ExcludeDates and RangeCheck.
Low Priority
Data validation.- user input validation
- data input validation
QA / QCLogging- Code clean up
check string comparison in whole code
L2 variables section can have as multiple checks . For Example : RangeCheck, DependencyCheck, ExcludeDates, DiurnalCheck, ExcludeHours, CorrectWindDirection, LinearCheck etc. The actual number of checks will depend on the pyfluxpro version.- Currently we check for only RangeCheck, DependencyCheck, ExcludeDates. The other checks are not supported in our code.
- The problem with checking for these many checks are :
- Variables may or may not have these checks : meaning a variable can have 0, 1,2 ,3 , 10 types of checks if necessary. No way to know ahead what checks are present
- These checks can appear in any order
- We do not know what is the ending index of each checks.
- validation for variables
- check in any order
- indexing by sorting or something
- it d
read/write module changes in utils- Google form? (It is for them to make some instant changes on their setting like sensors and others. Then pipeline automatically grab the information from the form and apply necessary changes)