Attendees
Agenda
- Automation of GHG file from the server
- reading multiple datalogger files from the server and unifying them into one raw met file that includes the entire timeline of interest
- ghg files, meteological tower file (.dat)
- ghg file: daily.
- downloaded it based on the daily bases
- time zone?
- met tower file: grabbing the certain time certain time of the day
- the new line will be added every 30 min
- one file until the changes in variable that is recorded.
- PyFluxPro demo
- PyFluxPro data formatting - should the timestamps start with yyyy-mm-dd 00:00 and end with yyyy-mm-dd 23:00? It depends on the time period selected for eddypro run.
- Not a requirement.
- PyFluxPro installation queries. Done.
Differences in 2010 raw met data:
- Field name : CPU:EBI_Maize_EC.CR3 -- is there a pattern?
- Units : W/meter≤ -- can we make a list of provided units? Or {column name:unit} mapping?
- NAN and INF in file
- Column names : SoilTempCA50_Avg -- is this in soil key?
- Dont do this. No need to put 2010 data in pipeline.
- Findings in comparing the outputs of EddyPro :
- Exactly the same
- Finding in comparing the inputs to EddyPro:
- Automated has 83 columns and manual has 38 columns.
- There are 37 common columns. SHF_1_1_2 is not present in the automated one, but present in manual.
- Common cols :
['SWC_2_2_1', 'Ts_1_5_1', 'Ts_1_3_1', 'Ts_2_1_1','SWC_1_2_1','PPFDr','RH','Ts_2_2_1','SWC_1_5_1','SWC_1_4_1','SWC_1_3_1','SWC_1_1_1',
'Ts_1_4_1','Ta_1_1_2','Ts_1_1_1','SWC_2_1_1','LWout','Rr','SWC_2_5_1','Tc','Ts_1_2_1','WD','TIMESTAMP','SWin','SWC_2_3_1','SWC_2_4_1',
'LWin','PPFD','SWout','Ts_2_5_1','Ta_1_1_1','Ts_2_3_1','SHF_1_1_1','Ts_2_4_1','P_rain','Rn','MWS'] - Required cols are present (SWin, RH, LWin, PPFD). Locators (_1_1) are present.
- SHF_1_1_2 is the only col that is not present in automated. Need to figure out why.
- SHF_1_1_2 is a mistake from the manual entry. Shoudl have been SHF_2_1_1.
- Common cols :
- TIMESTAMP format in automated is with hyphen. TIMESTAMP format in manual is with /
- Columns that are not-common to both dataframes :
{'Ah_fromRH','CDMBattV_Min','CDMPTempC_Avg(1)','CDMPTempC_Avg(2)','CDMPTempC_Avg(3)','CDMPTempC_Avg(4)','LWDn_Avg','LWUp_Avg', 'MoistureA_Avg','MoistureC_Avg','PTemp_Avg','RECORD','Rl_net_Avg','Rs_net_Avg','SBTempC_Avg','SBTempK_Avg','SHF_1_1_2','SHF_2_1_1', 'SW_out_Avg','SoilT_C_Avg','SoilTempA_Avg','SoilTempC_Avg','TC_100cm_Avg','TC_10cm_Avg','TC_20cm_Avg','TC_30cm_Avg','TC_40cm_Avg','TC_50cm_Avg','TC_5cm_Avg', 'TC_60cm_Avg','TC_75cm_Avg','TargTempC_Avg','TargmV_Avg','U_Avg','VWC_100cm_Avg','VWC_10cm_Avg','VWC_20cm_Avg','VWC_30cm_Avg','VWC_40cm_Avg','VWC_50cm_Avg', 'VWC_5cm_Avg','VWC_60cm_Avg','VWC_75cm_Avg','V_Avg','batt_volt_Min','cnr4_T_C_Avg','cnr4_T_K_Avg'}
Ignore SHF_1_1_2 and SHF_2_1_1 - it is a mistake in the manual entry.
EddyPro ignores the extra columns (the "not common" columns).
Can delete the processed columns, TC_100 SoilTC, SW_out etc. Can delete everything in this list.
e. Comparing values of both inputs (common cols only) cell by cell.
i. False indicates difference in values. True indicates same value. The attached file (compare_bool_result) gives the count of False and True for each cell.
iii. P_rain has False for 1467 times and True for 22 times -- need to check the values.
iv. Rr does not have any False values.
v. All SWC has 1488 False values and only 1 True value. -- because the manual variable naming convention is wrong. Can cross check this with Soil key EddyPro new moisture variables and EddyPro old moisture variables. Not critical.
vi. Same for Ts, False values are in 1480s and Trues are under 10. -- same prob as above
Send Bethany the P_rain values from automated file. And the manual one Precip_IWS.
Action items
- Send Bethany the P_rain values from automated file. And the manual one Precip_IWS. – Check for P_rain
- Check for SWC and Ts naming convention with soil keys . Not critical.