Versions Compared

Old Version 8

changes.mady.by.user Marcus Slavenas

Saved on

compared with

New Version Current

changes.mady.by.user Marcus Slavenas

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Attendees

Agenda

  1. Inputs to running model
  2. Inputs to training model


Notes by Shannon Bradley

Have been able to run model - but only in a very specific sense

Fortan Model code discussion:

Inputs to model code - how can we generalize it so we can use any region/location any year or date range

Start with State and Climate Division first and time range

One piece of code that can run on any of those selections

Training Model with R code - Surendra working on this - it will return coefficients

Marcus has broken code up - Nancy's code review (2017)

What files are we opening- week days 2016 csv and that is the cdc weeks - calling that year in specific

  • in the future - remove hard code of 2016 - pass in a year thru the interface - pass in input to choose static files - these can sit and should be able to come from CDC
  • there is a CDC definition - how many days start the year - if there are 4 full days in first week it is week 1 or else it is week 53 of previous
    • instead of creating static file for every year - there is an algorithm?  there is an algorithm we can use or create - do this for more sustainability - yes - just sent algorithm to Marcus to use
  • will people always run model by year? not running 2 years or anything? - yes plan is just 1 year at a time
  • weekly normals - 80s-2010 - baseline to be used until 2021 or 2022 the normals will change - for this project there will be a new 30 year normal - this will stay as is
  • After 2010 pull in next year and current year - we get data file - created by climate center and we move that into this directory - this is the prism data - is written in python now - is just for one day in a chron for daily input - runs daily to get it preupdated
    • just adds a row at the end of the file - if they QA they can be updating for past month - so we rerun whole file
    • should be able to update every day for model
  • Quarterly / Previous year differences - compute weekly data - then compute quarterly data - do for normals - then we do for previous year and then for current year
    • need to find differences from normal for previous seasons or quarters - and weekly differences
  • Current Normals - current year data - 1109 is the daily data that is base data used to compute weekly data for normals / previous year / and current year
    • can run automated - she ran at a specific time every day - what she would do for current week ... week would have to have 7 days - end on last week with 7 days
      • CDC week rather than a rolling week
      • 53 week year - would get left out
      • Run daily - but weekly would only get done at the 7 day marker
      • PRISM may not come in every day
      • Add forecast to PRISM data
      • Need to run every day ... data feeds are not perfect - do on daily basis but calculate weeks on the 7 day CDC format with updated data - NOT ROLLING
      • once or twice a summer a data feed will fail -
      • for previous years - should be the same - only current year can be tricky if you add in forecast data
        • if run thru model - may want to trigger an update at the beginning of the session
      • PRISM - does not make averages from climate division - their calculations make grids
      • other states - will need to make new grids for them
  • ILCD2017-831 - input files - MINT (minimum temp), QPF06 is precip - we get this from forecast center - tagged on to the end of daily data
  • save forecast data every day - just in case there is a glitch - ONLY for current year
  • CD-Daylight - daylight hours - static - pull as is - static year to year
    • would need to be computed for other states? Bill used Navy Observatory site to enter lat long and get number of daylight hours - took centroid to get daylight hours
    • not sure what state we would do next - Iowa? (Iowa, Michigan, Minnesota, Illinois and Wisconsin) (center is UWMadison)  right now Upper midwest
  • Weekly MIR avg - 0512 - got from Surendra - used to calculate the dependent variable - avg MIR - range of years - will this change? how long will this be good
    • Based on observations ...
    • we now have data 2013-2017 - should need to update this???? - should be updated to give best average possible (from IDPH trap data)
      • not available thru API?
      • need to get new MIR for 13-17 .... from somewhere?
    • need to update annually
    • this should be updated before mosquito season each year - dependent coefficents will change each year
    • would want to know MIR for each individual year - so we need to have MIR and coefficient for each year - get MIR and create new coefficient each year
      • R code calculates the coefficients
    • 2005-2012, 2005-2013, 2005-2014, 2005-2015, etc. - average across each week across each climate division - this is what goes into making the map - is the MIR
      • right now we only have 2016 - need to put all years in there
      • Bill sent to marcus - everything GEOcoded to date
  • 2012 - 1 year weekly MIR - used for plotting purposes - for output - can be removed (plotted 2012 because it was most extreme year) - Marcus will pull directly
    • if we ran this year - would need to update to 2017
    • average over all years
    • Calculated from Actual
    • can create every year from MIR Actual
    • can update map from MIR Actual
  • MIR actual - same thing - just a different range of years - CD_ACT - all the way to 2016
    • every year for every week in every climate division
    • Bill took time every week to calculate the averages from actual - app should do this weekly at predetermined time - add one line but update whole year because QA can update inputs from previous weeks
      • Bill has a 1 page - 7 steps - could be converted to an extractor for Clowder - Marcus has those steps 
      • most recent years did not do Dupage
  • CD_Coef - this is just coefficients - will feed params to R code and it will give back this
    • generate once a year?
    • daily running - use same coeff for whole model
    • could have a runtime file that incorporates their local files
    • local file saved could be unique and would re-run the coeff as needed - this will need fleshed out in the future

Iowa may be next - has the best data of the group

Do all states to at least climate division level

Mosquito season starts in April? The start looking in June for a concentrated effort

Never ran model before May 1


Notes by Marcus Slavenas

week_days_2016.csv
use algorthm http://www.angulartutorial.net/2017/09/calculate-epi-week-from-date-convert.html

use 1109n use as is

1109 prism update daily

forecaste ILCD_<year><month><day>
daily file only for current year

CD_Daylight.csv
static for every year
will need to get for other states
Navy observatory site - send Bill climate division shape file and he can pull data

Weekly_MIRavg_05_12.csv (05_12 is range of years of average data)
update before mosquito season
real trap data
use from data as far back as we have to previous year to running model year
get all trap data
calculate from actual

Weekly_MIR_2012.csv
not necesary for model - pull from database and don’t load to model

CD_ACT_mir_2005_13.csv
actual data that users would load
Bill sent set of instructions for creating extractor

CD_Coef_2017.csv 
use static when running on cron
create file when running though app