...
What we should have working (2/26/2018)
- Main deliverables
- https://www.workshop1.nationaldataservice.org/
- Main workbench instance
- https://github.com/datadrivenag
- https://www.workshop1.nationaldataservice.org/
- DNS pointing to 132.249.238.220 with valid TLS for www.workshop1.nationaldataservice.org
- Nagios monitoring http://132.249.238.214/nagios/
- workshop1-node1 - node10 + www.workshop1.nationaldataservice.org
- User accounts
- Scaled cluster
- 1 master + etcd
- 1 NFS server w/ 3TB volume
- 2 GLFS servers
- 10 compute nodes m1.2xlarge w/ 100GB docker volume each
- Customized UI
- Included NIFA/USDA logos
- Link to user guide
- Link to tutorial repo
- Description makes sense to DataDrivenAg users
- Link to contact us if they want to run their own hackathon
- Data available via Geoserver
- See https://github.com/datadrivenag/tutorials
- BETY db plot boundaries
- Sample of UAV data
- Sample of RGB fullfield data
- Example RGB pyramid
- Custom environments
- See https://github.com/datadrivenag/tutorials
- Cloud9 with Octave/GCC, etc
- Jupyter with Octave kernel
- RStudio geospatial
- PostgresSQL studio
- XPRA
- Sample data
- Mounted readonly in all containers under /data/
- UAV:
- sites/ua-mac/Level_1/uav
- UAV:
- Some available via geoserver (above)
- Available via direct HTTP, in case needed
- Mounted readonly in all containers under /data/
Setup and support log
- 2/6/2018
- Initial conference call with organizers
- 2/9/2018
- Sent new estimates based on increased scope based on organizers meeting
- Confirmed resource availability at SDSC
- 2/12/2018
- Decided to go foward with instance at SDSC
- Began provisioning instance. This workshop is special since we'll be setting up an external NFS server and transferring ~2+TB of sample data for users
- Created NFS instance manually (in hindsight, should've just made it a labeled node) and basic master+GFS (2) + node (1) Kubernetes cluster
- Setup local Globus personal endpoint on NFS server to handle transfer from TERRA-REF endpoint
- 2/13/2018
- Setup datadrivenag.slack.com and github.com/datadrivenag organizations
- 2/15/2018
- Extended cloud9all image to include gcc and octave
- After some coordination with organizers, began transfer of sample data (~1 week duration). Babysit transfers, troubleshoot performance problem, resolve permission and other issues
- 2/16/2018
- Basic Geoserver configuration to run under Kubernetes
- Requested DNS change
- Began investigation of how to setup/scale geoserver (Geoserver on Steroids –
https://www.slideshare.net/geosolutions/geoserver-on-steroids)
- Updated Jupyter image to include octave kernel, provided sample notebook for Roman's plant height estimation
- Had to scale up NFS volume size from 2TB to 3TB to support RGB, LAS and VNIR data
- 2/21 - 2/22/2018
- Began tracking down UAV data. Loaded UAV data into endpoint and Geoserver
- Had to transfer via Drive due to lack of access to Globus endpoint
- 2/23/2018
- Scaled up cluster to 4 nodes (48cores) with monitoring
- Confirmed geoserver can indeed scale horizontally (scaled RC), but appears to present problem to UI session?
- Created simple NGINX server to allow users to browse data for download
- 2/24/2018
- Had to troubleshoot networking issue (flannel subnet bug)
- Fixed Jupyter image dependency problems for terrautils
- Tried to create example code to get sensor files by plot (sensorquery API)
- Updated github repo, documentation, created videos for Workbench, QGIS and Xpra
- etcd2 error on master suggests possible problems with slow filesystem at SDSC.
...