2015-11-09 Stand Up Meeting notes

• Prepare for Argonne presentation next week; practice presentation, etc.
• GLTG tasks: GLGVO-105, GLGVO-176, GLGVO-160, GLGVO-178, GLGVO-161
• Meeting with Derek from IMLCZO about their new data sensor. They would like it to be automatically ingested.
• Earthcube -

• MSC
  • Continue writing AJT paper.
  • Study the influence of body temperature in the result of HMMs.
  • Study the influence of appetite in the result of HMMs.

• MSC
  • The logic behind the new defined limit on the number of data points used to develop HMMs was explained in the AJT paper.
  • New HMM were developed using body temperature as an additional parameter.
    • Results show the next to none influence of body temperature in the analysis.
  • New HMM are being developed using appetite as an additional parameter.
• Two days of vacation were used this week.

• NIST-CORE

• NIST-CORE

• BD: elasticity Tool Catalog integration design and implementation. Review Tool Catalog pull requests.
• BD/Earthcube: Review Luigi's jsonld search pull request. Test and fix for Clowder 1.0-RC1 changes.
• Earthcube: SAS units annotation.

• BD:
  • Reviewed Inna and Marcus' Tool Catalog pull requests.
  • Added 2 services on bd-elasticity to automatically start elasticity instances upon reboot. Added a wiki page to track elasticity instance and cloud VM status.
• BD/Earthcube:
  • Reviewed Luigi's jsonld search pull request, Brock's metadata definition GUI pull request.
  • Tested Clowder 1.0-RC1, documented many issues and a suggestion.
  • Added a web proxy method to solve the mixed active content blocking (HTTPS using HTTP url content) issue, for the adding metadata issue in Clowder, per Luigi's request.
  • Added a fix to remove duplicate metadata search results.
  • Deployed merged "develop" branch on ecgs-dev and tested.

• NDS prep for SC
• BD prep for SC

• NDS prep for SC
• BD prep for SC
• MWBDH Tools and Services coordination

• BD - work on polyglot .wf endpoint to call DataWolf and create a workflow from log file
• NIST - review pull requests, continue work on tornado scenario event

• BD - initial work for converting polyglot log files to workflows is complete, still need to document API
• NIST - reviewed pull requests (Monday PM did group code review), created build plan for NIST-Core develop branch, fixed logging level issue with geotools, started splitting out tornado scenario work into smaller tasks, emailed John about next tornado scenario event (a meeting is scheduled to demo what they have on 11/13/15)
• CyberSEES - meeting with Indira/Jong to discuss progress and issues

1. SEAD - Add url parameters to the Datasets and Collections lables (CATS-240)
2. SEAD - Normalize buttons links and icons (CATS-239)
3. MWRD - finalize Documentation after Barbara's comments
4. SEAD - more JIRA

1. DONE
2. Inventoried, started, half through
3. DONE
4. DONE CATS-280

• BD
  • Look into File Manager Brown Dog extension for Windows
  • Use Cases - Finish landsat extractor and deploy it, start chi-analysis extractor
  • SC Demo setup for BD use case
  • Clowder 1.0 RC1
• SDN
  • SDN flow rule update Journal paper - related work
  • Write up for realtime QoS module for the proposal

• BD - Landsat extractor
  • Finished writing the extractor
  • Some files were getting locked during run-time thus not letting subsequent steps to access it. Fixed this by creating a separate process for the arcgis operation
  • Even though the extractor program was correctly running, at the end of a execution it was giving socket error. After some investigation and talking to Sandeep, get to know that the socket is being closed before the final status update from the extractor. This happens as the computation takes around 7-8 mins and do not send/respond to the heartbeats sent by the rabbitmq server. 
  • To solve this, I tried to increase the heartbeat_interval as suggested by stackexchange forum from the client side. This did not work, as the heartbeat_interval is configured to be 60 sec at the server side.
  • Another way was to send status message to rabbitmq after every compute step. This resulted into the pickling error when tried to pass 'channel' object into the Process module as argument to the compute method.
  • Another method suggested by Rob was to run computation on a separate thread so that main thread is not blocked anymore and can reply to the heartbeats of rabbitmq server.
  • So, made changes to the pyclowder library and got it to work. One of the major change is the basic_ack needs to be moved from finally to the new thread for process File. Otherwise, the main thread immediately acknowledges of the process file completion, that is not correct. Futher testing needed for this.
• Participated in Clowder 1.0 RC1, tested new branch again DTS API, Chrome extension, other features of space and staging
• Attended Brock's practice presentation
• SDN
  • Sent write up for all the timing experiments, and other modules design for the CNS proposal to Sibin.

• BD
  • pyClowder HPC tasks
• VAT
  • Integrate and test precomuting distances code with develop
• DEBOD
  • Look into transforms for image de-skewing

• VAT
  • Integrated and tested precomuting distances code with develop
  • Bug fixes to Clowder
• DEBOD
  • Update meeting with the team
• BD
  • Working on sending parameters to remote method calls in Python

• BrownDog signup - update, see recent JIRA task by Kenton
• Tool Catalog pull req - work with Rui, make necessary changes

• Brown Dog signup
  • added Organization field
  • removed Terms and Conditions
  • changed email to admins to show user's Organization
• Tool Catalog
  • had a meeting with Kenton, Rui, on Monday, discussed changes for TC
  • need to have one form for buttons and URLs for source, docker, input/output
  • started implementing. Have two parts of create interface form
  • work in progress: process request with urls and/or uploaded files, "create interface part two"

• IARP - continue working on c++ extractor
• BD - fix green index extractor running on extractor-0014
• GLTG - gap filling for usgs

• IARP
  • c++ extractor - http requests
    • install and test curlcpp - get request worked but examples broken, posting broken
    • install curlpp - seems good - but more to do
  • tested upload script against clowder-jsonld - did some small fixes
• GLTG
  • release to production
  • gap filling almost working (one bug - not updating first gap datapoint)
  • create/submit pull request for add delete datatpoint endpoint
• BD
  • tool-catalog meeting
  • use case: meet with bill sullivan and xiangrong - will build green index within circle
• ISDA
  • clowder team work

• SEAD
  • Clean-Up Invites
  • Remove Files from CurationObject Datasets
• CyberSEES: Modify execute form from datawolf. Make it standalone.
• Vacation Thursday Afternoon and Friday.

• SEAD
  • Clean-up invites
  • Small fixes in merging spaces+staging into develop
• CyberSEES: 
  • Set up development environment in desktop (Ubuntu). 
  • Keep working on integrating datawolf and birdview editor
• Vacation Thursday afternoon and Friday

• BD
  • Review SC15 demo tasks
• MWRD
  • Confirm document transfer
• GLTG
  • Discuss demo results
  • Possibly re-prioritize tasks accordingly
• NIST
  • Work on action items from all-hands conference
• NCSA
  • SC15 prep
  • Onboarding team session

• MWRD
  • Final draft of documentation ready for delivery
• GLTG
  • Push to production planned
  • coordinating staffing adjustments
• NIST
  • wrapped up post-conference miscellany
• NCSA
  • SC15 prep work completed

• Change SWT map pane to swing map (redo due to separation of tornado plugin)
• Fix style opacity error
• Fix style close error
• Participate GIS day

• Participate GIS day
• Fixed raster data's nodata transparency issue
• Fixed ERGO-302 (scenario creation page's map selection)
• Fixed ERGO-285 data export error
• Fixed ERGO-307 multiple data selection

• Fix Eclipse Setup (Zest)
• OpenStack Setup
• Validate Orbit update has resolved ERGO-294
• Fix issue with SKOSMOS
• RDF

• Fixed Eclipse Setup
• ERGO-294
• ERGO-252 (in review)
• NIST-26 (in review)
• ERGO-287 (in review)
• ERGO-284 (in review)
• Updated Roadway Pull Request
• RDF

• SEAD
  • CATS-265 Fix listInsideDataset for Public/Private Instances
• MSC
  • find some new method to apply svm on different images (or subjects) , still not sure how to do it

• SEAD

*api.createCollection/Dataset without space & description, also check space and reject the request if space does not exist. or still space= default

*change error.url => illegal address, but url still has to begin with http
*list collection created by the user without attached to a space
*add owner to datasetlist.scala.html & collectionsList
*a simple fix metadata for curation. should be taken care later

• MSC
  • glom detect, not finish

TERRA

• Talk w/ Rob about EXIF image metadata extractor
• Refine technical documentation for data transfer from Danforth to NCSA
• Deploy extractor to dev instance of Clowder in Windows
• Refine GeoJSON previewer to accept JSON-LD format metadata

OTHER

• Thursday morning - academic employee orientation

TERRA

• Developed & committed working version of image metadata extractor
• Wrote previewer to handle lat/lon metadata and geoJSON metadata in google maps
• Began discussions over first draft of data transfer documentation
• Completed orientation