Date
Attendees
- Yan Zhao
- Bing Zhang
- Sandeep Puthanveetil Satheesan
- Luigi Marini
- Shannon Bradley
- Rob Kooper
- Kenton McHenry
- Gregory Jansen
Agenda
- Updates - LDAP
- Rob needs Yan to make a server side to make a form post to make sure it works - a couple lines in flask
- Check documentation in request library for info on post
- CZO use case development
- basic mapping function - changed to logic Luigi recommended
- dockerization and docs (readme) - sandeep
- wsgi standard - gunicorn is a recommended standard - look at BDBox definitions
- there is also a new thing
- what ever is quick and deployable
- Bing PR - yan reviewed - Bing has screen shot in PR of what it returns
- Rank - term is confusing?
- List is random
- do we need ID - need mapping, dat/time means this -
- should we list the input headers and then the mappings below? discussion
- Need a way to input new mappings - discuss design on Monday
- Tool Catalog Updates/Cleanup
- Need to clean it up - confusing and there is a lot of things we don't use now
- We are using docker images now (before we were putting code in the catalog)
- Script terminology is confusing
- Clusterman call the list to do the deployment
- Clowder extractors place to live
- Clean up code base or start from scratch and go smaller - more like what we do in Clowder
- Able to modify and add
- Like NPM - has metadata - code and docker image live someplace else
- Tags to provide info on different views - all extractors, all converters, all libraries/software used - all extractors/converters using that library
- We know services libraries dependencies
- Simple ways to add
- Should be slim design
- Discuss in design meeting
- Port existing
- PyPy lets you put anything in there and it is the user who has to be careful
- Deployments
- we can deploy any services
- Rob says we are missing labels for scaling to work correctly
- Need mongo entry to recreate if docker crashes - logging so it can be restarted
- Clusterman does not provide parameters - environment variable, IP of swarm cluster master, polyglot docker container needs that
- Can clusterman deploy an extractor and work with scaling?
- needs reviewed and confirmed all is working
- is it done for polyglot containers - no - clusterman can only set max min labels for scaling -
- Bing Zhang - walk thru this next Wednesday and confirm status / what needs done
- 0.5 release
- issues in to do - move to Dalmation
- https://opensource.ncsa.illinois.edu/jira/projects/BD/versions/16593 - released
- 0.6 start reviewing what is in here
Discussion items
| Time | Item | Who | Notes |
|---|---|---|---|
| Bing | Follow up
| ||
| Luigi | In-Core is also annotating CSV files - Chen's semantic work Yan talk to her to get info on what she is doing They are using EarthCube project basis w3c standard for annotating CSV files - was originally RDF - now making JSON ld version In-Core using JSON ld version Can the same information be encoded using that standard in BD | ||
| Yan | Pecan task - use fence key instead of token Page is based on current BD page Yan will give Shannon copy of what she is using now | ||
| Sandeep | CZO - 1 task - updating to include column numbers? mark as won't do If current implementation - if it does not find a mapping does it post to clowder? no (well it posts an empty one) Yan suggested posting headers to top? it should still post something that doesn't have mappings but at least has header technically it is an ordered array ... not explicit - consumer might not realize this Headers - it is an array - it has an index - that is enough Python3 of bd.py in review - has to be 3 and is not backward compatible - support ending this year Work on bdcli python 3 next | ||
| Greg | Students did not make much progress last semester - 2 students - bar code and punch card - back next month to continue work not CS students. Undergrads in the iSchool. (former CS students) It's a bit frustrating - but they are also volunteers and not all of it is "easy" students are new coders - They need a lot of coaching, which I will need to do 1 on 1. They don't have to write BD code at all. Greg is going to package the services for them - Greg will use the new packaging Looking at end of spring term New testbed website ready to go - need to deploy for BD testbed will have public dashboards and grafana All tests are in containers - updated to docker swarm Would like to start testing at new scale Get an instance of BD running in a Docker swarm 2 January tasks. (a) deploy new testbed website w/public dashboards, and (b) resume regular testing, including higher scale stress tests Who will send Greg list of docker compose files? Sandeep Puthanveetil Satheesan will find - could give bdbase, clowder has been updated, look at Ben's also | ||
| All |
|