These design notes concern exposing the NBI data ( NDS-992 - Getting issue details... STATUS ) to users via workbench.
See also Shared data directories
Download the NBI data
git clone https://github.com/kaleoyster/ProjectNBI
python ./nbiCsvJsonConverter-2/Downloadv1.py
Creates directory NBIDATA containing the raw data
python3 ./nbiCsvJsonConverter-2/ProcessMain.py
Converts the data to JSON format
Getting the data to Workbench
We can initially add the shared directory to Gluster and transfer or download the data directly.
For this project, the raw data is probably less of a concern than the Mongo data – which poses an interesting question about data sharing.
We could setup a globus endpoint or use Santiago?
- santiago.ncsa.illinois.edu runs a globus endpoint (https://data.share.nationaldataservice.org/)
- The data exposed through santiago could be mounted on Workbench – but it's at NCSA, not SDSC
Import into MongoDB
We will host the raw data under /shared/NBIDATA/ as a read-only volume
We will also host an instance of MongoDB in the "public"? namespace with the official database
Users can access via nbidata.public
This will require running a process to ingest the data.
The "public" namespace can have no service timeouts
Mongo must be accessible to all namespaces, even after we apply network security policies.