See also Shared data directories
Download the NBI data
git clone https://github.com/kaleoyster/ProjectNBI
python ./nbiCsvJsonConverter-2/Downloadv1.py
Creates directory NBIDATA containing the raw data
python3 ./nbiCsvJsonConverter-2/ProcessMain.py
Converts the data to JSON format
Getting the data to Workbench
We'll need to add the shared directory under gluster and transfer or download the data directly.
We could setup a globus endpoint or use Santiago?
Import into MongoDB
We will host the raw data under /shared/NBIDATA/ as a read-only volume
We will also host an instance of MongoDB in the "public"? namespace with the official database
Users can access via nbidata.public
This will require running a process to ingest the data.
The "public" namespace can have no service timeouts
Mongo must be accessible to all namespaces, even after we apply network security policies.