This readme describes how to build a EUPS stack using Fermicloud VMs.

Pre-Requisites

1. Access to FermiLab and FermiCloud machines
2. Access to bitbucket repository (https://opensource.ncsa.illinois.edu/bitbucket/login?next=/projects/desdm/repos/docker_utils/browse)
3. Publishing rights on cvmfs

Authorization Issues

If there are permission issues it is most likely due to Kerberos tickets (see: https://mu2ewiki.fnal.gov/wiki/Authentication). Type the following to renew authentication:

kinit

Steps

Log into Fermicloud

From a FermiLab machine log into Fermicloud.

ssh ${USER}@fcluigpvm01.fnal.gov


Ensure ssh keys are available in:
${HOME}/.ssh/id_rsa.pub


Otherwise generate them:
ssh-keygen

Clone Repository

Specify bitbucket user name:

BB_USER=<username>
git clone -b devtools7 https://${BB_USER}@opensource.ncsa.illinois.edu/bitbucket/scm/desdm/docker_utils.git

Create Script

A Python script is being used to build bash scripts. The file script_builder.py creates these scripts. Example usage:

# MEPipeline Gcc7
python3 docker_utils/stackbuilder/cvmfs_scripts/script_builder.py \
--scripts_output_dir './' \
--product_name 'MEPipeline' \
--product_version 'Y5A1dev+43' \
--docker_image 'desdm/eeups_stackbuilder:cvmfs-devtoolset7-latest' \
--cvmfs_path "/cvmfs/des.opensciencegrid.org/eeupsservices/debug/gcc7" \
--stack_tar_name 'mep43'

For more options:

python3 docker_utils/stackbuilder/cvmfs_scripts/script_builder.py --help

Available Docker containers:
https://hub.docker.com/r/desdm/eeups_stackbuilder/tags

Build Stack

The script above generates two files:

fermicloud.sh and vm.sh

fermicloud.sh is the script that needs to be executed (vm.sh is being copied into the VM and executed there automatically)

Verify authentication by running:

onevm list

(on authentication error you might need to re-run 'kinit' and/or log into fermicloud again)

Start the process with:

./fermicloud.sh

(Note: for long builds it may be helpful to use screen to detach the process)

Shutdown VM

After the stack has been built the VM can be shutdown. 

To optain the vm id:

onevm list

To delete the vm:

onevm delete [ID]

If the Kerberos ticket is not valid anymore, simply re-login to Fermicloud.

Example:

-bash-4.1$ onevm list
    ID USER     GROUP    NAME            STAT UCPU    UMEM HOST             TIME
 45730 mwilli   users    mwilli eups_bui runn    0    1.9G fcl410       1d 00h57
-bash-4.1$ onevm delete 45730
Are you really sure you want to delete this VM? Type a Capital Y
Y
You seem sure, we proceed

Log into Publishing Machine

In a separate terminal:

ssh -l cvmfsdes oasiscfs.fnal.gov

Go to Target (Root) Location

The target location is where the stack tar file is being extracted. 

cd /cvmfs/des.opensciencegrid.org/eeupsservices/debug/gcc4

Start a Transaction

cvmfs_server transaction des.opensciencegrid.org

(optional) Abort Transaction

cvmfs_server abort des.opensciencegrid.org

More info: https://cvmfs.readthedocs.io/en/stable/cpt-repo.html#repository-update

Copy the Stack from Fermicloud to the Publishing Machine

Replace <stack.tar> with the correct file name.

scp  <stack.tar> cvmfsdes@oasiscfs.fnal.gov:/cvmfs/des.opensciencegrid.org/eeupsservices/debug/gcc4

If there are permission issues, try on the fermicloud machine:

kinit ${USER}@FNAL.GOV

Extract the Stack on the Publishing Machine

The tar file includes the following folder structure:

<product_name>/<version>/eups/..  for example: (MEPipeline/Y5A1dev+41/eups/..)

Example:

tar -pxf mep_stack.tar

Delete tar file.

Finish Transaction

Change to home directory:

cd

Finish the transaction:

cvmfs_server publish des.opensciencegrid.org

(Optional) - Test Stack

cd /cvmfs/des.opensciencegrid.org/eeupsservices/debug/python2/gcc4/MEPipeline/Y5A1dev+39/eups source desdm_eups_setup.sh setup -v MEPipeline

NOTE: it may take a couple of hours until the stack has been fully published.