User's Guide
For an abridged version (but not a replacement) of this guide see The Impatient's Guide to DESDM EUPS installation, which contains specific information about installing on OSX (10.7-10.11).
This is the user manual for DESDM's new package management system. It targets both package authors and users that would like to use this package manager to install software on their systems.
This package management system should run on any Linux machine. Due to the differences of the various Linux distributions (and versions of them) we cannot guarantee that all the packages will build without errors on every possible machine. We do however automatically test all packages on a set of test machines running CentOS. In the future additional test systems will be provided.
The DESDM package management system is based on EUPS 1.2.30. EUPS was not modified for DESDM, instead some additional scripts were written to support some of the use-cases.
| Table of Contents | ||||||
|---|---|---|---|---|---|---|
|
Nomenclature
Term | Description |
|---|---|
Product | A product is a name for a software component. Typical products are 'cfitsio', 'atlas', 'sextractor' and so on. |
Package | A package is specific version of a product, compiled in a specific way, possibly linking to a specific set of libraries (other packages). |
Installing EUPS
Before any packages can be installed, EUPS must be installed. It its highly recommended to use our install script to do this, as it sets some additional environment variables which will be assumed to be present while working with the package manager. We do not support alternative installation methods.
...
It turns out to be quite hard to provide an accurate list of prerequisites (for example: How to reliably detect the minimum version of libc needed by sextractor?). The following list is updated with the current state of our knowledge.
Dependency | Used by | RHEL / centOS package (if known) | OSX |
|---|---|---|---|
python (some old version will do) | Package management system itself | python | - |
wget | Package management system itself | wget | wget (install wget from source) |
| curl | Package management system itself | curl | curl (install instructions soon) |
unzip | oracleclient 11.1.0.7 | unzip | - |
zlib | libpng 1.2.38, python 2.7.3 | zlib zlib-devel | zlib |
bzip2 | python 2.7.3 | bzip2 bzip2-devel | bzip2 |
libfreetype | plplot 5.8.0 | freetype freetype-devel | freetype (install freetype from source) |
libX11 | plplot 5.9.9 | libX11 libX11-devel |
|
libXau | plplot 5.9.9 | libXau libXau-devel |
|
libxcb | plplot 5.9.9 |
|
|
libXext | plplot 5.9.9 | libXext libXext-devel |
|
libxpm | root 5.26.00 | libXpm libXpm-devel |
|
libXft | root 5.26.00 | libXft libXft-devel |
|
libcrypt | python 2.7.3 | openssl openssl-devel glibc-devel |
|
libaio | oracleclient 11.1.0.7 | libaio libaio-devel |
|
libfreebl3 | python 2.7.3 | nss-softokn-freebl nss-softokn-freebl-devel |
|
libnsl | python 2.7.3 | glibc-devel |
|
gcc | various packages | gcc gcc-c++ make autoconf patch | Apple Xcode 5.1.1 (dmg) |
gfortran | atlas 3.8.4, netlib-lapack 3.4.1 | gcc-gfortran | version >= 4.8.2 (gfortran dmg) |
pkg-config | glib 2.29.2 | pkgconfig | |
subversion | Package management system itself | svn (subversion on rhel5) | - |
rsync | unitRunning 0.0.11 | rsync | rsync |
| tar | desdm_eupsinstall.py | tar |
We optionally support Intel's ICC and MKL (see the next section)
...
The following table lists the minimum versions required. Note that gcc and gfortran are still required since not all packages might support ICC and MKL (earlier versions of sextractor for example).
Dependency | Used by |
|---|---|
icc (>=11.1) | sextractor (icc&mkl build only) |
mkl (>=10.3) | sextractor (icc&mkl build only) |
ICC and MKL can only be used in combination.
...
Lets look at a few examples:
| Product Name | Configset | Product Version | Package Version |
|---|---|---|---|---|
cfitsio 3.300+0 | cfitsio |
| 3.300 | 0 |
libjpeg 6b+1 | libjpeg |
| 6b | 1 |
altas_netlib 3.8.4+2 | atlas | netlib | 3.8.4 | 2 |
Product Name
The product name is an non-empty string that may contain the following characters:
...
There are some environment variables that can be assumed to be present in every build script:
Variable | Description | Example Value | Example Value |
|---|---|---|---|
EXTERNAL | URL to the place were the tarballs | http://desbuild.cosmology.illinois.edu/eeups/webservice/resources/ | http://desbuild.cosmology.illinois.edu/eeups/webservice/resources/ |
SVNROOT | URL to the subversion repository. | https://dessvn.cosmology.illinois.edu/svn/desdm/devel/ | |
SVN_PATH | Relative path to the root folder of | tags/3.8.4+0 | tags/3.8.4+0 |
PRODUCT | Name of the product. For historical | atlas | atlas_netlib |
VERSION | Version of the product. | 3.8.4 | 3.8.4 |
PKG_VERSION | '+' part of the package version. | 1 | 1 |
FULL_VERSION | The complete package version | 3.8.4+1 | 3.8.4+1 |
FLAVOR | The flavor of the system where the | Linux64 | Linux64 |
PRODUCT_DIR | The base directory into which the | [...]/Linux64/altas/3.8.4+1 | [...]/Linux64/altas_netlib/3.8.4+1 |
[DEPENDENCY]_DIR | For every package this package | [...]/Linux64/cfitsio/3.300+0 | [...]/Linux64/cfitsio/3.300+0 |
If your package as dependencies (listed in the table file, see below), then all those packages will be installed and set-up before the script is executed. You can assume that all environment variables set by the table files of the dependencies are available.
...
Note that even if the table file looks a bit like a bash script on first sight, they are not. Regular bash commands will not work.
Command | Description | Examples |
|---|---|---|
setupRequired | Declares a dependency. | setupRequired(libpng 1.2.38+0) |
envPrepend | Adds a path in front of an environment | envPrepend(PATH, ${PRODUCT_DIR}/bin) |
envAppend | Adds a path to the end of an environment | endAppend(LD_LIBRARY_PATH, ${PRODUCT_DIR}/lib)) |
Rules for table files
- DO NOT USE version ranges. EUPS has some support to specify version ranges with setupRequirement. DESDM does not support this. Always specify the exact version. This ensures that we build packages exactly the same way everywhere, making installations reproducible.
- Always use curly brackets around variables. EUPS will not work without them.
- Use envPrepend instead of envAppend. This is important as it ensures that we really ending up using the software in the package and not some other installation that happens to be installed on the machine.
...
Further command line options available for the pkg_propagator:
| Parameter | |||||||
|---|---|---|---|---|---|---|---|
| --workdir | Allows you to specify a working directory where the old package versions are checked out from SVN and new package versions will be created. Defaults to the current directory. | ||||||
| --operational | Once set new 'operational' package versions are proposed (without 'E' appended to the package version). | ||||||
| --webservice | URL to the webservice from where the information on new package versions can be obtained. Defaults to http://desbuild.cosmology.illinois.edu/eeups/webservice/propagator . Note that this service can be called directly from a web browser. By entering a URL of the form
or
for 'experimental' or 'operational' package versions, respectively. In the angular brackets <replace>, <by>, <reference> you need to specify package versions such as python-2.7.6+1. As a result, you will see a json file with all the old ('orig') packages and the corresponding new ('prop') packages to be created. Example:
|
As usual, use
| No Format |
|---|
$ pkg_propagator --help |
...