...
and be sure to get the 'current' versions without explictly knowing them.
EUPS User Utilities
User utilities support users in interacting with the EUPS system. Typical examples are utilities for creating new packages or for propagating version updates of a give package all the way up through the dependency tree.
All these tools are shipped as part of the eeups product:
| No Format |
|---|
$ eups distrib install eeups -t EEUPS-stable
$ setup eeups -t EEUPS-stable |
or for a specific version to obtain new features in beta release status:
| No Format |
|---|
$ eups distrib install eeups 1.4.3+2
$ setup eeups 1.4.3+2 |
With the exception of trunk packages (see below), packages never change. If one releaes a new version of product, then all the other products that depend on it likely need a new + version as well, as one wishes to update its dependency. If a package 'at the bottom' of the dependency graph is released, many such new packages may have to be created. This is a time consuming and error-prone process to do manually.
Two different versions of package propagation tools exist:
- Dependency Updater (pkg_propagator): Used for updating updating a given package for version updates of a direct or indirect dependency.
- Bulk Propagator (eeups_propagator): Used for propagating version updates in the bulk of all packages.
The following rules hold true for both propagator tools:
- Existing packages are not modified.
- For each product-productversion at most one new + version is introduced, even if several changes have to be applied to it.
- Newly introduced + versions are copies of the previous + versions. The only changes are applied to the table files (
setupRequired(..)entries ) to reflect changes in the dependencies. - No new dependencies are added, nor are dependencies removed. They may be replaced by different versions.
Dependency Updater
The typical user scenario looks as follows: Create a new version of a given 'reference' package by replacing one or several of its direct or indirect dependencies by a new version.
Let's do a specific example - actually an artificial example to increase the chance that this will never be added to the system so that you can always follow the steps (PLEASE: Skip the commit step in the end!).
Assume that you would like to to create a new diffimg version by replacing in diffimg-8.0.28+2 the indirect dependency python-2.7.6+1 by the updated version python-2.7.7+0. This will assume that you have already added python-2.7.7+0 into the system (which is not at the time of writing). Then you can run the following command:
| No Format |
|---|
$ pkg_propagator --replace python-2.7.6+1 --by python-2.7.7+0 --reference diffimg-8.0.28+2 |
This results in the following steps:
Step 1: Identify new package versions
The tool does not interact with a local eups repository. Hence, you don't need to first have the 'old' diffimg version installed in your environment. Rather, the tool fetches the information on new packages to be created from the webserver () and prints them to the console.
In the given example, the relevant parts in the dependency graph are
diffimg-8.0.28+2
--- coreUtils-0.5.2+4
--- psycopg-2.4.6+4
--- python-2.7.6+1
--- cxOracle-5.1.2+8
--- python-2.7.6+1
The new packages proposed by the system would be the following:
diffimg-8.0.28+3E
--- coreUtils-0.5.2+5E
--- psycopg2-2.4.6+5E
--- cxOracle-5.1.2+9E
where psycopg-2.4.6+5E and cxOracle-5.1.2+9E would have dependencies on python-2.7.7+0. The 'E' appended to the package version indicates that by default EXPERIMENTAL packages are created. On the console, you would see something like:
cxOracle-5.1.2+8 ==> cxOracle-5.1.2+9E
psycopg2-2.4.6+4 ==> psycopg2-2.4.6+5E
coreUtils-0.5.2+4 ==> coreUtils-0.5.2+5E
diffimg-8.0.28+2 ==> diffimg-8.0.28+3E
Note that if there would already be a psyopg2-2.4.6 version with the same dependencies as psycopg-2.4.6+4 but the new python version and identical build file there would be no need to create a new package version for psyopg2-2.4.6. This is recognized by the system and no new psyopg2-2.4.6 package version would be proposed. This means that if you specify a reference and a package to be replaced for which a propagated version already exists no propagation is performed.
Furthermore, it gives you a warning for those 'existing' packages (in the propagated graph) that have not status OK (or INCOMPLETE). Propagating packages that have already some known problem in one of its dependencies might not be a good thing to do.
Step 2: Checkout old packages and create the new packages
On the command line you are asked whether you want to proceed with the checkout. When confirming that with 'yes' the old package versions to be replaced are checked out from SVN to the current directory - unless another directory has been specified with the --workdir parameter (see below). Then, the proposed new package versions are created in the local checkout directory. Note that changes are not committed automatically!
Step 3: Manually check/modifiy the changes
After the checkout has completed and the changes have been applied to the working copies, the pkg_propagator tool exits. It does not commit the changes directly.
The user can now look at the modified working copies and inspect the work done by pkg_propagator. All changes are already marked with svn add and are ready for commit.
It is perfectly possible to make manual changes at this point. The directories are regular subversion working copies. This can be very handy, for example to manually fix the problems reported in step 1.
Step 4: Commit new packages
Commit changes by executing the commit bash script:
| No Format |
|---|
$ ./commit "New diffimg 8.0.28 package version with python updated to 2.7.7." |
Further Options
Further command line options available for the pkg_propagator:
- --workdir : Allows you to specify a working directory where the old package versions are checked out from SVN and new package versions will be created. Defaults to the current directory.
- --operational : Once set new 'operational' package versions are proposed (without 'E' appended to the package version).
--webservice : URL to the webservice from where the information on new package versions can be obtained. Defaults to http://desbuild.cosmology.illinois.edu/eeups/webservice/propagator . Note that this service can be called directly from a web browser. By entering a URL of the form http://desbuild.cosmology.illinois.edu/eeups/webservice/propagator/E/<replace>/<by>/<reference> or http://desbuild.cosmology.illinois.edu/eeups/webservice/propagator/<replace>/<by>/<reference> for 'experimental' or 'operational' package versions, respectively. In the angular brackets (e.g. <replace>) you need to specify package versions such as python-2.7.6+1. As a result, you will see a json file with all the old ('orig') packages and the corresponding new ('prop') packages to be created. Example: http://desbuild.cosmology.illinois.edu/eeups/webservice/propagator/E/atlas_netlib-3.8.4+6/atlas_netlib-3.8.4+7/eeups-1.4.3+2/
As usual, use
| No Format |
|---|
$ pkg_propagator --help |
to get help on the command line.
Known Issues
- Currently, only one single dependency can be replaced at the time. Work in progress to replace multiple packages at the same time.
- No transaction handling: The propagator interacts with both, the continuous integration build system (for fetching the information which packages to create) and SVN for the checkout and the commit. The continuous build system needs some time to update its state for SVN changes (at the order of 30-60 secs) so that SVN and continuous integration build system are not guaranteed to always by concurrent. As a result, users may experience problems when submitting proposed changes - e.g. when two users want to do the same thing at the same time.
Bulk Propagator
Since the tool can easily create hundreds of packages, it is important to understand how the tool works before it is applied.
The tool operates in several steps:
Step 1: Tell eeups_propagate what to do
First the eeups_propagator analyzes the dependency graph and the arguments given to it, and decides which new packages should be created and what changes should be made their dependencies.
The following always holds:
- No existing packages are changed in any way.
- For each product-productversion at most one new + version is introduced, even if several changes have to be applied to it.
- Newly introduced + versions are a copy of the previous + version.
- The only changes made are to the
setupRequired(..)entries in the table file. - No new dependencies are added, nor are dependencies removed. They may be replaced by different versions.
Step 1.1: Most-recent dependencies only
The eeups_propagator updates the dependencies of every most-recent + version that has a dependency to a not-most-recent + version. This happens if no options are given to eeups_propagator:
| No Format |
|---|
$ eeups_propagator
|
For example, say the following packages exist:
| No Format |
|---|
python-2.7.3+0
python-2.7.3+1
python-2.7.3+2
pyfits-3.0.7+0 depends on python 2.7.3+0
pyfits-3.0.8+0 depends on python 2.7.3+0
pyfits-3.0.8+1 depends on python 2.7.3+1
|
Then two new packages would have to be created:
| No Format |
|---|
pyfits-3.0.7+1 depends on python-2.7.3+2
pyfits-3.0.8+2 depends on python-2.7.3+2
|
eeups_propagator always performs those changes as there is little point in having most-recent packages depend on packages with out-dated build and table files.
Such changes propagate through the dependency graph. If there is a package that depends on pyfits-3.0.8+1 a new version of it would be created as well, pointing to pyfits-3.0.8+2 and so on.
Step 1.2: Replace a product version
Using the --replace oldpackage newpackage option, eeups_propagator can also be used if a newly released package has not just changed its package version, but also the product version (the part before the +).
For example, say a problem with perl 5.10.1 forces us to switch to perl 5.18.1. A package perl-5.18.1+0 is created that should replace the old perl-5.10.1+1 which is currently used by countless other packages. By default eeups_propagator will not do anything here if perl-5.10.1+1 is the most recent + version of perl.5.10.1. In this scenario we can apply the --replace option:
| No Format |
|---|
$ eeups_propagator --replace perl-5.10.1+1 perl-5.18.1+0
|
Now eeups_propagator will 'replace' all dependencies to perl-5.10.1+1 with dependencies to perl-5.18.1+0, meaning that it will create a new + version for any package that depends on the old perl package.
Of course the change is propagated upwards through the dependency graph.
Step 2: Plan the changes and verify consistency
Once eeups_propagator is invoked, it uses the information given to it to assemble a list of all the packages to be created. It runs the same verification routines on them (locally), that would be run on the server if the packages would be created on the eups repository at this stage. This detects if the changes introduce cyclic dependencies or version conflicts of any kind.
If a problem is encountered, the list of all planned changes is printed out, together with a list of the problems found, and the user is asked if he would like to continue anyway.
if the --askfirst option is given, the user is asked for permission to continue even if no problems were found.
Step 3: Checking out and applying changes
eeups_propagator will now internally run a svn checkout for each affected product. The working copies are created in the local directory, unless specified differently with --work. It checks out the URL of the product specified in the eups.cfg file, that is, it checks out the directory tree under which the build and table files are stored.
The script then applies the changes as planned. It internally uses svn copy to create new packages and then modifies the table file as required.
The changes are not commited automatically!
Step 4: Manually check/modifiy the changes
After the checkout has completed and the changes have been applied to the working copies, the eeups_propagator tool exits. It does not commit the changes directly.
The user can now look at the modified working copies and inspect the work done by eeups_propagator. All changes are already marked with svn add and are ready for commit.
It is perfectly possible to make manual changes at this point. The directories are regular subversion working copies. This can be very handy, for example to manually fix the problems reported in step 2.
Step 5: Commit
Once sufficiently convinced that the changes are fine, they can be commited.
Since there might be a rather large number of working copies that need submitting, eeups_propagator has already prepared a bash script for this.
The script is called commit and is located in the same directory as the checked out working copies. Invoke it with the commit message as argument:
| No Format |
|---|
$ ./commit "Replaced perl-5.10.1+1 with perl-5.18.1+0 in all most recent \+ versions."
|
The new packages should appear in the eups repository and web pages after the usual delays.
Remember: If something goes very wrong, the changes can always be undone thanks to subversions versioning. See Red Book (scroll down to "Undoing Changes").