...
Remember not to change an existing EUPS tag when fixing bugs reported in the build tests. Always create a new package. It is perfectly OK to have old packages fail in a product. Most products have, or will eventually have, such obsolete, failing packages. The webreports show the packages with the highest package versions (the number after the '+').
Propagating changes through the dependency tree
With the exception of trunk packages (see below), packages never change. If one releaes a new version of product, then all the other products that depend on it likely need a new + version as well, as one wishes to update its dependency. If a package 'at the bottom' of the dependency graph is released, many such new packages may have to be created. This is a time consuming and error-prone process to do manually.
We have a tool called eeups_propagator that partially automates this process. The tool is part of the EEUPS package:
| No Format |
|---|
$ eups distrib install eeups -t EEUPS-stable
$ setup eeups -t EEUPS-stable
|
Since the tool can easily create hunderts of packages, it is important to understand how the tool works before it is applied.
The tool operates in several steps:
Step 1: Tell eeups_propagate what to do
First the eeups_propagator analyzes the dependency graph and the arguments given to it, and decides which new packages should be created and what changes should be made their dependencies.
The following always holds:
- No existing packages are changed in any way.
- For each product-productversion at most one new + version is introduced, even if several changes have to be applied to it.
- Newly introduced + versions are a copy of the previous + version.
- The only changes made are to the
setupRequired(..)entries in the table file. - No new dependencies are added, nor are dependencies removed. They may be replaced by different versions.
Step 1.1: Most-recent dependencies only
The eeups_propagator updates the dependencies of every most-recent + version that has a dependency to a not-most-recent + version. This happens if no options are given to eeups_propagator:
| No Format |
|---|
$ eeups_propagator
|
For example, say the following packages exist:
| No Format |
|---|
python-2.7.3+0
python-2.7.3+1
python-2.7.3+2
pyfits-3.0.7+0 depends on python 2.7.3+0
pyfits-3.0.8+0 depends on python 2.7.3+0
pyfits-3.0.8+1 depends on python 2.7.3+1
|
Then two new packages would have to be created:
| No Format |
|---|
pyfits-3.0.7+1 depends on python-2.7.3+2
pyfits-3.0.8+2 depends on python-2.7.3+2
|
eeups_propagator always performs those changes as there is little point in having most-recent packages depend on packages with out-dated build and table files.
Such changes propagate through the dependency graph. If there is a package that depends on pyfits-3.0.8+1 a new version of it would be created as well, pointing to pyfits-3.0.8+2 and so on.
Step 1.2: Replace a product version
Using the --replace oldpackage newpackage option, eeups_propagator can also be used if a newly released package has not just changed its package version, but also the product version (the part before the +).
For example, say a problem with perl 5.10.1 forces us to switch to perl 5.18.1. A package perl-5.18.1+0 is created that should replace the old perl-5.10.1+1 which is currently used by countless other packages. By default eeups_propagator will not do anything here if perl-5.10.1+1 is the most recent + version of perl.5.10.1. In this scenario we can apply the --replace option:
| No Format |
|---|
$ eeups_propagator --replace perl-5.10.1+1 perl-5.18.1+0
|
Now eeups_propagator will 'replace' all dependencies to perl-5.10.1+1 with dependencies to perl-5.18.1+0, meaning that it will create a new + version for any package that depends on the old perl package.
Of course the change is propagated upwards through the dependency graph.
Step 2: Plan the changes and verify consistency
Once eeups_propagator is invoked, it uses the information given to it to assemble a list of all the packages to be created. It runs the same verification routines on them (locally), that would be run on the server if the packages would be created on the eups repository at this stage. This detects if the changes introduce cyclic dependencies or version conflicts of any kind.
If a problem is encountered, the list of all planned changes is printed out, together with a list of the problems found, and the user is asked if he would like to continue anyway.
if the --askfirst option is given, the user is asked for permission to continue even if no problems were found.
Step 3: Checking out and applying changes
eeups_propagator will now internally run a svn checkout for each affected product. The working copies are created in the local directory, unless specified differently with --work. It checks out the URL of the product specified in the eups.cfg file, that is, it checks out the directory tree under which the build and table files are stored.
The script then applies the changes as planned. It internally uses svn copy to create new packages and then modifies the table file as required.
The changes are not commited automatically!
Step 4: Manually check/modifiy the changes
After the checkout has completed and the changes have been applied to the working copies, the eeups_propagator tool exits. It does not commit the changes directly.
The user can now look at the modified working copies and inspect the work done by eeups_propagator. All changes are already marked with svn add and are ready for commit.
It is perfectly possible to make manual changes at this point. The directories are regular subversion working copies. This can be very handy, for example to manually fix the problems reported in step 2.
Step 5: Commit
Once sufficiently convinced that the changes are fine, they can be commited.
Since there might be a rather large number of working copies that need submitting, eeups_propagator has already prepared a bash script for this.
The script is called commit and is located in the same directory as the checked out working copies. Invoke it with the commit message as argument:
| No Format |
|---|
$ ./commit "Replaced perl-5.10.1+1 with perl-5.18.1+0 in all most recent \+ versions."
|
The new packages should appear in the eups repository and web pages after the usual delays.
Remember: If something goes very wrong, the changes can always be undone thanks to subversions versioning. See Red Book (scroll down to "Undoing Changes").
Trunk Packages
In general the package management system follows the principle that a package is never changed once created. Trunk-packages are the exception to this rule.
...
The tool operates in several steps:
Step 1: Tell eeups_propagate what to do
First the eeups_propagator analyzes the dependency graph and the arguments given to it, and decides which new packages should be created and what changes should be made their dependencies.
...
- No existing packages are changed in any way.
- For each product-productversion at most one new + version is introduced, even if several changes have to be applied to it.
- Newly introduced + versions are a copy of the previous + version.
- The only changes made are to the
setupRequired(..)entries in the table file. - No new dependencies are added, nor are dependencies removed. They may be replaced by different versions.
Step 1.1: Most-recent dependencies only
The eeups_propagator updates the dependencies of every most-recent + version that has a dependency to a not-most-recent + version. This happens if no options are given to eeups_propagator:
...
Such changes propagate through the dependency graph. If there is a package that depends on pyfits-3.0.8+1 a new version of it would be created as well, pointing to pyfits-3.0.8+2 and so on.
Step 1.2: Replace a product version
Using the --replace oldpackage newpackage option, eeups_propagator can also be used if a newly released package has not just changed its package version, but also the product version (the part before the +).
...
Of course the change is propagated upwards through the dependency graph.
Step 2: Plan the changes and verify consistency
Once eeups_propagator is invoked, it uses the information given to it to assemble a list of all the packages to be created. It runs the same verification routines on them (locally), that would be run on the server if the packages would be created on the eups repository at this stage. This detects if the changes introduce cyclic dependencies or version conflicts of any kind.
...
if the --askfirst option is given, the user is asked for permission to continue even if no problems were found.
Step 3: Checking out and applying changes
eeups_propagator will now internally run a svn checkout for each affected product. The working copies are created in the local directory, unless specified differently with --work. It checks out the URL of the product specified in the eups.cfg file, that is, it checks out the directory tree under which the build and table files are stored.
...
The changes are not commited automatically!
Step 4: Manually check/modifiy the changes
After the checkout has completed and the changes have been applied to the working copies, the eeups_propagator tool exits. It does not commit the changes directly.
...
It is perfectly possible to make manual changes at this point. The directories are regular subversion working copies. This can be very handy, for example to manually fix the problems reported in step 2.
Step 5: Commit
Once sufficiently convinced that the changes are fine, they can be commited.
...