My apologies for reporting this so late.
HipChat server was updated on March 5th for a security update. This seemed to be small update that should not have any effects. As it turns out this was not the case. During this upgrade the authentication mechanism had changed, resulting in an issue where people could not authenticate. If you had a HipChat session open, it would continue to be validated, if you did not have an open HipChat session the server could not authenticate you.
Also during this upgrade XMPP was disabled (which is now enabled again).
Again apologies, if ever something does not work as expected, feel free to contact us at opensource@ncsa.illinois.edu
I’m planning on updating all hypervisors this weekend either Saturday or Sunday morning. This will require all VM’s to be shutdown. Downtime should be around 2 hours or so. This will also impact opensource.ncsa.illinois.edu (especially confluence and JIRA). Depending on how things go, I might also use this time to upgrade some of the software for opensource (jira, bitbucket, etc).
Update: All servers are back up and running. Downtime took a little longer due to the fact that a physical reboot was needed of the servers, requiring me to get to NCSA first.
Today there was an unexpected outage of NCSA Open Source. Most of the system came back relatively fast however JIRA and Confluence where down for extended period time. All systems should be up and running again. If not let me know.
Longer update. Today I tried to update the system to patch the 'dirty cow' exploit. This required an update to the kernel as well as a reboot. During the install of the kernel on either this system, or another system a wrong kernel was installed. The secondary system exports drives as NFS mounted fiesystem to the VM server. The secondary system did not come back up correctly and potentially caused the vm server to become unhappy. When reboot the VMs for opensource to mitigate the 'dirty cow' bug, the servers did not want to reboot and required me to physically shutdown the VM server. Once the secondary machine was fixed and the VM server was restarted all services came back as normal.
The following software was updated. Especially the update to Bitbucket is worth mentioning. Bitbucket now has a new homescreen that shows all your pull requests as well as the pull requests you are reviewing.
software : Bamboo
current : 5.13.0.1
version : 5.13.2
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.13+Release+Notes
software : JIRA
current : 7.2.0
version : 7.2.2
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.2.x+release+notes#JIRASoftware7.2.xreleasenotes-7-2-2
software : Confluence
current : 5.10.4
version : 5.10.7
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.10.7+Release+Notes
software : BitBucket
current : 4.9.1
version : 4.10.0
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.10+release+notes
software : FishEye
current : 4.1.2
version : 4.2.0
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.2+release+notes
software : CROWD
current : 2.9.1
version : 2.10.1
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+2.10.1+Release+Notes
Bitbucket was upgraded. The biggest change is that this allow project admins to select different merge strategies for each repository. The user doing the merge can select any of the merge strategies selected by the project admin when doing the final merge.
The merge strategy options are:
| Merge commit | Always create a new merge commit and update the target branch to it, even if the source branch is already up to date with the target branch. |
|---|---|
| Fast-forward | If the source branch is out of date with the target branch, create a merge commit. Otherwise, update the target branch to the latest commit on the source branch. |
| Squash | Combine all commits into one new non-merge commit on the target branch. |
| Fast-forward only | If the source branch is out of date with the target branch, reject the merge request. Otherwise, update the target branch to the latest commit on the source branch. |
| Squash, fast-forward only | If the source branch is out of date with the target branch, reject the merge request. Otherwise, combine all commits into one new non-merge commit on the target branch. |
UPDATE: All services have been updated and everything should be back to normal.
The weekend of the 2016-08-27/28 I will be upgrading software on NCSA Open Source:
software : Bamboo
current : 5.12.3.1
version : 5.13.0.1
released : 23-Aug-2016
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.13+Release+Notes
software : JIRA
current : 7.1.9
version : 7.2.0
released : 23-Aug-2016
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.2.x+release+notes
software : Confluence
current : 5.10.3
version : 5.10.4
released : 21-Aug-2016
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.10.4+Release+Notes
software : BitBucket
current : 4.8.3
version : 4.8.5
released : 21-Aug-2016
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.8+release+notes
software : FishEye
current : 4.1.1
version : 4.1.2
released : 16-Aug-2016
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.1+release+notes
I'm planning to upgrade the OS of the servers this coming Wednesday.I would like to take a full day of outage (hoping for it to be only a few hours). This would upgrade the OS on the main server from Ubuntu 12.04 to 16.04 and all other servers from Ubuntu 14.04 to 16.04. Please let me know ASAP if this would not work for your project.
UPDATE
The upgrade is completed and all systems are back up and running. Please email opensource@ncsa.illinois.edu if anything does not work as expected and I will look at it ASAP. The following problems were part of the blame for longer than expected outage.
During this upgrade we had some trouble with one of the plugins in confluence (that was not being used) and has been removed. I might contact some of you later with questions about some of the other plugins to see if we can disable some of those plugins in the hope that this will cause less trouble in the future and will make confluence a little faster.
A second problem during this upgrade an unexpected problem with Bitbucket appeared. Bitbucket does not support MySQL 5.7 which happens to be the default version in Ubuntu 16.04. To overcome this I have migrated the database for Bitbucket to PostgreSQL instead of MySQL. I might migrate over time others (FishEye and Crowd) to PostgreSQL as well to have a consistent set of databases. Both JIRA and Confluence are already using PostgreSQL.
The following software will be updated this weekend (2016-07-23):
software : Bamboo
current : 5.12.2.1
version : 5.12.3.1
released : 07-Jul-2016
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.12+Release+Notes
software : JIRA
current : 7.1.8
version : 7.1.9
released : 04-Jul-2016
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.9v7.1.9-06July2016
software : Confluence
current : 5.10.1
version : 5.10.2
released : 13-Jul-2016
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.10.2+Release+Notes
software : BitBucket
current : 4.7.1
version : 4.8.1
released : 19-Jul-2016
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.8+release+notes
software : FishEye
current : 4.1.0
version : 4.1.1
released : 08-Jul-2016
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.1+release+notes
The following pieces of software will be updated this weekend:
software : Bamboo
current : 5.12.1
version : 5.12.2.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.12+Release+Notes
software : JIRA
current : 7.1.7
version : 7.1.8
release notes :https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.8v7.1.8-20June2016
software : Confluence
current : 5.9.11
version : 5.10.0
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.10+Release+Notes
software : BitBucket
current : 4.6.2
version : 4.7.1
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.7+release+notes
software : HipCat
current : 1.4.1
version : 1.4.2
release notes : https://confluence.atlassian.com/hc/hipchat-server-release-notes-608731400.html
This weekend I will be updating the underlying hypervisor software and will need some total downtime of all machines. Once the update is complete the machines will come back up. During this time I will also update the following packages:
software : Bamboo
current : 5.10.3
version : 5.12.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.12+Release+Notes
software : JIRA
current : 7.1.4
version : 7.1.7
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.7v7.1.7-18May2016
software : Confluence
current : 5.9.8
version : 5.9.11
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.9.11+Release+Notes
software : BitBucket
current : 4.5.2
version : 4.6.2
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.6+release+notes
software : FishEye
current : 4.0.3
version : 4.0.4
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.0+release+notes
software : CROWD
current : 2.8.4
version : 2.9.1
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+2.9+Release+Notes
This upgrade was originally slated for 2015-05-09 but was postponed to the next day.
Tonight the HipChat server will be upgraded for an emergency patch. This upgrade will update the server to version 2.0 (from 1.4). Unfortunately there is no fix to the server unless we do this major upgrade. This upgrade will take around 1 hour and requires a reboot, which will forcefully log everybody out. For a list of all changes see: https://confluence.atlassian.com/hc/hipchat-server-release-notes-608731400.html
We seem to be getting hit with SPAM. We try to remove them as fast as we can. If you see any please forward it to opensource@ncsa.illinois.edu and we'll remove them. The only option to prevent SPAM is to disable the account creation, which is something we don't want to do.
I installed a plugin that will allow you to mark items as spam, please us this to help us remove SPAM ASAP.
JIRA, Confluence, Bamboo, Bitbucket, FishEye, HipCHat and Nexus all have been updated to the latest version.
During some rearranging of equipment in the machine room I accidentally unplugged opensource which resulted in an outage. At around 14:00 the system was back up and running. No data was lost during this time and all services are back up and running.
The following upgrades will be done on Sunday 2016-04-10. During this time some of the applications will be unavailable. This page will be updated when the updates are done.
software : HipChat
current : 1.3.8
version : 1.3.9
released : 24-Mar-2016
release notes : https://confluence.atlassian.com/display/HC/HipChat+Server+Release+Notes
software : Bamboo
current : 5.10.1.1
version : 5.10.3
released : 14-Mar-2016
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.10+Release+Notes
software : JIRA
current : 7.0.10
version : 7.1.4
released : 06-Apr-2016
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.4
software : Confluence
current : 5.9.6
version : 5.9.7
released : 17-Mar-2016
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.9.7+Release+Notes
software : BitBucket
current : 4.3.2
version : 4.5.1
released : 05-Apr-2016
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.5+release+notes
software : FishEye
current : 3.10.3
version : 4.0.3
released : 23-Mar-2016
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.0+release+notes