Blog

Blog

Open Source Outage

Today there was an unexpected outage of NCSA Open Source. Most of the system came back relatively fast however JIRA and Confluence where down for extended period time. All systems should be up and running again. If not let me know.

Longer update. Today I tried to update the system to patch the 'dirty cow' exploit. This required an update to the kernel as well as a reboot. During the install of the kernel on either this system, or another system a wrong kernel was installed. The secondary system exports drives as NFS mounted fiesystem to the VM server. The secondary system did not come back up correctly and potentially caused the vm server to become unhappy. When reboot the VMs for opensource to mitigate the 'dirty cow' bug, the servers did not want to reboot and required me to physically shutdown the VM server. Once the secondary machine was fixed and the VM server was restarted all services came back as normal.

Software Updates

The following software was updated. Especially the update to Bitbucket is worth mentioning. Bitbucket now has a new homescreen that shows all your pull requests as well as the pull requests you are reviewing.

software : Bamboo
current : 5.13.0.1
version : 5.13.2
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.13+Release+Notes

software : JIRA
current : 7.2.0
version : 7.2.2
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.2.x+release+notes#JIRASoftware7.2.xreleasenotes-7-2-2

software : Confluence
current : 5.10.4
version : 5.10.7
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.10.7+Release+Notes

software : BitBucket
current : 4.9.1
version : 4.10.0
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.10+release+notes

software : FishEye
current : 4.1.2
version : 4.2.0
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.2+release+notes

software : CROWD
current : 2.9.1
version : 2.10.1
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+2.10.1+Release+Notes

 

Bitbucket Upgrade

Bitbucket was upgraded. The biggest change is that this allow project admins to select different merge strategies for each repository. The user doing the merge can select any of the merge strategies selected by the project admin when doing the final merge.

The merge strategy options are:

Merge commitAlways create a new merge commit and update the target branch to it, even if the source branch is already up to date with the target branch.
Fast-forwardIf the source branch is out of date with the target branch, create a merge commit. Otherwise, update the target branch to the latest commit on the source branch.
SquashCombine all commits into one new non-merge commit on the target branch.
Fast-forward onlyIf the source branch is out of date with the target branch, reject the merge request. Otherwise, update the target branch to the latest commit on the source branch.
Squash, fast-forward onlyIf the source branch is out of date with the target branch, reject the merge request. Otherwise, combine all commits into one new non-merge commit on the target branch.
Software Updates

UPDATE: All services have been updated and everything should be back to normal.

 

The weekend of the 2016-08-27/28 I will be upgrading software on NCSA Open Source:

software : Bamboo
current : 5.12.3.1
version : 5.13.0.1
released : 23-Aug-2016
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.13+Release+Notes

software : JIRA
current : 7.1.9
version : 7.2.0
released : 23-Aug-2016
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.2.x+release+notes

software : Confluence
current : 5.10.3
version : 5.10.4
released : 21-Aug-2016
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.10.4+Release+Notes

software : BitBucket
current : 4.8.3
version : 4.8.5
released : 21-Aug-2016
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.8+release+notes

software : FishEye
current : 4.1.1
version : 4.1.2
released : 16-Aug-2016
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.1+release+notes

 

OS Upgrades

I'm planning to upgrade the OS of the servers this coming Wednesday.I would like to take a full day of outage (hoping for it to be only a few hours). This would upgrade the OS on the main server from Ubuntu 12.04 to 16.04 and all other servers from Ubuntu 14.04 to 16.04. Please let me know ASAP if this would not work for your project.

UPDATE

The upgrade is completed and all systems are back up and running. Please email opensource@ncsa.illinois.edu if anything does not work as expected and I will look at it ASAP. The following problems were part of the blame for longer than expected outage.

During this upgrade we had some trouble with one of the plugins in confluence (that was not being used) and has been removed. I might contact some of you later with questions about some of the other plugins to see if we can disable some of those plugins in the hope that this will cause less trouble in the future and will make confluence a little faster.

A second problem during this upgrade an unexpected problem with Bitbucket appeared. Bitbucket does not support MySQL 5.7 which happens to be the default version in Ubuntu 16.04. To overcome this I have migrated the database for Bitbucket to PostgreSQL instead of MySQL. I might migrate over time others (FishEye and Crowd) to PostgreSQL as well to have a consistent set of databases. Both JIRA and Confluence are already using PostgreSQL.

Software Updates

The following software will be updated this weekend (2016-07-23):

software : Bamboo
current : 5.12.2.1
version : 5.12.3.1
released : 07-Jul-2016
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.12+Release+Notes

software : JIRA
current : 7.1.8
version : 7.1.9
released : 04-Jul-2016
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.9v7.1.9-06July2016

software : Confluence
current : 5.10.1
version : 5.10.2
released : 13-Jul-2016
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.10.2+Release+Notes

software : BitBucket
current : 4.7.1
version : 4.8.1
released : 19-Jul-2016
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.8+release+notes

software : FishEye
current : 4.1.0
version : 4.1.1
released : 08-Jul-2016
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.1+release+notes

 

Software Updates

The following pieces of software will be updated this weekend:

software : Bamboo
current : 5.12.1
version : 5.12.2.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.12+Release+Notes

software : JIRA
current : 7.1.7
version : 7.1.8
release notes :https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.8v7.1.8-20June2016

software : Confluence
current : 5.9.11
version : 5.10.0
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.10+Release+Notes

software : BitBucket
current : 4.6.2
version : 4.7.1
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.7+release+notes

 

software : HipCat
current : 1.4.1
version : 1.4.2
release notes : https://confluence.atlassian.com/hc/hipchat-server-release-notes-608731400.html

This weekend I will be updating the underlying hypervisor software and will need some total downtime of all machines. Once the update is complete the machines will come back up. During this time I will also update the following packages:

software : Bamboo
current : 5.10.3
version : 5.12.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.12+Release+Notes

software : JIRA
current : 7.1.4
version : 7.1.7
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.7v7.1.7-18May2016

software : Confluence
current : 5.9.8
version : 5.9.11
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.9.11+Release+Notes

software : BitBucket
current : 4.5.2
version : 4.6.2
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.6+release+notes

software : FishEye
current : 4.0.3
version : 4.0.4
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.0+release+notes

software : CROWD
current : 2.8.4
version : 2.9.1
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+2.9+Release+Notes

 

This upgrade was originally slated for 2015-05-09 but was postponed to the next day.

Tonight the HipChat server will be upgraded for an emergency patch. This upgrade will update the server to version 2.0 (from 1.4). Unfortunately there is no fix to the server unless we do this major upgrade. This upgrade will take around 1 hour and requires a reboot, which will forcefully log everybody out. For a list of all changes see: https://confluence.atlassian.com/hc/hipchat-server-release-notes-608731400.html

SPAM

We seem to be getting hit with SPAM. We try to remove them as fast as we can. If you see any please forward it to opensource@ncsa.illinois.edu and we'll remove them. The only option to prevent SPAM is to disable the account creation, which is something we don't want to do.

I installed a plugin that will allow you to mark items as spam, please us this to help us remove SPAM ASAP.

Updated finished

JIRA, Confluence, Bamboo, Bitbucket, FishEye, HipCHat and Nexus all have been updated to the latest version. 

During some rearranging of equipment in the machine room I accidentally unplugged opensource which resulted in an outage. At around 14:00 the system was back up and running. No data was lost during this time and all services are back up and running.

2016-04-10 Upgrades

The following upgrades will be done on Sunday 2016-04-10. During this time some of the applications will be unavailable. This page will be updated when the updates are done.

software : HipChat
current : 1.3.8
version : 1.3.9
released : 24-Mar-2016
release notes : https://confluence.atlassian.com/display/HC/HipChat+Server+Release+Notes

software : Bamboo
current : 5.10.1.1
version : 5.10.3
released : 14-Mar-2016
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.10+Release+Notes

software : JIRA
current : 7.0.10
version : 7.1.4
released : 06-Apr-2016
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.4

software : Confluence
current : 5.9.6
version : 5.9.7
released : 17-Mar-2016
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.9.7+Release+Notes

software : BitBucket
current : 4.3.2
version : 4.5.1
released : 05-Apr-2016
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.5+release+notes

software : FishEye
current : 3.10.3
version : 4.0.3
released : 23-Mar-2016
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.0+release+notes

 

Last night the server hosting opensource crashed. It was brought back online at 8:30am central time.

The following software will be updated on Sunday:

software : Bamboo
current : 5.10.1.1
version : 5.10.2
released : 01-Mar-2016
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+5.10+Release+Notes

software : JIRA
current : 7.0.10
version : 7.1.1
released : 02-Mar-2016
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.1.x+release+notes#JIRASoftware7.1.xreleasenotes-v7.1.1

software : Confluence
current : 5.9.4
version : 5.9.6
released : 03-Mar-2016
release notes : http://confluence.atlassian.com/display/DOC/Confluence+5.9.6+Release+Notes

software : BitBucket
current : 4.3.2
version : 4.4.1
released : 02-Mar-2016
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+4.4+release+notes