Opensource will be brought down on Friday September 27th around 4pm central time and will be down until Saturday the 28th in the afternoon.During this time there will be power upgrades done as well as software upgrades.
Expectation is for opensource to be back online around 3pm central time on Saturday. Effort will be made to bring it up earlier.
The plan is to update NCSA Open Source the first week of April, exact data is still being determined. The outage will be require some updates to the server as well the following services below. During the updates to the server all of NCSA Open Source will be down, during the updates of the services listed below they will be down for shorter times. The expected work will be all day, but services can be brought up earlier.
software : Bamboo
current : 6.6.0
version : 6.8.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.8+Release+Notes
software : JIRA
current : 7.10.2
version : 8.0.2
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+8.0.x+release+notes
software : Confluence
current : 6.10.0
version : 6.15.1
release notes : https://confluence.atlassian.com/display/DOC/Confluence+6.15+Release+Notes
software : BitBucket
current : 5.11.1
version : 6.1.2
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+6.1+release+notes
software : CROWD
current : 3.2.3
version : 3.4.0
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+3.4+Release+Notes
The NCSA 3003 data center lost power this morning at approximately 0735 CDT. This impacted both the VM servers used by the ISDA group as well as the openstack instance hosted by NCSA. This outage caused most of these servers to reboot, which in turn caused most of the virtual machines hosted on these servers to crash and restart.
We are currently investigating if there are any machines that did not come back up and will restart them. Feel free to contact the admins with any questions or if you notice one of the machines you need is not back up and running by noon central time.
As some of you might have noticed the confluence server started to send out emails with an invalid FROM header as well as a prefix to the subject. The emails would have "Actividad sospechosa detectada" added to the subject header as well as made it look like they came from "Soporte BBVA Colombia <someemail>". These emails did indeed originate from confluence, but the outgoing email address of confluence was modified.
Once this was brought to our light we immediately started to work with the NCSA security team to investigate this incident. We have discovered how this was done and have mitigated the issue. We are currently investigating if anything else was modified, however we not discovered any other modifications done. We will continue to work with the security team as well as do a more thorough investigation of what this user accessed. If we discover anything specific we will contact those projects and people affected.
These changes were made on September 27th 2018, at around 5:30pm central time. We undid these changes September 28th at around 11:00am central time.
software : Bamboo
current : 6.4.0
version : 6.6.0
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.6+Release+Notes
software : JIRA
current : 7.8.0
version : 7.10.2
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.10.x+release+notes
software : Confluence
current : 6.7.2
version : 6.10.0
release notes : https://confluence.atlassian.com/display/DOC/Confluence+6.10+Release+Notes
software : BitBucket
current : 5.8.1
version : 5.11.1
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.11+release+notes
software : CROWD
current : 3.1.2
version : 3.2.3
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+3.2+Release+Notes
The main groups using HipChat and FishEye/Crucible have moved to using other software. With none of the main groups using this software I would like to retire these services on NCSA Open Source. My plan is to shut them down at the end of May 2018 and. At the end of May I will backup all the data and remove the services.
HIPCHAT AND FISHEYE ARE NOW STOPPED
FishEye/Crucible alternatives:
You can use bitbucket for code reviews. Since this is the service used to host the code in git, this works really nice together.
HipChat alternatives:
For Hipchat most groups have moved to Slack.
- Clowder: https://join.slack.com/t/clowder-software/shared_invite/enQtMzQzOTg0Nzk3OTUzLTUxYzVhMzZlZDlhMTc0NzNi...
- BrownDog: Contact either Rob Kooper or Shannon Bradley
Feel free to contact Rob Kooper if you have any questions.
The HipChat server has been migrated to a different hypervisor. If you notice any issues please contact opensource@ncsa
The goal is to do a software update Friday Afternoon and should result in minor downtime of most services, this could require a quick reboot of the servers as well as updates to the software.
software : Bamboo
current : 6.2.3
version : 6.3.0
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.3+Release+Notes
software : JIRA
current : 7.6.0
version : 7.6.2
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.6.x+release+notes
software : Confluence
current : 6.5.1
version : 6.6.0
release notes : https://confluence.atlassian.com/display/DOC/Confluence+6.6+Release+Notes
software : BitBucket
current : 5.6.0
version : 5.6.2
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.6+release+notes
The buildservers will be down this week due to maintenance of the hardware hosting these machines. These machines are expected to be back up on Friday.
software : Bamboo
current : 6.2.2
version : 6.2.3
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.2+Release+Notes
software : JIRA
current : 7.5.2
version : 7.6.0
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.6.x+release+notes
software : Confluence
current : 6.5.0
version : 6.5.1
release notes : http://confluence.atlassian.com/display/DOC/Confluence+6.5+Release+Notes
software : BitBucket
current : 5.5.0
version : 5.6.0
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.6+release+notes
software : FishEye
current : 4.5.0
version : 4.5.1
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.5+release+notes
software : CROWD
current : 3.0.1
version : 3.1.1
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+3.1+Release+Notes
An emergency patch will be applied to bamboo tonight and will result in a short outage.
software : Bamboo
current : 6.1.1
version : 6.2.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.2+Release+Notes
Upgrades are complete. During the upgrades the database of JIRA had to be dumped and reloaded. Send email to opensource@ncsa.illinois.edu if you discover anything missing or incorrect.
September 10 there will be an upgrade of the following software pieces. The upgrade is expected to take a total of 4 hours from 10am to 2pm central time. During this time some services will be unavailable for a short amount of time.
software : Bamboo
current : 6.0.3
version : 6.1.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.1+Release+Notes
software : JIRA
current : 7.3.8
version : 7.5.0
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.5.x+release+notes
software : Confluence
current : 6.2.2
version : 6.4.0
release notes : http://confluence.atlassian.com/display/DOC/Confluence+6.4+Release+Notes
software : BitBucket
current : 5.1.0
version : 5.3.1
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.3+release+notes
software : FishEye
current : 4.4.1
version : 4.4.2
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.4+release+notes
software : CROWD
current : 2.12.0
version : 3.0.1
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+3.0+Release+Notes
software : NEXUS
current : 2.12.1
version : 2.14.5
2.12.1-02.12.1-0
I will need to update Bamboo for a security update. This will be done Tuesday at 09:00 central time. Expected downtime is about 1 hour. During this time I will also update some other components of opensource.
software : Bamboo
current : 6.0.0
version : 6.0.3
released : 07-Jun-2017
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.0+Release+Notes
software : JIRA
current : 7.3.6
version : 7.3.7
released : 04-Jun-2017
release notes : http://confluence.atlassian.com/display/JIRA/JIRA+7.3.7+Release+Notes
software : Confluence
current : 6.1.3
version : 6.2.2
released : 12-Jun-2017
release notes : https://confluence.atlassian.com/display/DOC/Issues+resolved+in+6.2.2
software : BitBucket
current : 5.0.0
version : 5.1.0
released : 04-Jun-2017
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.1+release+notes
software : FishEye
current : 4.4.0
version : 4.4.1
released : 15-May-2017
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.4+release+notes
update
Finished updating HipChat, JIRA, bamboo, crowd, fisheye and confluence and Bitbucket. Confluence now has the collaborative editing enabled.
The plan is to upgrade the following pieces of software on Saturday 2017-05-06 in the morning. Confluence is a full version upgrade and will be done last, if I can't get it to work before 11am it will be rolled back to the previous version.
All buildservers will be down today due to hardware maintenance. All other services will operate as normal.
Apologies for the late notice.