Blog

Blog

NCSA Power Outage

The NCSA 3003 data center lost power this morning at approximately 0735 CDT. This impacted both the VM servers used by the ISDA group as well as the openstack instance hosted by NCSA. This outage caused most of these servers to reboot, which in turn caused most of the virtual machines hosted on these servers to crash and restart.

We are currently investigating if there are any machines that did not come back up and will restart them. Feel free to contact the admins with any questions or if you notice one of the machines you need is not back up and running by noon central time.

NCSA OpenSource Emails

As some of you might have noticed the confluence server started to send out emails with an invalid FROM header as well as a prefix to the subject. The emails would have "Actividad sospechosa detectada" added to the subject header as well as made it look like they came from "Soporte BBVA Colombia <someemail>".  These emails did indeed originate from confluence, but the outgoing email address of confluence was modified.

Once this was brought to our light we immediately started to work with the NCSA security team to investigate this incident. We have discovered how this was done and have mitigated the issue. We are currently investigating if anything else was modified, however we not discovered any other modifications done. We will continue to work with the security team as well as do a more thorough investigation of what this user accessed. If we discover anything specific we will contact those projects and people affected.

These changes were made on September 27th 2018, at around 5:30pm central time. We undid these changes September 28th at around 11:00am central time.

software      : Bamboo
current       : 6.4.0
version       : 6.6.0
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.6+Release+Notes

software      : JIRA
current       : 7.8.0
version       : 7.10.2
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.10.x+release+notes

software      : Confluence
current       : 6.7.2
version       : 6.10.0
release notes : https://confluence.atlassian.com/display/DOC/Confluence+6.10+Release+Notes

software      : BitBucket
current       : 5.8.1
version       : 5.11.1
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.11+release+notes

software      : CROWD
current       : 3.1.2
version       : 3.2.3
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+3.2+Release+Notes

The main groups using HipChat and FishEye/Crucible have moved to using other software. With none of the main groups using this software I would like to retire these services on NCSA Open Source. My plan is to shut them down at the end of May 2018 and. At the end of May I will backup all the data and remove the services.

HIPCHAT AND FISHEYE ARE NOW STOPPED

FishEye/Crucible alternatives:

You can use bitbucket for code reviews. Since this is the service used to host the code in git, this works really nice together.

HipChat alternatives:

For Hipchat most groups have moved to Slack.

Feel free to contact Rob Kooper if you have any questions.

Migration HipChat Server

The HipChat server has been migrated to a different hypervisor. If you notice any issues please contact opensource@ncsa

Software Update 2017-01-05

The goal is to do a software update Friday Afternoon and should result in minor downtime of most services, this could require a quick reboot of the servers as well as updates to the software.

software : Bamboo
current : 6.2.3
version : 6.3.0
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.3+Release+Notes

software : JIRA
current : 7.6.0
version : 7.6.2
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.6.x+release+notes

software : Confluence
current : 6.5.1
version : 6.6.0
release notes : https://confluence.atlassian.com/display/DOC/Confluence+6.6+Release+Notes

software : BitBucket
current : 5.6.0
version : 5.6.2
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.6+release+notes

Buildservers Down

The buildservers will be down this week due to maintenance of the hardware hosting these machines. These machines are expected to be back up on Friday.

software      : Bamboo
current       : 6.2.2
version       : 6.2.3
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.2+Release+Notes

software      : JIRA
current       : 7.5.2
version       : 7.6.0
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.6.x+release+notes

software      : Confluence
current       : 6.5.0
version       : 6.5.1
release notes : http://confluence.atlassian.com/display/DOC/Confluence+6.5+Release+Notes

software      : BitBucket
current       : 5.5.0
version       : 5.6.0
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.6+release+notes

software      : FishEye
current       : 4.5.0
version       : 4.5.1
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.5+release+notes

software      : CROWD
current       : 3.0.1
version       : 3.1.1
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+3.1+Release+Notes

An emergency patch will be applied to bamboo tonight and will result in a short outage.

software      : Bamboo
current       : 6.1.1
version       : 6.2.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.2+Release+Notes

Upgrades are complete. During the upgrades the database of JIRA had to be dumped and reloaded. Send email to opensource@ncsa.illinois.edu if you discover anything missing or incorrect.

September 10 there will be an upgrade of the following software pieces. The upgrade is expected to take a total of 4 hours from 10am to 2pm central time. During this time some services will be unavailable for a short amount of time.

software      : Bamboo
current       : 6.0.3
version       : 6.1.1
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.1+Release+Notes

software      : JIRA
current       : 7.3.8
version       : 7.5.0
release notes : https://confluence.atlassian.com/display/JIRASOFTWARE/JIRA+Software+7.5.x+release+notes

software      : Confluence
current       : 6.2.2
version       : 6.4.0
release notes : http://confluence.atlassian.com/display/DOC/Confluence+6.4+Release+Notes

software      : BitBucket
current       : 5.1.0
version       : 5.3.1
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.3+release+notes

software      : FishEye
current       : 4.4.1
version       : 4.4.2
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.4+release+notes

software      : CROWD
current       : 2.12.0
version       : 3.0.1
release notes : https://confluence.atlassian.com/display/CROWD/Crowd+3.0+Release+Notes


software      : NEXUS
current       : 2.12.1
version       : 2.14.5

2.12.1-02.12.1-0

I will need to update Bamboo for a security update. This will be done Tuesday at 09:00 central time. Expected downtime is about 1 hour. During this time I will also update some other components of opensource.


software : Bamboo
current : 6.0.0
version : 6.0.3
released : 07-Jun-2017
release notes : https://confluence.atlassian.com/display/BAMBOO/Bamboo+6.0+Release+Notes


software : JIRA
current : 7.3.6
version : 7.3.7
released : 04-Jun-2017
release notes : http://confluence.atlassian.com/display/JIRA/JIRA+7.3.7+Release+Notes


software : Confluence
current : 6.1.3
version : 6.2.2
released : 12-Jun-2017
release notes : https://confluence.atlassian.com/display/DOC/Issues+resolved+in+6.2.2


software : BitBucket
current : 5.0.0
version : 5.1.0
released : 04-Jun-2017
release notes : http://confluence.atlassian.com/display/BitbucketServer/Bitbucket+Server+5.1+release+notes


software : FishEye
current : 4.4.0
version : 4.4.1
released : 15-May-2017
release notes : https://confluence.atlassian.com/display/CRUCIBLE/Crucible+4.4+release+notes


update

Finished updating HipChat, JIRA, bamboo, crowd, fisheye and confluence and Bitbucket. Confluence now has the collaborative editing enabled.


The plan is to upgrade the following pieces of software on Saturday 2017-05-06 in the morning. Confluence is a full version upgrade and will be done last, if I can't get it to work before 11am it will be rolled back to the previous version.


Software      : Bamboo
current       : 5.15.3
version       : 6.0.0
released      : 26-Apr-2017
software      : JIRA
current       : 7.2.2
version       : 7.3.6
released      : 26-Apr-2017
software      : Confluence
current       : 5.10.7
version       : 6.1.3
released      : 01-May-2017
software      : BitBucket
current       : 4.10.0
version       : 5.0.0
released      : 04-May-2017
software      : FishEye
current       : 4.2.0
version       : 4.4.0
released      : 11-Apr-2017
software      : CROWD
current       : 2.11.1
version       : 2.12.0
released      : 26-Apr-2017
Buildservers down

All buildservers will be down today due to hardware maintenance. All other services will operate as normal.

Apologies for the late notice.

Hipchat Upgrade

My apologies for reporting this so late.

HipChat server was updated on March 5th for a security update. This seemed to be small update that should not have any effects. As it turns out this was not the case. During this upgrade the authentication mechanism had changed, resulting in an issue where people could not authenticate. If you had a HipChat session open, it would continue to be validated, if you did not have an open HipChat session the server could not authenticate you.

Also during this upgrade XMPP was disabled (which is now enabled again).

Again apologies, if ever something does not work as expected, feel free to contact us at opensource@ncsa.illinois.edu

 

    

 

Downtime 2017-01-21/22

I’m planning on updating all hypervisors this weekend either Saturday or Sunday morning. This will require all VM’s to be shutdown. Downtime should be around 2 hours or so. This will also impact opensource.ncsa.illinois.edu (especially confluence and JIRA). Depending on how things go, I might also use this time to upgrade some of the software for opensource (jira, bitbucket, etc).

Update: All servers are back up and running. Downtime took a little longer due to the fact that a physical reboot was needed of the servers, requiring me to get to NCSA first.