Possible outage on April 19

 * * * POSSIBLE Outage Announcement * * *

On Tuesday, April 19th between 8:00am and 4:00pm Eastern time, the data center where Litech systems are co-located will undergo repairs to the automatic transfer switches that failed last week. Since our servers do not have redundant power supplies, it is possible we will be expected to power down briefly to switch between the A and B PDUs in the rack.

As much notice as possible will be given if an outage is necessary, but it may be a matter of only a few moments. Please avoid leaving unsaved work unattended or starting long-running tasks.

Update 9:22pm: Unsurprisingly, work remains ongoing, and will continue into Wednesday. It looks increasingly likely that no outage will be necessary, as the necessary repairs may be able to be completed without a loss of power.

Loss of power at data center

 * * * Outage Announcement * * *

On Friday, April 15th around 3:00am Eastern time, the data center where Litech systems are co-located suffered a complete power loss due to the simultaneous failures of both the A-side and B-side utility power transfer switches. Data center and electrical technicians are on site attempting to resolve the issue.

More information will be provided here as it becomes available.

Update 10:50am ET: The physical servers are online and VMs are booting up now.

Update 11:15am ET: All systems are back online.

Emergency maintenance on March 24th

  * * * Maintenance Announcement * * *

Due to a botched kernel livepatch (thanks Ubuntu), puck will be going offline on Wednesday, March 24th, around 3:00pm Eastern time. There is no reason to believe there will be any data integrity issues, but systemd is hung and it may eventually affect service availability without a reboot.

During this outage, the kernel and supporting environment will be updated, and an fsck will be run.

This outage should last 45-90 minutes.

Maintenance on September 23rd

 * * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, September 23rd, around 9:30am Eastern time, to upgrade the kernel and supporting environment. This outage should last 45-90 minutes.

Maintenance on October 30th

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, October 30th, around 9:30am to 10:00am Eastern time, while the hardware is moved to another data center. This will likely take one to two hours.

The new facility is part of the same infrastructure company, so no network configuration changes are necessary. All IP addresses, etc, will stay the same.

Maintenance on October 10th

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, October 10th, around 2:00pm Eastern time, to upgrade the kernel and supporting environment. This outage should last 45-60 minutes.

Maintenance on May 10th

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, May 10th, around 10:00am Eastern time, to upgrade the kernel and supporting environment. This outage should last 45-60 minutes.

Maintenance on February 21st

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, February 21st, around 10:00am Eastern time, to upgrade the kernel and supporting environment. This outage should last 30-45 minutes at the most.

Unplanned outage

Puck was shut down unannounced this afternoon due to failing power distribution equipment at the data center.  It was a controlled shutdown so hopefully nobody lost data. Everything should be back online now. I apologize for the inconvenience.

Maintenance on October 7th

* * * Maintenance Announcement * * *

Puck will be going offline on Friday, October 7th, around 10:15pm 11:30am Eastern time, to upgrade the kernel, storage configuration and supporting environment. This outage will last two to three hours due to the need to copy puck's filesystems to new drive images.

Maintenance on August 18th

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, August 18th, around 12:30pm Eastern time, to replace the physical hardware. This outage will likely last around three hours due to the need to migrate all data to new storage.

Update 8/18 3:55pm: Hardware replacement went fine, but conversion of existing VM disk images from the older format is taking quite some time. Puck may be down for another few hours.

Maintenance on August 2nd

Maintenance today was completed successfully, but it turns out one of the drives in the RAID array is starting to fail and should be replaced, so here is another...

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, August 2nd, around 10am Eastern time, to replace a failing hard drive. This outage should last for no more than 20 minutes.

Maintenance on July 31st

* * * Maintenance Announcement * * *

Puck will be going offline on Tuesday, July 31st, around 10am Eastern time, to upgrade the OS on the VM host system and relocate the physical hardware. This outage should last for no more than 60 minutes, and hopefully much less.

New IP address for Puck

Effective tomorrow, July 5th, puck's primary IPv4 address will be changed to 204.16.242.132. The current address, 204.16.242.142, will continue to be a secondary address for puck.

There will be a network drop-out of a few seconds as I take down the interface and reconfigure it. Once the interface is back up, existing TCP sessions will continue uninterrupted.

* * * Maintenance Announcement * * *

Puck will be going offline on Monday, September 27th, around 10am Eastern time, to upgrade the kernel on the VM host system and puck's VM. I expect this to take between 15 and 30 minutes, as I'm also making some changes to puck's boot environment.

Maintenance on June 23rd

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, June 23rd, around 10am Eastern time, to upgrade the OS on the VM host system. (No changes will be made to puck's VM.) This outage should last for no more than 60 minutes, and hopefully much less.

Down time schedule

I'm finally happy with the VM configuration I'll be using for puck, so tomorrow I will be transferring the filesystems to the new server. If all that seems stable, I'll be moving the server to the data center on Thursday morning. So...

* * * Maintenance Announcement * * *

Puck will be going offline on Tuesday, June 30th, around 10am Eastern time, to transfer the system image to new hardware. This outage may last for several hours depending on how long it takes to copy all the data to new filesystems on the new server.

Puck will be going offline again on Thursday, July 2nd, around 9:30am Eastern time, to move the new hardware into the new data center. This outage will last for around one hour.

Back online

Power has been restored to puck and it is currently checking quotas and will be fully online shortly.

Power outage

As a result of the thunderstorms in Pittsburgh, we lost power at our house, taking down all of the equipment in our basement including puck. Naturally this had to happen during the only time in May and June that both of us are out of town for more than 36 hours.

Once power is restored I'll have somebody visit our house and reset all the equipment, but until then, there's nothing I can do.

Puck should be moved into the new data center by the end of this month, which will eliminate any risk of outages due to power loss.

Back up

Puck is back up as of around 2am last night. Everything seems to be working, although there was a massive load average spike at first due to all the backed up email being delivered very quickly over Ethernet from our backup mail servers. :-)

The hardware upgrade and move to the new data center will take place in two phases (reboots): first I'll upgrade the hardware, which will live in my basement for a short time; once I'm comfortable that the new setup is stable, I'll move it to the new DC. I'll announce a schedule for this once the new hardware arrives.