Emergency maintenance on March 24th

  * * * Maintenance Announcement * * *

Due to a botched kernel livepatch (thanks Ubuntu), puck will be going offline on Wednesday, March 24th, around 3:00pm Eastern time. There is no reason to believe there will be any data integrity issues, but systemd is hung and it may eventually affect service availability without a reboot.

During this outage, the kernel and supporting environment will be updated, and an fsck will be run.

This outage should last 45-90 minutes.

Maintenance on September 23rd

 * * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, September 23rd, around 9:30am Eastern time, to upgrade the kernel and supporting environment. This outage should last 45-90 minutes.

Maintenance on October 30th

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, October 30th, around 9:30am to 10:00am Eastern time, while the hardware is moved to another data center. This will likely take one to two hours.

The new facility is part of the same infrastructure company, so no network configuration changes are necessary. All IP addresses, etc, will stay the same.

Maintenance on October 10th

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, October 10th, around 2:00pm Eastern time, to upgrade the kernel and supporting environment. This outage should last 45-60 minutes.

Maintenance on May 10th

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, May 10th, around 10:00am Eastern time, to upgrade the kernel and supporting environment. This outage should last 45-60 minutes.

Maintenance on February 21st

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, February 21st, around 10:00am Eastern time, to upgrade the kernel and supporting environment. This outage should last 30-45 minutes at the most.

Unplanned outage

Puck was shut down unannounced this afternoon due to failing power distribution equipment at the data center.  It was a controlled shutdown so hopefully nobody lost data. Everything should be back online now. I apologize for the inconvenience.

Maintenance on October 7th

* * * Maintenance Announcement * * *

Puck will be going offline on Friday, October 7th, around 10:15pm 11:30am Eastern time, to upgrade the kernel, storage configuration and supporting environment. This outage will last two to three hours due to the need to copy puck's filesystems to new drive images.

Maintenance on August 18th

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, August 18th, around 12:30pm Eastern time, to replace the physical hardware. This outage will likely last around three hours due to the need to migrate all data to new storage.

Update 8/18 3:55pm: Hardware replacement went fine, but conversion of existing VM disk images from the older format is taking quite some time. Puck may be down for another few hours.

Maintenance on August 2nd

Maintenance today was completed successfully, but it turns out one of the drives in the RAID array is starting to fail and should be replaced, so here is another...

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, August 2nd, around 10am Eastern time, to replace a failing hard drive. This outage should last for no more than 20 minutes.

Maintenance on July 31st

* * * Maintenance Announcement * * *

Puck will be going offline on Tuesday, July 31st, around 10am Eastern time, to upgrade the OS on the VM host system and relocate the physical hardware. This outage should last for no more than 60 minutes, and hopefully much less.

New IP address for Puck

Effective tomorrow, July 5th, puck's primary IPv4 address will be changed to 204.16.242.132. The current address, 204.16.242.142, will continue to be a secondary address for puck.

There will be a network drop-out of a few seconds as I take down the interface and reconfigure it. Once the interface is back up, existing TCP sessions will continue uninterrupted.

* * * Maintenance Announcement * * *

Puck will be going offline on Monday, September 27th, around 10am Eastern time, to upgrade the kernel on the VM host system and puck's VM. I expect this to take between 15 and 30 minutes, as I'm also making some changes to puck's boot environment.

Maintenance on June 23rd

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, June 23rd, around 10am Eastern time, to upgrade the OS on the VM host system. (No changes will be made to puck's VM.) This outage should last for no more than 60 minutes, and hopefully much less.

Down time schedule

I'm finally happy with the VM configuration I'll be using for puck, so tomorrow I will be transferring the filesystems to the new server. If all that seems stable, I'll be moving the server to the data center on Thursday morning. So...

* * * Maintenance Announcement * * *

Puck will be going offline on Tuesday, June 30th, around 10am Eastern time, to transfer the system image to new hardware. This outage may last for several hours depending on how long it takes to copy all the data to new filesystems on the new server.

Puck will be going offline again on Thursday, July 2nd, around 9:30am Eastern time, to move the new hardware into the new data center. This outage will last for around one hour.

Back online

Power has been restored to puck and it is currently checking quotas and will be fully online shortly.

Power outage

As a result of the thunderstorms in Pittsburgh, we lost power at our house, taking down all of the equipment in our basement including puck. Naturally this had to happen during the only time in May and June that both of us are out of town for more than 36 hours.

Once power is restored I'll have somebody visit our house and reset all the equipment, but until then, there's nothing I can do.

Puck should be moved into the new data center by the end of this month, which will eliminate any risk of outages due to power loss.

Back up

Puck is back up as of around 2am last night. Everything seems to be working, although there was a massive load average spike at first due to all the backed up email being delivered very quickly over Ethernet from our backup mail servers. :-)

The hardware upgrade and move to the new data center will take place in two phases (reboots): first I'll upgrade the hardware, which will live in my basement for a short time; once I'm comfortable that the new setup is stable, I'll move it to the new DC. I'll announce a schedule for this once the new hardware arrives.

Moving puck

As I mentioned in my update to yesterday's post, WVT is not interested in continuing to provide us co-lo space for free. While I've been planning for a while to move puck to a data center located more convenient to me, this provided me the incentive I needed to get the job done.

Unfortunately, I can't pick up puck and move it directly to the new data center, since USAD has two-post racks and the new DC has four-post racks. Rather than spending $150 to buy new rails for the existing hardware, I'll be replacing the hardware entirely, which means I need to wait a week or so for Dell to ship me a new server.

Until I have the new hardware, puck will be living in the rack in my basement. During that time, the upstream link will be limited to 2 Mbps. As soon as the new hardware arrives, I'll move puck into the new DC.

So here's the plan:

• On Monday (May 11th) between 3pm and 4pm I'll arrive in Syracuse to shut down puck and load it in my car.
• Late Monday night when I get home, I'll install puck into the rack in my basement and bring it online. Everything will be operational except that the data rates between puck and the Internet will be limited to the speed of my cable modem.
  
• The week of the 18th, when the new hardware arrives, I'll make another announcement here of what day and time I will next shut down puck to upgrade the hardware and move it to a new data center. That should be a shorter outage.

If you have any questions, you know where to find me.

Puck and other servers hosted at USA Datanet have been down since 3pm on May 7th. I left a message with their tech support department but have not yet received a call back.

It is unclear whether this outage is related to the recent acquisition of USA Datanet's operations by WVT Communications. According to the customer service representative I spoke to, WVT has not made any decision to discontinue any existing USA Datanet services.

**Update 5/7 10:30pm EDT:

While puck is down, our backup mail servers are holding all incoming mail indefinitely. No mail will be lost during this outage.

Due to the uncertain future of USA Datanet's facilities, I will be moving puck to a new data center as soon as I can obtain access to retrieve our hardware.

**Update 5/8 9:45am EDT:

I spoke with a tech at USAD who confirmed that our equipment was disconnected and powered down as they were unable to identify the owner. (Most of the techs at USAD have quit since the bankruptcy filing, including all of the people who knew who we were.) He offered to power up the equipment for the weekend and I agreed to come take it out on Monday. Puck should be up within the next two hours.

**Update 5/8 11:45am EDT:

Puck has been back up for a few minutes. The 10,000 (!) emails queued up on our backup mail servers are slowly flowing in.