Maintenance on October 10th

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, October 10th, around 2:00pm Eastern time, to upgrade the kernel and supporting environment. This outage should last 45-60 minutes.

Maintenance on May 10th

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, May 10th, around 10:00am Eastern time, to upgrade the kernel and supporting environment. This outage should last 45-60 minutes.

Maintenance on February 21st

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, February 21st, around 10:00am Eastern time, to upgrade the kernel and supporting environment. This outage should last 30-45 minutes at the most.

Unplanned outage

Puck was shut down unannounced this afternoon due to failing power distribution equipment at the data center.  It was a controlled shutdown so hopefully nobody lost data. Everything should be back online now. I apologize for the inconvenience.

Maintenance on October 7th

* * * Maintenance Announcement * * *

Puck will be going offline on Friday, October 7th, around 10:15pm 11:30am Eastern time, to upgrade the kernel, storage configuration and supporting environment. This outage will last two to three hours due to the need to copy puck's filesystems to new drive images.

Maintenance on August 18th

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, August 18th, around 12:30pm Eastern time, to replace the physical hardware. This outage will likely last around three hours due to the need to migrate all data to new storage.

Update 8/18 3:55pm: Hardware replacement went fine, but conversion of existing VM disk images from the older format is taking quite some time. Puck may be down for another few hours.

Maintenance on August 2nd

Maintenance today was completed successfully, but it turns out one of the drives in the RAID array is starting to fail and should be replaced, so here is another...

* * * Maintenance Announcement * * *

Puck will be going offline on Thursday, August 2nd, around 10am Eastern time, to replace a failing hard drive. This outage should last for no more than 20 minutes.

Maintenance on July 31st

* * * Maintenance Announcement * * *

Puck will be going offline on Tuesday, July 31st, around 10am Eastern time, to upgrade the OS on the VM host system and relocate the physical hardware. This outage should last for no more than 60 minutes, and hopefully much less.

New IP address for Puck

Effective tomorrow, July 5th, puck's primary IPv4 address will be changed to 204.16.242.132. The current address, 204.16.242.142, will continue to be a secondary address for puck.

There will be a network drop-out of a few seconds as I take down the interface and reconfigure it. Once the interface is back up, existing TCP sessions will continue uninterrupted.

* * * Maintenance Announcement * * *

Puck will be going offline on Monday, September 27th, around 10am Eastern time, to upgrade the kernel on the VM host system and puck's VM. I expect this to take between 15 and 30 minutes, as I'm also making some changes to puck's boot environment.

Maintenance on June 23rd

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, June 23rd, around 10am Eastern time, to upgrade the OS on the VM host system. (No changes will be made to puck's VM.) This outage should last for no more than 60 minutes, and hopefully much less.

Down time schedule

I'm finally happy with the VM configuration I'll be using for puck, so tomorrow I will be transferring the filesystems to the new server. If all that seems stable, I'll be moving the server to the data center on Thursday morning. So...

* * * Maintenance Announcement * * *

Puck will be going offline on Tuesday, June 30th, around 10am Eastern time, to transfer the system image to new hardware. This outage may last for several hours depending on how long it takes to copy all the data to new filesystems on the new server.

Puck will be going offline again on Thursday, July 2nd, around 9:30am Eastern time, to move the new hardware into the new data center. This outage will last for around one hour.

Back online

Power has been restored to puck and it is currently checking quotas and will be fully online shortly.

Power outage

As a result of the thunderstorms in Pittsburgh, we lost power at our house, taking down all of the equipment in our basement including puck. Naturally this had to happen during the only time in May and June that both of us are out of town for more than 36 hours.

Once power is restored I'll have somebody visit our house and reset all the equipment, but until then, there's nothing I can do.

Puck should be moved into the new data center by the end of this month, which will eliminate any risk of outages due to power loss.

Back up

Puck is back up as of around 2am last night. Everything seems to be working, although there was a massive load average spike at first due to all the backed up email being delivered very quickly over Ethernet from our backup mail servers. :-)

The hardware upgrade and move to the new data center will take place in two phases (reboots): first I'll upgrade the hardware, which will live in my basement for a short time; once I'm comfortable that the new setup is stable, I'll move it to the new DC. I'll announce a schedule for this once the new hardware arrives.

Moving puck

As I mentioned in my update to yesterday's post, WVT is not interested in continuing to provide us co-lo space for free. While I've been planning for a while to move puck to a data center located more convenient to me, this provided me the incentive I needed to get the job done.

Unfortunately, I can't pick up puck and move it directly to the new data center, since USAD has two-post racks and the new DC has four-post racks. Rather than spending $150 to buy new rails for the existing hardware, I'll be replacing the hardware entirely, which means I need to wait a week or so for Dell to ship me a new server.

Until I have the new hardware, puck will be living in the rack in my basement. During that time, the upstream link will be limited to 2 Mbps. As soon as the new hardware arrives, I'll move puck into the new DC.

So here's the plan:

• On Monday (May 11th) between 3pm and 4pm I'll arrive in Syracuse to shut down puck and load it in my car.
• Late Monday night when I get home, I'll install puck into the rack in my basement and bring it online. Everything will be operational except that the data rates between puck and the Internet will be limited to the speed of my cable modem.
  
• The week of the 18th, when the new hardware arrives, I'll make another announcement here of what day and time I will next shut down puck to upgrade the hardware and move it to a new data center. That should be a shorter outage.

If you have any questions, you know where to find me.

Puck and other servers hosted at USA Datanet have been down since 3pm on May 7th. I left a message with their tech support department but have not yet received a call back.

It is unclear whether this outage is related to the recent acquisition of USA Datanet's operations by WVT Communications. According to the customer service representative I spoke to, WVT has not made any decision to discontinue any existing USA Datanet services.

**Update 5/7 10:30pm EDT:

While puck is down, our backup mail servers are holding all incoming mail indefinitely. No mail will be lost during this outage.

Due to the uncertain future of USA Datanet's facilities, I will be moving puck to a new data center as soon as I can obtain access to retrieve our hardware.

**Update 5/8 9:45am EDT:

I spoke with a tech at USAD who confirmed that our equipment was disconnected and powered down as they were unable to identify the owner. (Most of the techs at USAD have quit since the bankruptcy filing, including all of the people who knew who we were.) He offered to power up the equipment for the weekend and I agreed to come take it out on Monday. Puck should be up within the next two hours.

**Update 5/8 11:45am EDT:

Puck has been back up for a few minutes. The 10,000 (!) emails queued up on our backup mail servers are slowly flowing in.

Filesystem problems

Around noon today puck started spewing reiserfs errors about missing metadata. The load average shot up and I was forced to do a controlled reboot. Currently I am running reiserfsck on all affected filesystems, and hopefully there will be no errors and I can bring puck back up as soon as possible.

I don't think any data will be missing, but just in case, our last backup was completed on the 8th, so all older data is definitely safe.

I'm quite unimpressed by ReiserFS at this point. I've used ext3 on almost every other Linux system I own, and only ReiserFS has ever failed of its own accord. Next time there is even a hint of trouble on puck I plan to reformat /home and /srv as ext3.

Puck seems to have crashed around midnight 4/22. It is currently rebooting.

Power outage

Apparently there was a 30-minute power outage at the co-lo this morning which was enough to kill the old dying battery in the UPS. Everything shut down nicely, though, and now the power has been restored and we're back up.