New IP address for Puck

Effective tomorrow, July 5th, puck's primary IPv4 address will be changed to 204.16.242.132. The current address, 204.16.242.142, will continue to be a secondary address for puck.

There will be a network drop-out of a few seconds as I take down the interface and reconfigure it. Once the interface is back up, existing TCP sessions will continue uninterrupted.

* * * Maintenance Announcement * * *

Puck will be going offline on Monday, September 27th, around 10am Eastern time, to upgrade the kernel on the VM host system and puck's VM. I expect this to take between 15 and 30 minutes, as I'm also making some changes to puck's boot environment.

Maintenance on June 23rd

* * * Maintenance Announcement * * *

Puck will be going offline on Wednesday, June 23rd, around 10am Eastern time, to upgrade the OS on the VM host system. (No changes will be made to puck's VM.) This outage should last for no more than 60 minutes, and hopefully much less.

Down time schedule

I'm finally happy with the VM configuration I'll be using for puck, so tomorrow I will be transferring the filesystems to the new server. If all that seems stable, I'll be moving the server to the data center on Thursday morning. So...

* * * Maintenance Announcement * * *

Puck will be going offline on Tuesday, June 30th, around 10am Eastern time, to transfer the system image to new hardware. This outage may last for several hours depending on how long it takes to copy all the data to new filesystems on the new server.

Puck will be going offline again on Thursday, July 2nd, around 9:30am Eastern time, to move the new hardware into the new data center. This outage will last for around one hour.

Back online

Power has been restored to puck and it is currently checking quotas and will be fully online shortly.

Power outage

As a result of the thunderstorms in Pittsburgh, we lost power at our house, taking down all of the equipment in our basement including puck. Naturally this had to happen during the only time in May and June that both of us are out of town for more than 36 hours.

Once power is restored I'll have somebody visit our house and reset all the equipment, but until then, there's nothing I can do.

Puck should be moved into the new data center by the end of this month, which will eliminate any risk of outages due to power loss.

Back up

Puck is back up as of around 2am last night. Everything seems to be working, although there was a massive load average spike at first due to all the backed up email being delivered very quickly over Ethernet from our backup mail servers. :-)

The hardware upgrade and move to the new data center will take place in two phases (reboots): first I'll upgrade the hardware, which will live in my basement for a short time; once I'm comfortable that the new setup is stable, I'll move it to the new DC. I'll announce a schedule for this once the new hardware arrives.

Moving puck

As I mentioned in my update to yesterday's post, WVT is not interested in continuing to provide us co-lo space for free. While I've been planning for a while to move puck to a data center located more convenient to me, this provided me the incentive I needed to get the job done.

Unfortunately, I can't pick up puck and move it directly to the new data center, since USAD has two-post racks and the new DC has four-post racks. Rather than spending $150 to buy new rails for the existing hardware, I'll be replacing the hardware entirely, which means I need to wait a week or so for Dell to ship me a new server.

Until I have the new hardware, puck will be living in the rack in my basement. During that time, the upstream link will be limited to 2 Mbps. As soon as the new hardware arrives, I'll move puck into the new DC.

So here's the plan:

• On Monday (May 11th) between 3pm and 4pm I'll arrive in Syracuse to shut down puck and load it in my car.
• Late Monday night when I get home, I'll install puck into the rack in my basement and bring it online. Everything will be operational except that the data rates between puck and the Internet will be limited to the speed of my cable modem.
  
• The week of the 18th, when the new hardware arrives, I'll make another announcement here of what day and time I will next shut down puck to upgrade the hardware and move it to a new data center. That should be a shorter outage.

If you have any questions, you know where to find me.

Puck and other servers hosted at USA Datanet have been down since 3pm on May 7th. I left a message with their tech support department but have not yet received a call back.

It is unclear whether this outage is related to the recent acquisition of USA Datanet's operations by WVT Communications. According to the customer service representative I spoke to, WVT has not made any decision to discontinue any existing USA Datanet services.

**Update 5/7 10:30pm EDT:

While puck is down, our backup mail servers are holding all incoming mail indefinitely. No mail will be lost during this outage.

Due to the uncertain future of USA Datanet's facilities, I will be moving puck to a new data center as soon as I can obtain access to retrieve our hardware.

**Update 5/8 9:45am EDT:

I spoke with a tech at USAD who confirmed that our equipment was disconnected and powered down as they were unable to identify the owner. (Most of the techs at USAD have quit since the bankruptcy filing, including all of the people who knew who we were.) He offered to power up the equipment for the weekend and I agreed to come take it out on Monday. Puck should be up within the next two hours.

**Update 5/8 11:45am EDT:

Puck has been back up for a few minutes. The 10,000 (!) emails queued up on our backup mail servers are slowly flowing in.

Filesystem problems

Around noon today puck started spewing reiserfs errors about missing metadata. The load average shot up and I was forced to do a controlled reboot. Currently I am running reiserfsck on all affected filesystems, and hopefully there will be no errors and I can bring puck back up as soon as possible.

I don't think any data will be missing, but just in case, our last backup was completed on the 8th, so all older data is definitely safe.

I'm quite unimpressed by ReiserFS at this point. I've used ext3 on almost every other Linux system I own, and only ReiserFS has ever failed of its own accord. Next time there is even a hint of trouble on puck I plan to reformat /home and /srv as ext3.

Puck seems to have crashed around midnight 4/22. It is currently rebooting.

Power outage

Apparently there was a 30-minute power outage at the co-lo this morning which was enough to kill the old dying battery in the UPS. Everything shut down nicely, though, and now the power has been restored and we're back up.

Brief outage on Sunday, July 11 at noon

I hate to do this so soon after the last major outage, but we will be taking puck down briefly on Sunday to add another 512MB of memory and upgrade the kernel. This should only take 10-20 minutes, barring any unexpected hardware problems.

Update 6/13: Of course, I meant June 11, not July 11 as the title says. Hope nobody was confused. The upgrade was completed without trouble.

Memorial Day Weekend outages

On a trip through Syracuse, we decided to stop by the switch room and do a little maintenance on puck. One of the new 400GB drives was reporting read errors during SMART tests, so it needed to be replaced before it started failing for real. This took 90 minutes but seemed to go OK and puck was running when we left. By the time we got back to Pittsburgh, though, puck was nonresponsive on the network and on the console port. Cycling the power didn't increase the load on the PDU, which suggested a serious hardware failure. So, yesterday we drove back to Syracuse. It turned out to be that a HDD power cable was smashed under the case lid when it was reassembled yesterday, and it took a little while for the rack vibration to wear through the insulation and finally short the +5VDC line to the chassis. Some creative use of cable ties fixed the problem and puck is now back online.

To cap things off, today when I was upgrading atd, I accidentally triggered the checkfs script which failed and automatically rebooted the system.

So, sorry about all the unexpected downtime this weekend. Soon everyone will have their own huge hunk of space on /data to make up for it.

In the near future, I will be making an email announce list to notify everyone of outages in advance, and hopefully that will bring more coordination to maintenance scheduling.

Bookmark this site so when any Litech services are down, you can find out what's going on.