We want to thank each and every one of our clients for their patience and understanding during this incident. The storms hurt us badly, and we will incur substantial costs for many more months as a result, but we worked through two very dark and cold nights to get back online after this hardship, and we have persevered.
We even took the time in the pitch black, silent datacenter, to perform some hardware upgrade tickets by flashlight, and move forward on some incremental facility upgrades that would have otherwise required their own outage to complete.
Despite the unexpected disruption, we're thankful to have had very few cancellations, and we want to thank each of you for your commitment with a 5% credit to your monthly bill. We believe this is the right thing to do.
We believe the longest service disruptions experienced by any client were around 30 hours from start to finish. 5% of one month is about a day and a half. It may take a little longer to credit those of you on terms longer than a month, but we will. You can expect that your next bill will be a little cheaper than usual because of this. We will also be providing remote hands and KVMoIP and/or IPMI where possible for free to those affected by the power outage if you need it to recover unsaved network changes or filesystem corruption. We will do everything we can now to support you. We have personnel with experience in data recovery should you need it as well, for free, in connection with the Severe Wind Event of 2021.
2021-12-17 0300 Central Time Update:
We have completed verifying all servers operational that had active tickets about the outage. We will continue tomorrow verifying every server reported as down at any time during the course of this outage is responding to the network again.
2021-12-17 0120 Central Time Update
Power and cooling has been restored now to all racks. Sorry this was a little later than expected. We will be checking all client systems shortly.
2021-12-16 1700 Central Time Update:
We have restored function of our generator now. The UPS is charging up. Power and AC is coming online row by row. Thank you for your patience as we check all services are online. This verification will continue through the night. If you are not online by midnight Central Time, please update your ticket. And as a reminder from earlier, if you have configured your server or servers to NOT respond to ping, please respond on your ticket to note this and tell us how we may verify your server is online most effectively over the network. .... now back to work....
2021-12-16 1000 Update:
Work continues throughout our city to restore electrical service as does our work to restore generator operation. We have an ETA of the generator of 1700 central time from the machine shop that will fabricate replacement parts. We are monitoring the condition of the electrical utility power closely. We detected erroneously out-of-spec power on one phase yesterday, and we isolated all client machines from the electrical utility to protect your equipment. It's bad enough that you're offline, but we will not take any chance of damage to your equipment. So electrical utility power can not be utilized until the utility is supplying it safely. The utility, Evergy Energy has not provided an ETA at this time.
As soon as one of (1) the electrical utility or (2) our generator is restored, your service will resume automatically. We will review every server of every client after electrical service is back online to make sure your systems are responding. PLEASE make a note in your ticket now if your servers are configured to never respond to network traffic, or if they do not listen to standard ports. If you tell us in your ticket how to verify your server is responding as you expect, this will expedite our response, which we very much intend to do.
Also we would like to provide the following in case your clients may be asking:
- No. We have not been hacked. There is no known security breach at this time. We are keeping personnel on-site to physically secure your hardware while we await restoration of power.
- No. Your data is not lost. It is inaccessible and we are working to restore it ASAP. We will provide KVMoIP and IPMI access where possible after restoration of power, to give you control of any recovery process if you need it.
- Natural Disaster was the cause of this outage. Tornadoes are not uncommon in our area, but the disaster yesterday was "straight line wind" which has never been this powerful in our city. It has damaged much of the central continental United States.
- No employees were hurt during the disaster or response.
Original 2021-12-15 announcement: 2021-12-15 19 to 2021-12-17 01
Greetings UnReal friends, it has been an unreal day indeed!
Around 1800 Central time Wednesday, December 15th, 2021, catastrophic wind storms struck our home in the Kansas City area. Wind as fast as 95 miles (150 km) per hour has leveled buildings, thrown trucks through buildings, and even blown 15 tonne generators over. Many have died. Please keep our city in your thoughts and prayers as families mourn.
Over 200,000 electrical customers are without service this evening with one of our power companies, in a city of only 250,000 homes.
UnReal Servers has also experienced damage to our generators as you are have no doubt noticed.
None of our employees were injured during this storm or our response to it, and we're going to keep it that way.
At 1935 Central time, electrical utility power was lost to our building and we went on generator. All staff began in transit to the datacenter.
Repair technicians have been working since 2000 central time in the cold and in the dark, on restoring generator service, and on restoring electrical utility (grid) power to our building, and also to the wider area. Hospitals are receiving priority by the utility.
Thank you for your continued patience as we work through this disaster. Instead of responding to individual tickets, we will be focusing our time to recover service as soon as possible. So stay tuned to these posts for updates.
Thursday, December 16, 2021