This is the story of the two worst nights in the existence of VPS HiSpeed.
In the night of 27 to 28 March we experienced a power outage in one of our racks. This was caused by our fast growth in VPSs numbers. The more VPSs, the more power consumption. Every device in our racks has two power supplies. If one power outlet fails, the servers will consume double amount of power on the other outlet. This resulted in overload and a failure of the other power outlet as well. So, no power in that rack anymore. The data center was able to restore power in 50 minutes.
Shortly after that, we meet the data center staff how to solve this issue. They will solve the problem the same day, so that it will not happen again. Unfortunately the meeting took place just after an earthquake hit Bangkok. All staff in the NT tower had to exit the building. Only after NT building inspection, staff was allowed. The data center staff had to delay the power fix in our rack with at least one day.
Then in the night of 28 March 23:20, the same thing happened as the night before. Power outage in the rack again. Same reason.
The data center staff was unavailable, as no one was allowed to enter the building. So, no one was able to restore power in our racks. After calling all mobile numbers of the data center staff we had, we had no option than to wait until the next day. This was very hard to accept for me.
Then, one person called me back. There was someone who was willing to enter the building at their own risk and restore power again. After 3 hours almost all servers were up again. Even the load of the power was shared over more power groups, to prevent this from happening again.
We are very grateful for the work of this person.
Right now (29 March), we are still fixing issues with some severs that was caused by the power outage. Two servers lost a part their network configuration, and one server has a defective disk controller. We are still fixing these issues with our whole team.
We are truly sorry for all inconvenience it causes. Please understand that we are doing really everything to fix all issues as soon as possible, as your problem is our problem.
If you have any questions, then please don't hesitate to contact us.
Best regards,
Diederik
Managing Director
VPS HiSpeed / Cloud Hosting Co.,Ltd.
Saturday, March 29, 2025