Outage SJB right now?
On Monday, September seventh, our chief architect was alerted to a security exploit affecting hardware across our network. Because the risk was immediate, we began an emergency upgrade rather than wait for a scheduled window.
At the time, we were partway through migrating our core services — DNS, DHCP, and mail — onto a new server cluster. When our architect rebooted during the upgrade, some of those services didn’t come back up as expected on the new hardware. He rolled everything back to the previous configuration to restore service, and the migration has since been completed — we’re now running fully on the newer platform. That partially migrated state is what caused the disruption, and it no longer exists.
Service interruptions began around eleven in the morning. Most customers were restored by three thirty in the afternoon, and everything was fully restored by seven thirty that evening.
We have put new configurations, protocols, and procedures in place to strengthen resiliency and redundancy, so a failure like this one should not repeat.
Thank you for your patience, and for your business."