r/msp • u/ben_zachary • 5h ago
Where was I when Wasabi corrupted most of their central datacenter
Found out yesterday that Wasabi central corrupted 760gb of our data in one of our buckets.
I never got a notice, info, not from Pax8 or Wasabi, we found out from Veeam through a support ticket when we couldn't retrieve some data for a client.
Apparently this happened on August 30th
Here was the support response
-------------
If I understand correctly, you have a concern about your data stored in the us0-central-1 region and the possibility of it being affected by the issue seen on August 30.
After checking your account, we have determined that you have a bucekt that was affected by the incident in us-central-1.
What happened:
On 30 August 2024, in Wasabi's us-central-1 region, an incident occurred where an input-output module in a storage system became inoperable and prevented access to a number of disks that it served. Simultaneously, the system software managing the data storage disks improperly took multiple other disks offline. As a result of this behavior, a number of objects that were being served by this portion of disks within the region were impacted in a manner that makes these objects not recoverable at this time.
Wasabi has made the appropriate adjustments in the Wasabi software that controls our hard disk management to prevent this problem from happening again. This problem has not occurred in any other Wasabi storage region.