Photo

Emergency Maintenance :: Cloud SAN :: 01/20/12


  • Please log in to reply
4 replies to this topic

#1 -ASO- Ryan

-ASO- Ryan

    Technical Support Manager

  • Admin
  • 70 posts

Posted 20 January 2012 - 05:24 PM

We are currently experiencing intermittent issues with the SAN at the datacenter where the cloud hypervisors are located. This is causing some cloud package accounts to appear down or unable to start. The datacenter is looking in to this issue currently and we will have it resolved as soon as possible. We will update this thread once this has been resolved. Thank you for your patience.
  • 0

#2 -ASO- Ryan

-ASO- Ryan

    Technical Support Manager

  • Admin
  • 70 posts

Posted 20 January 2012 - 07:22 PM

The datacenter is still checking in to this issue and we are awaiting updates. We will be posting any information we receive as it's available in this thread. We thank you again for your patience.
  • 0

#3 -ASO- Jen

-ASO- Jen

    Director of Customer Service

  • Admin
  • 131 posts

Posted 21 January 2012 - 04:53 AM

I want to apologize for the lack of updates on this issue - it has taken us a while to fully look at the situation before we had any information to release, much longer than would have hoped.

The cloud storage configuration, which is made up of a 16 disk SAN array, can tolerate one drive failure and continue to operate normally. This automatic failover provides the reliability that has made cloud platforms popular. In last night’s incident, however, the SAN array had two drive failures, which is beyond the tolerance fault level of the system.

Currently, attempts are being made to rebuild the array and regain access to the data, but this is a tedious process due to the nature of the failure and the methods of recovery. At this time, we have no definitive ETA for when this will be completed.

We will update you as soon as we have more definitive information regarding the state of the drives and the array.
  • 0
Jen Lepp
Director of Customer Service
A Small Orange LLC
http://www.asmallorange.com

#4 -ASO- Jen

-ASO- Jen

    Director of Customer Service

  • Admin
  • 131 posts

Posted 21 January 2012 - 11:28 AM

All those affected by this issue should have an email describing the situation - if you are affected and did not get the email, please email the help desk and we'll be happy to send you a copy.

Thanks, everyone, for your patience.
  • 1
Jen Lepp
Director of Customer Service
A Small Orange LLC
http://www.asmallorange.com

#5 -ASO- Jen

-ASO- Jen

    Director of Customer Service

  • Admin
  • 131 posts

Posted 23 January 2012 - 09:23 PM

Many of you have received an individual email by now explaining that nearly all of the data on our original cloud platform was recoverable, and there were only a few instances that could ultimately not be recovered. Those that have not gotten an email should be getting one tomorrow as we get closer to your turn up time in the queue. This is great news and we were very excited to let our customers know about this.

However, nothing is ever easy:

We attempted to get all overwrites and instances overwritten or turned up as quickly as possible, but the activity of copying three virtual machines of data onto the storage server simultaneously, along with the mad rush of our users to copy files has elevated the I/O load on the new cloud far beyond its normal expected activity. Due to the elevated I/O, we are restoring instances far more slowly than previously in order to keep the I/O load to a minimum and the service more stable for those already on the cloud.

We will continue working to get all available instances restored as quickly as possible and, again, thank you for your patience.
  • 0
Jen Lepp
Director of Customer Service
A Small Orange LLC
http://www.asmallorange.com




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users