Data Center Updates

**UPDATE 8/8/18 - 5:15 p.m.**

We have decided to proceed as if the data recovery process will not yield any additional data. The recovery company has told us they will continue to work on the drives for a few more weeks but the remaining options have low probability of success. This means we will be proceeding with the plans we've outlined in previous updates. First, we will be moving our data out of the existing data center as soon as practical. We have created our own physical backups of the current data. We will also implement our own ongoing backup routine, in addition to the backups our new data center will provide. We will also continue helping existing clients (at no cost) reconstruct pages and functionality that have changed since October 2017. For most of you who have sent us your input on top priority updates/fixes, we have provided timelines for completion if the work hasn't already been completed. If you haven't provided input, please forward that to support@creativestate.com or create a ticket at http://support.creativestate.com/ For those clients who experienced significant data loss, we will be providing web hosting at no cost for a minimum of 12 months. As always, you can always contact me directly at brent@creativestate.com

**UPDATE August 3, 2018 - 1:45 p.m.**

We will have a final word from the data recovery firm by Tuesday, August 7. If they recover additional data, we will begin restoring that data immediately. If they do not recover additional data, we will continue to work with individual clients to bring their sites current. If you have questions, you can email directly at brent@creativestate.com or if you have immediate changes that need to be made, please email those to support@creativestate.com (this email address creates a ticket and notifies our entire team of the request).

**UPDATE July 31, 2018 - 2:15 p.m.**

Today is day 7 of the 7-10 Business Day window from the second data recovery firm. They sent an update today stating that Step 1 in their process was not successful in recovering the data. They begin Step 2 (the final step) today. They have not given us any reason to be more or less optimistic than before. They indicated that the fact that Step 1 was not successful did not mean that it's less likely for Step 2 to be successful. It's two different processes. We will continue to update you as we get more information from them. Although we won't know definitively until we can be more aggressive in questioning the data center, our current belief is that instead of storing our backups in a second secure location (which is what paid the data center to do), our live data and all backups were stored in the same physical location and that something occurred in that location that caused all of our hardware to sustain physical damage. Once we recover as much data as possible, we will be doing a number of things. First, we will immediately move all client data to a new data center. We will physically inspect the facility and our equipment on a regular basis. Second, in addition to the backup services the new data center will provide, we will maintain our own physical copies of the backups and provide those to individual clients upon request. We will provide additional details as we move forward. As mentioned in a previous post, if they are not able to recover additional data, our team will work with clients at no cost to assist in reconstructing content/data that was added to sites after Oct 2017. As we have mentioned to some of you individually, in many cases we have elements of the missing/outdated pages on our internal network and in email or other forms of electronic communication. This will make the process of reconstructing some pages go much quicker than if we were starting from scratch. We will also be waiving the web hosting fee for the foreseeable future for existing clients. If you have additional questions or immediate needs on your current site, please email support@creativestate.com

**UPDATE July 26, 2018 - 4:00 p.m.**

The second data recovery company is still working on our drives. As we mentioned in a previous update, their process takes 7-10 days. We are in Day 4 of that process. I spoke with them yesterday and they are still optimistic that they will be able to recover some or all of the data. We will continue to update this space as new information becomes available. If you have an immediate need on your current site, email support@creativestate.com

**UPDATE July 23, 2018 - 1:45 p.m.**

The second data recovery company has done an initial analysis and will begin working on the actual data recovery today. They listed the same physical damage as the previous company so there was no surprise there. They also told us that there is a chance that they will recover the data. Unlike the other company, they do not give a probability as a percentage (i.e. 80% likelihood of recovery). In some cases after their initial analysis they simply say that data recovery is not possible. The fact that they believe there is a chance of data recovery with our drive is encouraging. We will continue to update this space as we have more information. Thank you for your patience.

**UPDATE July 19, 2018 - 4:30 p.m.**

The drives have been delivered by courier to the second data recovery firm. We have paid an expedite fee for the diagnostic phase. We are hoping to have some initial diagnostics by Monday at the latest. We have also spent the day working with customers to meet their immediate website needs. See the previous post for instructions for submitting the most urgent items that need to be addressed on your website. We will update this space as we receive additional information.

**UPDATE July 18, 2018 - 10:15 p.m.**

This is an update I hoped (and prayed) not to have to share with you. I'll start with the summary and then provide as much detail as possible. The data recovery firm in Los Angeles has not been able to recover any additional data after October 2017. They have recommended one final option that we intend to explore. The firm they suggest specializes in the most extreme cases of data loss, drive damage and corruption. This is a costly process and can be time consuming, but we want you to know that we do not plan to quit until we either recover all the remaining data or are told there are no additional options. This firm estimates 3-10 business days from the time they receive the drives. They are also in Los Angeles so we intend to send a courier to pick-up and hand deliver these drives tomorrow morning. However, it could be the first week of August before we have a resolution from them.

We are confident that all of our data backup and security policies and processes meet or exceed industry best practices. However, as with any major incident, we will analyze what happened and take measures to avoid it in the future. We have attempted to avoid pointing fingers publicly, but when this incident is resolved we will have many questions for our data center. Virtually every company similar to Creative State pays for a variety of services from a data center including backups. Unfortunately, when we actually needed to do a full restoration, our data center represented to us that both our backups AND our current hard drives had corruption and/or damage that would not allow them to be restored properly. This is highly unusual. In fact, I’ve never heard of this happening. They suggested we send them to a data recovery firm which we did. The firm they recommended was not able to recover our data. They indicated that the drives had bad heads, platter damage and a bad spindle and/or motor.

The data recovery firm (a separate company from the data center) shared with us that the physical damage to the drives they received is consistent with an entire server rack being damaged or knocked over. This could explain the widespread damage to all of our equipment. There will be a time for a full review of the data center and what led up to this incident.

We also understand that you want to know what short and long-term options are available. Here are several options to consider.

SHORT TERM:

Wait until the second data recovery effort is complete (3-10 business days)
We know this is not an ideal option for anyone. However, we will work with you at no cost to make any stop-gap changes to your site to get through this period of time. In the event that the data recovery effort is successful, your site will be restored to the state it was in on July 10, 2018.

LONG TERM OPTIONS ASSUMING NO FURTHER DATA RECOVERY:

Re-create pages and content that have changed since October 2017
If the second data recovery firm is unable to recover additional data, the Creative State team will work with you at no cost to get your site current. For smaller sites that don't change often, this can be accomplished fairly easily. Larger sites will be more time consuming and will require more cooperation between our team and yours. We will work to remove or turn off outdated content that would be confusing to your customers/stakeholders and add pages and content with up-to-date information. Due to the number of sites we currently host, we would ask you to send an email to info@creativestate.com with the subject line "WEBSITE MODIFICATIONS - www.yoursitename.com" In the email, please provide details on the most urgent items that need to be addressed and the best contact information for the person in your organization that will be our point of contact for this effort. We will reach out to you to get more details if necessary.

Move to another hosting provider
While we do not want to lose you as a customer, we understand the seriousness of this incident. We can provide you with the files related to your site including a backup of the website and database from October 2017 (NOTE: If you choose to stay with Creative State but would still like a physical backup of your files, please let us know and we can provide this to you.)

On a personal note, I appreciate your patience during this process. We host a large number of sites and, without exception, everyone has been very respectful during this difficult process. Everyone is clearly concerned and anxious for this incident to be resolved but we appreciate the professionalism you've exhibited. As I mentioned in one of the first updates, we have been in business over 18 years and have never had an incident like this. As a company, we have tried to be open, honest and transparent throughout the process. I appreciate all of the kind notes and prayers you have offered. This has obviously been a very difficult period. We remain hopeful that the data recovery efforts will be successful and we will continue to update this space as new information becomes available.

Feel free to email me directly at brent@creativestate.com

Sincerely,
Brent Lollis, President
Creative State

**UPDATE 7/18/2018 - 7:45 a.m.**

We are still in the service window for the data recovery company. We expect to receive a status update from them this morning. They are in LA so there is a two hour time difference. On another note, now is the time we would normally prepare and send web hosting invoices for the 3rd quarter. We have made the decision to cancel those invoices altogether. Every customer will receive 3 months of hosting at no cost. If you've paid annually, an additional 3 months will be added to your account. As we've mentioned in previous updates, we sincerely apologize for the inconvenience. While the problem originated within the data center we contract with, ultimately you are our customer and we take responsibility for the problem. We will do everything within our power to make it right. We will provide another update as soon as we have additional information to share.

**UPDATE 7/17/18 - 1:15 p.m.**

We've tried to provide updates at least twice a day during this incident. We are still working through the file level restore process and we are still in the service window for the data recovery company. We expect to have some more details on both fronts by the close of business today and will provide additional updates as new information becomes available.

**UPDATE 7/16/18 - 2:00 p.m.**

We just received word from the data recovery firm that based on their initial analysis, we have an 80% probability of full recovery of the data on our drives. Obviously, everyone would like that number to be 100% probability, but they cannot provide a 100% probability until they actually begin the recovery which is the next step in the process.

**UPDATE 7/15/18 11:00 a.m.**

In the effort to keep everyone informed on a timely basis, here is where we currently stand. The failed drives were delivered to the data recovery firm first thing this morning. We have requested their most expedited service which is 24-72 hours. We are also in the process of attempting the file level restore mentioned in the previous update. It is posible that this process could produce positive results quicker than the data recovery firm. This work is ongoing and we will provide another update as soon as we have additional information.

**UPDATE 7/15/18 9:15 p.m.**

As with everything we've experienced during this incident, restoring our customer sites from the October 2017 state to the July 10, 2018 state has proven more complicated and time consuming than anticipated. We are working the problem from two angles. The first is a file-level restore from the original drives that failed. We are also sending some of the drives to a data recovery company in Los Angeles. We are hopeful that we will be able to recover the data with one or both of these methods. We'd like to also answer some questions you've asked over the last few days:

1) Can we add current data to our site? The answer is YES. You can update your content through the CMS. (Remember, you are essentially dealing with your site as it existed in October) When the new data comes online we will work with you to merge the new data you've added with the data from July 10th. Please help us in this process by keeping notes regarding which areas of your site you've updated. This will reduce the amount of time it takes to merge the data. We would also encourage you to only make updates to the areas of the site that provide critical/time sensitive information (news updates, calendars, etc...) or areas where the outdated content would be confusing to your customers.

2) Would it be possible to take the site with the old data down and put a notice that the site will be back online soon? YES. If you would like us to do this, please email info@creativestate.com

3) Can we point our domain to another hosting provider temporarily? YES. If you have an old site or would like to point your URL to another provider until this incident is fully resolved, please email us at info@creativestate.com to coordinate.

If you have other questions, feel free to reach out. We will update this space as new information becomes available.

**UPDATE 7/15/18 9:15 a.m.**

As of this morning we have 95% of our sites back up and running. We are working to restore the data to the most recent version. We will keep you posted as that progresses.

**UPDATE 7/14/18 6:50 p.m.**

We continue to make slow but steady progress. The data center believes they have found a workaround that should significantly speed up the process. They are testing that now. If that works as planned and we don’t encounter any additional unexpected obstacles, we hope to have all clients up running no later that Monday morning. As I’ve mentioned, once the issue is fully resolved we will be reaching out to all clients regarding the credit we will be applying to your account and the measures we are taking to prevent something like this from ever happening again. Thank you for your patience.

**UPDATE 7/13/18 9:45 p.m.**

We have brought approximately 1/3 of our client sites back online and restored them to their Oct. 2017 state. The others are in the process of being brought online now. We will then begin the process of restoring each site to the Jul 10, 2018 state. We will test each site as we do this. We will continue to work around the clock until this issue is fully resolved. We will post updates to this space as they become available. Keep in mind that until your site is restored to its Jul 10 state, any changes you make will be temporary. We appreciate your patience and the encouraging messages many of you have sent our team.

**UPDATE 7/13/18 - 12:05 p.m.**

In an effort to keep you posted on the current status, we have begun to bring the first batch of sites online. We have multiple servers that must be brought online individually. As with everything we've experienced in this incident, it's taking longer than we had hoped and expected. We have people in Tulsa, Chicago, Dallas and LA working to get the issue resolved. We are still making steady progress and are working to get everything fully restored as quickly as possible.

**UPDATE 7/13/18 - 4:15 a.m.**

Our team and the team in the data center have been working through the night. As a temporary measure, we are restoring sites to their October 2017 status which should be complete in the next 4-6 hours. This is the most recent date that we had a FULL/clean backup with no errors or corruption. The data center conducted periodic tests on our backups and did not receive any indication that there were underlying problems. However, when they attempted to do a full restoration to a production environment, there were some issues. The data center technicians seem confident that they will be able to restore all sites back to their July 10 state within the next 24 hours. We chose to restore using the October 2017 data in an effort to as many customer websites back up and running as soon as possible. PLEASE NOTE: Any changes you make via the Content Management System will either not save or be overwritten when we restore from the July 10 backup. We will update this space as soon as the sites are fully restored to their most recent version and notify you that you can begin updating the sites as normal. Once again, we apologize for the inconvenience. When this issue is resolved we will be reaching out to all clients. We will do everything we can to make this right and regain your trust.

Creative State Customers -

As you know, we have experienced a major downtime event in our primary web hosting environment. This is - by a large margin - the longest unplanned downtime we have experienced in our 18 year history.

I will provide some technical details of what caused the issue. But first, I would like to personally apologize for the inconvenience this downtime has caused. I am keenly aware that many of us rely on our websites to be the digital "face" of our business. We will be applying a credit to the accounts of every customer who has been impacted.

Our client sites are hosted in a private cloud environment within a Class A data center with industry-leading security, scalability and redundancy. Like many similar companies, we do all of our planned maintenance in the overnight hours to minimize the impact on our customers. On Wednesday, July 11 at 10:00 p.m. Central, we had planned to swap a drive that had shown early signs indicating it needed to be replaced. This is not uncommon and because of the redundancy that is built into our environment, this usually requires less than 5 minutes to complete.

Unfortunately, this set off an extremely unlikely and unfortunate series of events. When the Data Center technician attempted to reboot the servers after swapping the drive, he informed us that it looked like a boot drive had also failed. This took several hours to diagnose and replace. The environment still would not come back online as expected. The technicians spent the ensuing 20+ hours removing the servers from the racks, testing, troubleshooting, etc... They were not able to come to a successful resolution. Ultimately they were forced to replace all of our hardware and rebuild our environment. This included the need to restore from one of our backups. As of 11:00 p.m. Central on Thursday, July 12 this process is still underway.

Again, I apologize for the inconvenience. If you have any additional questions, please feel free to email me at brent@creativestate.com

We will post additional updates to this space as we receive more information from our data center.

Sincerely,
Brent Lollis, President
Creative State, LLC