We are working directly with affected customers and/or customer instances and will be restoring data to them over the next ~2 weeks. We will contact affected customers directly to confirm when this has been done. Since instances are functioning correctly at this time, we are treating this incident as resolved and no further updates will be made here.
Posted Nov 08, 2021 - 17:06 UTC
Update
We have successfully recovered data that was added to affected instances between September 28, 2021, 1:11:11 AM UTC and September 28, 2021, 12:12PM UTC. We will reach out to provide more details about how we will be restoring recovered data to affected TestRail instances once we have more information. Please note that you are still able to use your instance at this time, as we will be working to merge your current data with the restored data.
Posted Oct 04, 2021 - 23:09 UTC
Update
There are no new updates at this time. We are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. Please note that you are still able to use your instance at this time, as we will be working to merge the current data after the restore has completed. We will post another update on October 4, 2021, at 15:00 UTC, unless we receive more information before that.
Posted Oct 01, 2021 - 18:58 UTC
Update
There are no new updates at this time. We are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. We will post another update at or before October 1, 2021, at 15:00 UTC, unless we receive more information before that.
Posted Sep 30, 2021 - 23:04 UTC
Update
All Cloud systems are operational, but we are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. We will post another update at or before September 30, 2021, (today) at 22:30 UTC, unless we receive more information before that.
Posted Sep 30, 2021 - 15:06 UTC
Update
Per our earlier update, all affected instances should now be accessible. It’s possible that affected instances may still be experiencing challenges with lost data however. This is due to the fact that the instances have been restored from a backup prior to the crash. The backup used has the following timestamp: September 28, 2021, 1:11:11 AM UTC.
What this means: Data added to your instance after the timestamp but before the outage may not have been fully restored to your instance. We are continuing to work with AWS to recover and restore data added to your instance after the backup timestamp, but this process is expected to take a little while.
We will update this status once we have completed our work with AWS to restore all of your data. We appreciate your patience and apologise for any inconvenience caused in the meantime.
Posted Sep 29, 2021 - 14:16 UTC
Monitoring
All affected instances are now functioning. Data has been restored to the affected instances from a backup prior to the crash: September 28, 2021, 1:11:11 AM UTC. We are working to restore any changes after this time to the affected instances, and will continue to provide updates on progress as we make it.
Posted Sep 29, 2021 - 10:15 UTC
Update
We’re continuing to work with AWS Support and we will keep you updated as the situation progresses.
Posted Sep 29, 2021 - 07:34 UTC
Update
We are continuing to implement the fix identified earlier today, and service has been restored to a subset of affected instances. We will provide an update as soon as we have more information.
Posted Sep 28, 2021 - 17:03 UTC
Identified
The issue has been identified and a fix is being implemented.