...
Key contacts | CloudCard Support
Luke Rettstatt / Managing Director
Anthony Erskine / Principal Engineer
|
Key Locations | Primary Office: 1103 Wise Street, Lynchburg, VA 24504 Online Meeting Room: |
External contacts
Name | Organization | Contact details | DR process owned |
Todd Brooks | Color ID | Phone: (704) 897-1959 Email: | Communicating status and process to current customer base. |
Zack Walker | Vision Database Systems | Phone: (561) 386-1534 |
...
Hard copies of this plan should be kept in each CloudCard office, as well as the home office of all relevant employees.
Review
This plan must be reviewed and tested annually and updated to address any issues identified. The plan must also be reviewed and updated after any activation of the plan to determine improvements for future disaster scenarios.
...
Person | Roles | Responsibilities |
|---|---|---|
Managing Director | Coordination and Communication | Determine activation of plan Coordinate employee response Communicate status internally and externally Review and Test plan annually Ensure pizza is provided |
Principal Engineer | Technical Execution; Alternate for Managing Director | Ensure all failovers complete smoothly Deploy new infrastructure to replace failed infrastructure where necessary. Review and Test plan annually Designate and brief alternate person in case of unavailability. |
Revision History
...
Version |
|---|
...
| Change History |
|---|
This document was imported to Confluence from Google Docs on 2/24/2023. Below is the history of the document from Google Docs:
...
Date | Changes | |
|---|---|---|
1.1 |
...
October 2018 |
...
Initial Plan | |
1.1 |
...
November 2019 |
...
Clarity and Accuracy Updates | |
1.2 |
...
March 2021 |
...
Updates to Contact Details | |
1.2 |
...
February |
...
2023 | Accuracy Update | |
1.3 | March 2023 | Updated to reflect Active-Active AWS strategy |
Appendices
Appendix: Disaster Recovery Strategies
...
RDS - core application database is provisioned on two servers: a primary server and a read replica, located in separate Availability Zones. In the event of a failure of the primary server, the database fails over to the read replica, which is promoted to become the primary server. The former primary server can be rebooted and recovered to become the read replica, or if it is completely out of commission, a new read replica can be instantiated.
In addition to active data, Aurora backs up the database automatically and continuously for a 7 day period. Additionally Aurora takes a daily snapshot to ensure further redundancy above the continuous backups.
Elastic Beanstalk - the application servers run in an auto scaling group distributed across more than one availability zone. If an entire availability zone were to become unavailable, the auto scaling logic would provision more servers in the other availability zone until the load from the users was met.
S3 (Simple Storage Service) - redundantly stores objects on multiple devices redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region, and is designed to sustain data in the event of the loss of an entire Amazon S3 Availability Zone. (from https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html )
...
Identify issue, coordinate initial response (Managing Director)
Contact Amazon to determine extent unavailability (Managing Director)
Evaluate unavailability timeframe and impact (Managing Director)
Monitor service Failover (Principal Engineer)
Ensure database failover occurs; Add additional read replica if needed.
Ensure auto scaling replaces lost services with new nodes.
Determine if any data loss occurred, or if data needs to be corrected (e.g. to prevent stuck jobs). If so, restore or recover the data from backups.
Determine if any secondary services are down, and recover them.
Determine service and data recovery timeframes (Principal Engineer)
Share timeframes with customers and external organizations Contact customers and External Organizations to make them aware of the situation and share timeframes. (Managing Director)
Improve upon the process in case of a future disaster (Managing Director and Principal Engineer)
...
Identify issue, coordinate initial response (Managing Director)
Contact Amazon to determine extent unavailability (Managing Director)
Evaluate unavailability timeframe and impact (Managing Director)
Instruct with Principal Engineer to restore services from backups (Managing Director)
Contact External Organizations to make them aware of the situation (Managing Director)Deploy new infrastructure and restore data from backup (Principal Engineer)
If database failover was successful, add additional read replica if needed.
If the database completely failed, create a new cluster and restore from backup.
If auto scaling infrastructure is still in place, ensure auto scaling replaces lost services with new nodes.
If the application scaling infrastructure is disabled, create a new application environment from backed up code artifacts.
Determine if any data loss occurred, or if data needs to be corrected (e.g. to prevent stuck jobs). If so, restore or recover the data from backups.
Determine if any secondary services are down, and recover them.
Determine service and data recovery timeframes (Principal Engineer)
Share timeframes with customers and external organizations Contact External Organizations to make them aware of the situation (Managing Director)
Improve upon the process in case of a future disaster (Managing Director and Principal Engineer)
...