D3S maintains detailed plans for the recovery of our systems and infrastructure in the event of a disaster. These plans inform our staff of the procedures they should follow to restore normal IT operations.
Our DRP focuses, in particular, on our data center operations (DC Ops) and the critical infrastructure located within our primary data center, where critical services are located.
The DRP provides a complete manual for our IT staff. It includes analyses of the various risks to individual parts of our computing systems and descriptions of their importance for business support, as well as the consequences of their outages for business processes in our company.
More importantly, our DRP describes in detail the specific steps necessary to restore critical functions. SOPs include mandatory communication procedures before, during and after the restoration of the affected systems and the tests that need to be performed after the DRP activities have ceased.
Frequency of DRP Testing
We test our disaster recovery plans with regular frequency, typically 1-2 times per year, with tests focused on both our primary and secondary data centers. In addition, we assess our ability to recover production data on a daily basis via automated monitoring of incremental backups of our production environments, which are copied daily to paired staging environments and then validated for records consistency and integrity.
Primary DC DRP Tests:
Following any major upgrade to our primary data center infrastructure, we perform controlled shutdowns of the entire infrastructure in order to mitigate any hardware or software issues detected during previous tests that cannot be performed in live operational environments. We also use these shutdowns to address known bottlenecks and improve network throughput and storage subsystem IO.
During the controlled shutdowns, we test redundancy within the primary data center, making sure that systems will remain operational when one or more core infrastructure components are down. Following this, we test environment recovery according to the DRP, following the test steps for recovery in the event of a complete data center loss. The purpose of these tests is to validate the primary DC as fully operational and confirm the correct configuration of newly installed hardware.
Secondary DC DRP Tests:
Tests focused on the secondary data center are typically done following any new hardware upgrade. A complete secondary DC loss is simulated, with data and configuration recovery from the primary DC. The goal is to test the complete replication of the primary DC within the secondary DC. Additional tests validate the secondary DC as fully operational without access to the primary DC.
