The process for managing risk includes:

• risk analysis
• risk assessment
• risk control
• manufacturing and post-release information

Risk Analysis

The first step in managing risk is identifying it — and understanding it.

During risk analysis, we write down all of the known or possible defects (“hazards”) of a given product, and then analyze the risks of each.

Our goal, in doing this, is to identify the seriousness of each risk. To that end, we classify each risk in terms of its severity, likelihood of occurrence, and probability of detection, following this scheme:

• Severity (Critical, Major, Minor, Trivial)
• Likelihood of Occurrence (High, Low, Very Low, Occurrence Not Anticipated)
• Probability of Detection (Practically Impossible to Detect, Random, High, Almost Certain / Impossible to Miss)

After that, we assess the risk to determine what, if any, mitigation actions are needed to reduce or eliminate it, along with their priority.

Risk Assessment

A risk assessment has three possible outcomes:

1. Accept the Risk
2. Reduce the Risk
3. Remove the Risk

Our Risk Management Plan tells us how to respond based on the overall seriousness of the risk, as measured by the criteria mentioned previously.

Generally, we accept risks only when we believe the severity of their impact will be low and/or we think their occurence is unlikely. We also consider how easily they can be detected and dealt with.

In rare cases, when mitigation isn’t possible or easily achievable, we eliminate the risk entirely by removing its source, whether it’s a component or feature within an application, or a process flow that introduces more problems than it solves.

In most cases, however, we work to reduce risks by mitigating them. When risk reduction is selected, the risk is recorded as part of project and/or product risk documentation, along with any planned remediations. (Alternatively, if we decide to accept a given risk, we note that as well, along with the reasons for its acceptance.)

Rick Control describes the process for dealing with risks that need to be reduced.

Risk Control

Risk Control comprises four key areas:

1. Risk Reduction
2. Recommended Measures
3. Assessment of Residual Rick
4. Risk / Benefit Analysis

Risk Reduction

We take into account the following elements (listed in order of importance) when reducing a risk:

• The product’s own reliability (in terms of information security, data privacy, data integrity and availability, etc.)
• Protective measures that are part of the product or development process
• Additional reliability or performance information

Based on the above, we determine the mitigation measures we need to implement to reduce the risk and assign them to responsible staff members, along with a target implementation date.

Recommended Measures

Once they’ve implemented the recommended measures, the team responsible for the product checks the results. In some cases, we might find there are still residual risks, even after the mitigation measures have been implemented. In such cases, we analyse the residual risks and assess them acceptability/unacceptability.

Assessment of Residual Risk

We assess any residual risk that still exists against the same quantitative assessment criteria as the primary risk and document it accordingly. If the residual risk does not meet the required criteria, we implement additional risk control measures, then carry out risk-benefit analyses for any residual risk that still remains.

Risk / Benefit Analysis

In cases where the residual risk is unacceptable (per our risk management plan) and further risk control is no longer feasible, we assess whether the benefits of the product exceed its risks.

If they don’t, it means that the risk is unacceptable and we have to suspend the development and/or distribution of the product. But if the benefits of the product do exceed the residual risk, then we’re more likely to accept the risk.

Other Created Hazards

Risk control measures are reviewed for their potential to cause other hazards. As a result of the application of risk control measures, new hazards may arise which need to be assessed by analogy as existing risks.

The post Our IT Risk Management Methodology appeared first on Inception CRM.

In all cases, we require the following configuration elements to be fully described:

• required configuration settings or parameters
• reasons for each setting, with reference to controlling specifications
• tools or methods use to set the required options
• dependencies and impacts on other modules or systems
• infrastructure items (e.g. OS, layer software)
• security of settings

For infrastructure components, the basic rule for configuration is to apply custom configurations and not use default settings, especially those governing access and user authentication. For software products, configuration reflects customer needs, and changes are performed on the basis of customer demands.

We manage all changes to our software applications and their supporting environments through our ticketing and issue tracking system. We typically handle these changes as new features or change requests. Each ticket has an author, specification, approval clause (if necessary) and version number. Per our release policy, all requested changes are assigned to a new software version.

When the scope of the version is final and we’ve implemented all included features, we prepare the version for the quality assurance (QA) process. If the version has an impact on documentation, the relevant documents are also updated. Each revision includes a version number, date, author and description of changes for traceability of changes.

Click here to learn more about how we manage software versions.

The post Our Configuration Management Procedure appeared first on Inception CRM.

Our DRP focuses, in particular, on our data center operations (DC Ops) and the critical infrastructure located within our primary data center, where critical services are located.

The DRP provides a complete manual for our IT staff. It includes analyses of the various risks to individual parts of our computing systems and descriptions of their importance for business support, as well as the consequences of their outages for business processes in our company.

More importantly, our DRP describes in detail the specific steps necessary to restore critical functions. SOPs include mandatory communication procedures before, during and after the restoration of the affected systems and the tests that need to be performed after the DRP activities have ceased.

Frequency of DRP Testing

We test our disaster recovery plans with regular frequency, typically 1-2 times per year, with tests focused on both our primary and secondary data centers. In addition, we assess our ability to recover production data on a daily basis via automated monitoring of incremental backups of our production environments, which are copied daily to paired staging environments and then validated for records consistency and integrity.

Primary DC DRP Tests:

Following any major upgrade to our primary data center infrastructure, we perform controlled shutdowns of the entire infrastructure in order to mitigate any hardware or software issues detected during previous tests that cannot be performed in live operational environments. We also use these shutdowns to address known bottlenecks and improve network throughput and storage subsystem IO.

During the controlled shutdowns, we test redundancy within the primary data center, making sure that systems will remain operational when one or more core infrastructure components are down. Following this, we test environment recovery according to the DRP, following the test steps for recovery in the event of a complete data center loss. The purpose of these tests is to validate the primary DC as fully operational and confirm the correct configuration of newly installed hardware.

Secondary DC DRP Tests:

Tests focused on the secondary data center are typically done following any new hardware upgrade. A complete secondary DC loss is simulated, with data and configuration recovery from the primary DC. The goal is to test the complete replication of the primary DC within the secondary DC. Additional tests validate the secondary DC as fully operational without access to the primary DC.

The post Our Approach to Disaster Recovery appeared first on Inception CRM.

Quantitative criteria include the quality of their services, their prices, the reliability and speed with which they can deliver, and their payment terms and conditions.

We also consider, from a more qualitative perspective, their industry standing, service history, track record with similar customers, audit certifications, and compliance with international standards.

Invariablly, we also take into account more subjective criteria, such as personal relationships with their management and staff, and the over relationship approach of their organization.

In the case of our data center provider, for example, we selected the vendor who offered the highest quality service within the Czech Republic. We based our quality assessment on their tier rating, contractual guarantees, network availability and reach, and the security of their physical premises. Their position as a trusted vendor for major telecom companies, banks, and government ministries also helped identify them as a reliable site for hosting sensitive information.

We source hardware only from reputable suppliers. Typically, we base our selection on functional fit and price, as well as product availability. This is especially true when deciding between suppliers who vendor the same products. However, we favor reliability and speed of delivery, as well, and are willing to pay more for better conditions.

We source software only from vendors who meet industry standards. Price is less important, in this case, since those vendors tend to dominant their respective business areas and their products are well known and represent standards within the business — for example, Microsoft, Cisco, etc.

Use of Subcontractors in our Projects

For each customer project, we document all 3rd-party services and subcontractors whose services we plan to use during software development, or whose components we plan to integrate into the final product. We include these references both within the list of deliverables and in the product/service specification prior to implementation.

If, for some reason, we need to use a different sub-contractor, we inform customers prior to making any changes. We also make sure customers understand the reason for the change. The customer can accept or dec line to accept the change, accepting any stated risks.

While changes are rare within a given project lifecycle, they do happen on occasion, in most cases because the 3rd-party no longer provides / plans to discontinue the service. Other reasons can include changes in the subcontractor’s policies that conflict with our own guarantees, price changes, and changes in their terms and conditions.

In such cases, we source an alternative provider and present it to the customer for their acceptance. Upon the customer’s acceptance, we initiate the change and update the list of 3rd-party services and subcontractors accordingly.

The post How We Select Vendors appeared first on Inception CRM.

These include policies governing the remote work of employees in the face of office closures; handover of duties in the event of employee incapacitation; restoration of critical business systems and communication channels; continued service provision and billing; and procedures for restoring a normal flow of operations.

Business continuity planning is integral to our ability to maintain the continuity of our services. It addresses the key functions of our business: software development, software/service availability, project management, project delivery, customer support, and billing.

During the recent Covid-19 public health emergency, when a government-imposed moratorium on public gatherings closed our business offices, we shifted internal work communication entirely to online channels. Employees who were unable to attend to their duties had their work temporarily reassigned until they could get back online from a stable location.

As a result, our customers experienced zero interruption in service, support, or software availability. Payments and invoices were issued on time. From an outside perspective, our business continued as usual.

While our offices have since reopened with normal on-site operations, our customers can expect total continuity of service from D3S in the event of another emergency.

The post How We Ensure Business Continuity appeared first on Inception CRM.

D3S’s hosted computing environment uses connectivity provided by the hosting provider, Dial Telecom, a.s., an international provider and one of the leading backbone providers in the Czech Republic.

Data Center Locations

D3S’s hosted computing environment operates within two data centers, each located just over 200 km away from the other in Prague and Brno, which meets the precondition for minimal safe distance in the event of a natural disaster or terrorist attack.

Both data centers meet the highest safety and operational parameters required for data centers providing professional hosting services and secure hosting of sensitive data. Both centers are ISO 27001 certified (among other certifications) and are subject to regular corporate and government audits.

Internet Connectivity

The hosting provider supports D3S with independent internet connectivity to our services.

Primary Data Center (PDC)

The primary data center (PDC) provides services necessary for the operation of client applications (such as Inception CRM) and their services. This data center has the following operational and technical parameters:

• The PDC is located in our lockable racks
• The PDC is geographically located in Prague (12 km from D3S HQ)
• The PDC is located on the secure premises of Dial Telecom, a.s.
• Dial Telecom is responsible for the physical security of (access perimeter security) of the data center and its assets.
• Physical access to data center is restricted to authorized persons only
• Internet connectivity and power supplies are secured by redundant connections and backup electricity (both diesel generators and multiple grids)
• Air-conditioned rack space
• VPN access for most administrative tasks
• Connectivity to secondary data center enabling continuous data exchange to ensure security of backups, availability of failover services, and overall business continuity

Secondary Data Center (2DC)

To reduce the risks associated with the possible failure of the PDC and the increase in resistance in the event of its failure, D3S operates a separate backup data center. This secondary data center (2DC) has the following operational and technical parameters:

• It is geographically distant from the PDC (more than 200 km away)
• Physical perimeter protection of the data center is at the same level as the PDC
• The 2DC has the same functionality as the PDC
• The 2DC contains full backups of customer data to meet RTO and RPO objectives
• The 2DC is connected to the PDC via permanent VPN
• Server Manager and other management roles are the same for both 2DC and PDC

Administration

All application, database, storage servers located in data centers use the D3S Active Directory structure: primary and secondary domain controllers running Windows Servers. Management of these domain controllers and domain accounts are performed by D3S technicians.

All server administrators (application, database, backup) in the data center have their own unique domain accounts for managing servers.

For remote data center management, we use a built-in IpSEC VPN provided by Fortigate Firewall.

Servers

A brief list of primary server types used in the D3S data center for DevOps and ITOps is provided for illustration:

DMZ

Our DMZ – or “demilitarized zone” – contains the following servers:

• Email server
• Web server

The email and web servers are the only servers located outside of our internal network.

The post Our Data Center Architecture appeared first on Inception CRM.

Loki collects and labels logs from our applications. Prometheus meanwhile collects data that are published by blackbox_exporter (services reachability and self tests), db_checker (database integrity checks) and metrics from Kubernetes.

Prometheus also contains alerting rules. All alerts are caught by AlertManager, which provides pager services through Slack. We use Slack to coordinate our monitoring activities.

We use Grafana to view and analyse all data collected by Prometheus and Loki. To achieve better reliability of our monitoring, we use UptimeRobot (a 3rd-party tool that runs outside of our infrastructure), to mirror reachability tests from blackbox_exporter.

What We Monitor

• Services Reachability
  
  • We conduct tests via HTTP GET (blackbox_exporter). We collect test results from two instances of blackbox_exporter, both of which run in docker on Synology servers. These Synology servers are located outside of D3S’s infrastructure.
  • We use HTTP GET to run two types of tests:
    • A basic test checks to see if the landing page of an application or service is reachable and returning a 2xx response.
    • Additional testing is performed using a self-test, which is triggered by RestAPI.
      
      
• Database Integrity
  
  • We use db_checker, an internally developed tool, to check database integrity. It is a C# application running in docker that runs SQL queries against tested databases. Tests are specified in XML with use of CRON-like syntax.
  • Tests are divided into general (synchronization, jobs, …) and client specific queries to ensure a high level of quality assurance.
    
    
• Kubernetes Metrics
  
  • Prometheus Operator runs on Kubernetes and collects all runtime metrics from it. Grafana runs on Synology in our data center and is connected to it. We observe Kubernetes nodes, pods, network, RabbitMQ, and more.
    
    
• Dead Man Switch
  
  • All blackbox_exporters periodically call the API in StatusCake. If StatusCake does not receive an “I am alive” signal every five minutes it starts issuing alerts. We do this to make sure that the entire Prometheus stack is working and that Slack is receiving correct data.
    

Technologies

Logs Collection

We use Loki, a log aggregation system, to collect logs. Loki does not index data but rather identifies each log stream by set of labels. This results in a relatively low demand on system resources. Even though searching though it is not as fast as an indexed system, its low demand allows it to run continuously.

Metrics Collection

Prometheus

Prometheus periodically collects data from configured metrics that are published by application components. We implement alerts using Prometheus’s query language.

AlertManager

AlertManager handles alerts sent by Prometheus. We use AlertManager to deduplicate and group errors so that our response team responds only to relevant errors. We also use AlertManager to specify silences in the event of planned/controlled outages to prevent the IT Security Team from receiving irrelevant messages. Alerts in a “firing” status are sent to Slack, where our service teams receive notifications.

Data Viewing

We use Grafana for querying, visualizing and alerting over data metrics. Grafan allows us to create custom dashboards for faster and easier analysis of ingested data.

Monitoring Architecture

The Prometheus stack mainly runs on a Synology server in Docker, where Loki, Prometheus, Grafana, AlertManager, and other data monitoring tools are located. Blackbox_exporters are located on Synology devices outside of our data center network, though we keep one instance of blackbox_exporter on Synology in our data center as a “dead man switch”. Each application we monitor has a plugin that sends data to Loki for labeling and storage.

Prometheus reads data from Prometheus metrics servers, which are connected to production environment applications. Prometheus also periodically runs alerts (queries in Promql language) and if it detects an error, it sends it to AlertManager. AlertManager than deduplicates and groups these errors and sends relevant ones to Slack, which we use as our pager system.

Best Practices

We apply the following best practices to alerting and dashboard configuration:

Alerting

• Alerts should be as short as possible while containing important information
• Alerts that are fired too often should be revised
• Fire alerts after a reasonable downtime (10 minutes for front end services is reasonable)
• Alerts should be divided into groups of interest (developers and DevOps subscribe to relevant groups)

Dashboards

• Do not use the ROWS component of dashboards
• There should not be more than 6 graphs or tables on one dashboard
• Use drill down (Main dashboard → Component overview → Detail)

The post How We Monitor Our Systems appeared first on Inception CRM.