Plan for Failure

Yesterday Kevin Dunn from NCC Group gave an interesting talk at BSides DFW; “Plan to Fail: Failure Planning and Worst Case Thinking”. He made the point that, even after a company has properly employed the standard security best practices, there will still be a number of ways that the company can be breached. He claimed that his company (and most penetration testers) can gain enterprise domain administrator level access within six to seven hours of attempting penetration in most cases.

Breaches of Control Systems

Kevin’s presentation was mainly focused on IT systems, but a recent report from Billy Rios on his penetration testing at the Snohomish Public Utility District seems to indicate that the same is probably true for control systems. The time frame may be different, but the systems are hackable. So what is a control system owner going to have to do to protect the production system from being owned.

Kevin pointed out in his talk, that while it would be nice to keep attackers completely out of the corporate system, what is really necessary is to protect the company’s ‘secret sauce’; that information asset that, if compromised, will do severe damage to the company’s bottom line. The same is also true for control systems. While we would like to keep adversaries out of the control system completely, we must keep them out of that portion of the system that can have catastrophic results.

Problems and Controls

What is going to be considered a ‘catastrophic result’ is going to vary between companies and even locations. It is going to be something that could be a business ending result. For control system vulnerabilities it will be something that falls into one of three categories:

• Safety;

• Quality; or

• Inventory.

Safety events are going to be the easiest to identify. Fires, explosions or chemical releases are the most obvious, but death and damage can also happen at the lower end of the event size spectrum. Analog or stand-alone electronic safety systems are common mitigating measures that can be put into place to deal with these types of issues.

Quality issues are usually not considered catastrophic events, but in the pharmaceutical industry, for example, failure to control certain process variables can lead to the formation of chemical byproducts, or under formation of active ingredients, that cannot be identified by the quality tests used in a production environment. The presence of these non-standard chemicals in a drug can lead to death due to unexpected side effects or underactivity of the drug. Where these process variables have been properly identified in advance of production a facility can employ the same sort of systems used to identify and mitigate process safety incidents.

Most people completely overlook inventory events when they consider catastrophic issues. For companies working on slim margins using a just-in-time manufacturing philosophy, running out of key raw materials substantially before the planned re-supply is due to arrive will lead to unplanned facility shut downs. These shut downs and subsequent start-ups are not only very expensive (both for the facility and down-stream customers), but they are the most common times for encountering production problems that can cause additional production delays. Redundant inventory controls are the most readily available tools to prevent these sorts of problems.

Additional Security Protections

In addition to these mitigation measures, facility management should also look at putting additional security measures into place to slow the ability of an attacker to gain access to the critical systems that control the potentially catastrophic consequences. Those controls need to include monitoring tools that allow for an attack to be discovered in process rather than after it has been successful.

For the most critical systems, owners need to consider isolated standby systems to which production can be manually switched when the primary control system is breached. This is not likely to be of much use when end-point devices like PLCs have been compromised, but it the attack is identified early enough these stand-alone control systems may allow continued production or even just orderly process shutdown.

Security is More than Preventing Attacks

A well implemented security system will be able to stop most attacks on a manufacturing facility. But, since an advanced attacker will be able to bypass even the most secure system, a facility needs to take additional steps to prevent a catastrophic attack on the facility. What constitutes a catastrophic attack needs to be identified and additional security and operational controls need to be put into place to stop a successful attack.