Measurement Data Corruption

For a blogger it is always gratifying when a post generates intelligent discussions in multiple fora. My recent post about Marina Krotofil’s presentation Damn Vulnerable Chemical Process has done just that. One of the most important discussions has been taking place on the  ICS-ISAC Group on LinkedIn. Marina has been having an interesting back and forth discussion with Sinclair Koelemij that is well worth reading.

In her latest reply Marina is discussing control system measurement integrity making the point that if you can corrupt the measurement system or its controller than all of the data security measures in the world will not protect the control system from reacting inappropriately to the actual state of the process.

In that discussion Marina states:

“The attackers are becoming very sophisticated, I know the case of the spoiled batch at the pharmaceutical factory (extortion case).”

As a process chemist this is the type of cyber-attack that worries me most, not the catastrophic attack on critical infrastructure. This is the type of attack that you can see motive from multiple parties for the conduct of an attack. This is the type of attack that is the least likely to be reported to the authorities. And finally, it is the least likely attack to be detected (except in cases of extortion).

The catastrophic attack on a chemical facility is going to be much harder to effect due to all of the safety systems in place to prevent a catastrophic accident. Many of those systems will be able to be overcome by a properly motivated, skilled and resourced team of attackers, but that will almost certainly require the resources of something approaching a nation-state actor. Those capabilities are almost certainly being developed in multiple places in the world, but most nation-state actors fully understand that the deploying of such capabilities against critical infrastructure will result in retaliation in kind (or worse).

Making a batch of chemicals commercially unusable, however, would not require the same level of sophistication because it can be done without ever having to involve a production safety system. It can be done by corrupting the output of a single measurement device; a flow meter, a load-cell, temperature or pressure indicator. In many batch chemical processes it can even be done by corrupting the output of an HMI since the operator is actually the process controller.

I really became interested in control system cybersecurity as the capabilities of the Stuxnet worm were being described by Ralph Langner. His descriptions of how the man-in-the-middle attack presented the system operators with the information that they wanted and expected to see while it was slowly destroying their equipment offended me at the most basic level as a process chemist. I have literally spent thousands of hours going over process historian data trying to track down process problems or improve a chemical process and never once questioned the veracity of the data that I was looking at. If I cannot trust the information that the control system is presenting me, I might as well go back to standing next to the reaction vessel with a clipboard, recording the output of analog measurement devices.

Few people outside of the chemical process community really understand how far we have come in production quality and process performance since the industry started to use modern process control systems. I would venture to claim that the vast majority of modern pharmaceuticals could not be produced without properly functioning industrial control systems.

I know that the batch to batch variability of all sorts of industrial chemicals has been reduced by orders of magnitude by the deployment of these control systems. That in turn makes products made from those chemicals (also made with modern control systems) more reproducible and more effective.

The ability of a potential attacker to change the measurement outputs that form one of the most important bases for the modern chemical process control system scares me to the core. This capability alone allows an attacker to destroy a modern chemical facility without physically hurting anyone or endangering the environment. Such a capability would be useable by many who would never consider blowing up a chemical facility because of the collateral damage a physical attack would entail. I have met plenty of people in my lifetime with grudges (real and imagined) against chemical companies would love to be able to attack a facility in this manner.

And the capability to do so is becoming more and more readily available.

Reader Comment – Defending DVCP

A long-time reader and noted security researcher (I’ve mentioned his name many times here) Chris Sistrunk left a valuable comment on yesterday’s post about Marina Krotofil’s presentation, Damned Vulnerable Chemical Process (DVCP). Chris reminds us that an attack like Marina described will take a great deal of time and multiple trips to your system before the actual cyber-physical attack can be initiated. This provides plenty of opportunity to detect and prevent the attack if you are paying close attention to your control system (see his comment for more details).

But even before we start the kind of monitoring that Chris describes we need to take the same kind of look at our control system as we do the rest of our chemical process in our process hazard analysis (PHA). This will help us to identify those controls that could place our facilities at the most risk if/when a cyber-attack should take place.

In a well conducted PHA we look at each step in our process in great detail to look at all of the things that could go wrong. We look at each variable and ask question about what would happen if it were too high, too low, too fast or too slow, etc. For those events that could have catastrophic consequences (or were very likely to happen with lesser consequences) we put compensating controls in place to help prevent those occurrences. The more severe the consequence, the more compensating controls we put into place.

Given the new cybersecurity environment, we should now consider extending that process down to the controller level when we identify high consequence vulnerabilities in our chemical processes. When we determine, for instance, that a high temperature will lead to a catastrophic consequence we need to take a detailed look at the sensors and controllers that directly impact temperature control.

This detailed look would include the specific vulnerabilities associated with those devices. For example, are these devices that can have their programming changed by anyone with access to the device (Dale’s unsecure by design PLCs)? If so, we would want to take special precautions to limit access to that device.

Where process safety rules require multiple mitigating measures we could use multiple sensors for instance with a ‘tell me three times’ requirement familiar to rocket scientists. Or we could use stand-alone safety systems, air-gapped from both the control and IT networks, and provided with an uninterruptable power supply to provide the ultimate control system protection.

We shouldn’t forget Chris’ monitoring requirements. In fact, for those really sensitive portions of the process where the really bad things can happen (the things that go boom in every process engineer’s nightmares) we might want to ensure specific log checks for the most critical devices controlling that portion of the process.

In short, we really want to make safety and security two sides of the same coin. After all the goal of each is to keep chemical processes within the narrow confines necessary to keep employees and the community safe and healthy.

BTW: An anonymous commenter provided a YouTube link for Marina’s talk (without the annoying 15 minute delay at the start) - https://www.youtube.com/watch?v=TPUzNMcFb4A  

Damn Vulnerable Chemical Process

As I continuously report, the Internet is a wonderful information sharing tool. Where else could I watch a video [https://www.youtube.com/watch?v=aa9fjm8mzt4 Link updated 8-10-26]of a presentation (the actual presentation starts at about 15:50) presented earlier today at the 31st Chaos Communication Congress in Hamburg, Germany by a young German lady teaching computer security professionals how to attack a chemical plant.

Marina Krotofil provides a very good and detailed explanation about why it is so difficult to conduct a cyber attack on a chemical manufacturing process. Or at least a successful attack that produces a pre-selected outcome; as she mentions in passing an attack causing disruption or economic damage may be much easier to accomplish.

She does a good job of explaining the cyber-technical details of why it is so hard to cause specific damage to a chemical facility even with a vulnerable control system. This isn’t so much because of the security aspects of the control system, but rather because of the complexity of the chemical system and the complex systems needed to safely control that system.

As a process chemist with some experience in developing the processes by which chemicals are produced and dealing with the upsets that can affect those processes I can fully appreciate how difficult it would seem to an outsider to figure out a way to catastrophically disrupt those systems. Chemists, chemical engineers, and control systems engineers spend the better part of their careers developing systems to prevent those upsets.

But a person with the appropriate background and working experience in process control could take a quick look at the P&ID that Marina showed in her talk and point out dozens of process vulnerabilities that could be susceptible to outside attack. Interestingly these would almost certainly be clearly identified in process hazard analysis that OSHA requires to be conducted on most reasonably hazardous processes.

An effective cyber-attack on something as complex as a chemical manufacturing process is not something that is going to be accomplished by a lone hacker over a highly caffeinated weekend. It will take the skills of a hacker, a control systems engineer and a chemical engineer and perhaps a chemist or two to really effectively execute a catastrophic attack on a modern chemical facility. And it will take time and resources to affect. That is the good news. The bad news is that any nation-state or large sophisticated terrorist organization will have access to plenty of the appropriate talent and resources.

Take the time to look at this hour and a quarter video. If you’re a process control professional, it will scare the hell out of you.

BTW: More about Marina’s brief mention about the NIST test bed effort see my post here - http://chemical-facility-security-news.blogspot.com/2014/08/reconfigurable-industrial-control.html