Cyber Forensics

The debate about the Illinois Water Hack continued this weekend with two different security experts pointing out that DHS and the FBI did not say that there wasn’t a hack, just that they couldn’t find evidence of a hack. The difference is not that subtle and according to these two unrelated authors may be more than just political correctness.

Lack of Forensics

Joe Weiss, in a short posting on his blog over at ControlGlobal.com, said it most plainly:

“The point of the [DHS ICSB-11-327-01.pdf] statement was "no evidence". That means not only could they not confirm a cyber intrusion occurred, they could not confirm a cyber intrusion did not occur.”

He goes on to point out that there is no data log for communications with the disabled pump so that there is no way to determine with certainty if the system unnecessarily cycled the pump by command (either a hack or a bug) or if there was some other fault in the system that caused the pump to fail.

Now I have never worked with the control system logs that Joe is talking about, but I have worked extensively with data historians. Our control system had literally thousands of potential data point available. This was simple stuff like valve states (open, closed, moving) and measurement data (temperature, pressure). We had to select which data points we wanted the historian to record because of the memory limitations in our system. Every time we upgraded our control system the available memory expanded as did the number of points that we could monitor; there was always more data available than was recordable.

If you overlay that complexity with a log of the intra-system commands [both within the system itself, with the HMI and any remote access] and data exchanges that occur in a modern control system and I would assume that one would have an even greater problem. Even with today’s cheap memory, a complex control system just produces too much data to record it all.

What to Record

Jake Brodsky in a lengthy comment (he unfairly labels it a ‘rant’) this weekend over on the SCADASEC list (at InfoCritical.com, registration required) addresses this issue with a high-level summary of the type of information that control system logs should contain to allow for at least some level of forensic analysis of control system anomalies. He doesn’t provide a point-by-point analysis (impossible since it would be different for each systems and installation) but gives a general description of the types of information that should be tracked.

It would be nice if regulatory agencies would specify some minimum level of forensic data recording for systems under their purview. I think NERC CIPs attempt this at some level, but I am not familiar enough with those standards to really comment on their forensic recording status. The CFATS program is stuck with their ‘no requirement’ mandate from Congress. The closest they come is stating:

“Recognizing and logging events and incidents is a critical component of network monitoring.” (Risk-Based Performance Standards guidance document, pg 75)

The EPA is even less helpful to water systems since their mandate from Congress is only to require vulnerability assessments and action plans.

Of course, any forensics data recording requirement would have to be risk based to be effective. It certainly would not make sense to require a blending operation without hazardous materials or DHS chemicals of interest (COI) on site to have the same level of recording as a major oil refinery. Nor for a water system with just 2000 customers to maintain the same records as say the Long Beach, CA water system serving millions.

Cannot Prove a Negative

Of course, we have to remember that it is not possible to prove a negative. If the affected water system in Illinois had had extensive forensic data recorded, DHS and the FBI would still have had to say that ‘no evidence has been found’ rather than ‘there was no cyber-intrusion’. With adequate data you might be able to prove (and even maybe prosecute suspects) that an attack has occurred, but you can never be sure that someone hasn’t come up with a new vulnerability to exploit that you weren’t watching.

One last point to remember; in this case there was some sort of data logging, the communications with the system from a Russian IP was clearly recorded and widely reported. I would almost be willing to bet that it was this data point that caused the Illinois Terrorism & Intelligence

Center (the name for the State fusion center for those who are confused about who the initial report came from) analysts to decide that it was a cyber-attack that needed reporting.