Reader Comment – ISCD ICS Expertise

I like reader comments that start out ‘good points, but…’ and then go on to suggest additional point that I may have missed. One of those was posted to yesterday’s post on the SHODAN search engine by an anonymous reader. It’s well worth reading, but I would like to take a closer look at the first comment made by Anonymous. In response to my comments about DHS ISCD using SHODAN as part of their facility inspection process Anonymous wrote:

“However, as for ISCD not having the manpower or skill sets to do Shodan...well, that's just plain scary, imho.”

In many ways I have to agree with Anonymous, and I need to make it clear that I have not met a single chemical facility inspector yet and I have not had a chance to take a detailed look at the training that they receive in the Chemical Security Academy, so I am making some assumptions about the skills that they may or may not have. Having said that, I still think that they are reasonable assumptions for the reasons I’ll explain below.

Chemical Security Inspectors

First the Infrastructure Security Compliance Division has had to create an inspection force from scratch. Before CFATS there was no such profession as Chemical Security Inspector, so they could not hire such people off the street. So ISCD had to hire people with a general security background and then train them on the security issues specific to the chemical industry. Not only that, but to have these people walking around chemical facilities, they had to train them in the very real personal safety issues that can arise in such facilities. And they had to train them in the CFATS regulations, including the very real restriction that DHS could not require facilities to use any specific security measure.

Chemical Security Inspectors are going to be expected to take a detailed look at a wide range of security issues. First there are all of the standard physical security measures; perimeter barriers and detection systems, video surveillance, vehicle and personnel entry control systems and procedures, internal restricted area systems and procedures, improvised explosives detection and blast protection. Then there are the loss prevention measures including inventory control, order processing and customer vetting. Add in personnel surety measures, vetting and verification of training, records and maintenance management issues. And, oh yes, they must be familiar with cyber security issues for both standard IT systems and industrial control systems.

Then there are the chemical specific issues that these inspectors must understand. First they must be able to identify and understand the chemical hazards associated with almost 200 specific chemicals of interest. They are also going to have to understand the basic physical and chemical processes that are used to manufacture these chemicals or in which these chemicals are used including the process safety systems employed to protect these chemicals. They are going to have to understand the physical protection of storage, internal transport, loading/unloading, and off-site transportation systems related to the chemicals of interest.

Finally, these inspectors are going to be going into almost every type of industrial setting imaginable. The almost 200 chemicals of interest are manufactured in chemical facilities (and most people have no idea of how many different types of chemical manufacturing processes there are), but they are used in almost every type of industry you can think of and some are even sold in publicly accessible commercial establishments.

How much do these inspectors need to know about all of these areas? Well, the inspections that these folks are doing aren’t the typical three-hour compliance visits one gets from OSHA or EPA. These five day inspections look at the systems in a level of detail usually only seen when an OSHA team comes in after a major accident causes a death at a facility.

Finally, until just recently, most people in the chemical industry (actually most manufacturing industries) thought that the only vulnerability they really had to worry about with their control systems was keeping the IT people away from their systems. Their control systems were just so complex, specialized and isolated that there was no way that they were going to be attacked. So there is not a lot of security expertise in the control system community in general and less so in the chemical process community.

One last thing to consider; there are about 200 Chemical Security Inspectors with the responsibility for inspecting almost 6,000 high-risk chemical facilities. So there is not going to be a lot of room for specialization.

So, that is why I suspect that the ISCD folks may not have the people or skill sets necessary to do the SHODAN searches for vulnerable control systems at high-risk chemical facilities. I’m not finger pointing; they are just trying to do too much with too few folks (are you listening Congress?).

ICS Security Expertise

This is not a problem unique to the chemical processing industry. DHS has made a big thing of trying to hire 1,000 cyber security experts, but most of those are going to look at the protection of information processing systems, not control systems. The only Federal agency that is really looking at control system issues is ICS-CERT and that is a relatively small organization.

An example of how bad this problem is can be found in a recent blog posting over at Symantec.com. Symantec has been having problems finding people with ‘verifiable’ expertise in STL coding to help them continue their work on the Stuxnet worm. If Symantec is having problems what kind of issue is ICS-CERT going to have in expanding their operations, something that is obviously going to be necessary as it becomes more obvious that there is a real potential threat to control systems.

Expanding the number of control system security experts is going to have to be a high priority for control system venders, the control system user community, and the government. The size of the problem makes it a near certainty that it will take some sort of Federal intervention to make this happen. Unfortunately, there has been very little mention of control system security issues in the recent discussions of cyber security in Congress.

Unfortunately, there will be little incentive for anyone to spend the requisite money for developing the education and training programs necessary to produce the number of ICS security experts. After all, there have been no real attacks (other than begrudged insiders) on control systems, so why should anyone spend serious money to protect them. So we can expect the politicians to be blind to this issue until there is a clearly verifiable attack on a US chemical facility. But then it will be too late; it takes time to train security professionals, especially control system security professionals.