President’s Cyber Security Legislation

I had been going to ignore the recently released cyber security plan from the Obama administration, I have wasted too many hours in the past reading this type policy document from this administration only to find vague generalities and platitudes without actionable recommendations. Apparently this was a mistake; according to a recent blog by Dale Peterson at DigitalBond.com this bill proposed by the President would provide for ICS regulation at critical infrastructure facilities. Dale’s blog is well worth the read and he’s convinced me that this bears closer attention. BTW: Make sure you read the reader comments at the end of the blog post.

Senators Lieberman (I, CT) and Collins (R, ME) apparently think so as well, they will be holding a hearing on the proposed legislation on May 23rd at 10:30 am. No details are yet available about who will be appearing at the hearing. If Sen. Rockefeller (D, WV) holds a hearing on this as well, then we’ll know that the Senate is taking this legislative language seriously.

I promise that I’ll read the President’s proposal and see if I can add anything to Dale’s analysis.

Bundled Software Issues

Earlier today I did a blog post on the updated advisory for the multiple vulnerabilities for the ClearSCADA software package. The update was necessary, not because of a change in knowledge about the vulnerability or its mitigation, but because ICS-CERT became aware that the ClearSCADA software had been bundled into another ICS package, extending the vulnerabilities to a new system. This raises a whole host of potential issues that need to be addressed in the ICS security community.

Identifying Bundled Software

Back in the good-ole days software was not complex; a program did only one thing; data processing. The first program that I wrote (okay I was part of a six kid programming team) was in 1964 and we wrote a program to print out the ‘n’ powers of 2 in decimal and binary notation from n = 0 to 100. We watched the code being punched into the cards, the code compiled in a card reader and then run on a huge computer two rooms distant from where we pushed our noses up against the glass.

I learned about running routines when I progressed to Basic, calling sub-routines in Fortran and then using libraries in C++. That was the last of my programming and the world has changed much in the decade and a half since then. Now systems engineers, particularly in control systems, bundle a number of different programs together, each one doing a different job, or handling a different piece of equipment or interface with another system. The larger companies use mainly their own software, smaller companies buy programs from what ever provider gives them the best price. The pieces of the equipment are wired together and the software is virtually connected to every other piece of software in the system. From that point forward the users and owners are unlikely to know who supplied what part of the bundle; most won’t even know there is a bundle.

This system obviously works pretty well because most people don’t even realize that it is there. Unfortunately, when one piece of the bundle is susceptible to an attack because of an internal vulnerability, it most likely makes the entire bundle potentially subject to that attack. When a software developer writes a patch for a piece of software it is unlikely to address downstream issues or connectivity issues driven by the vulnerability.

So for example when the Serck realized that one of the components of their bundled SCX software (ClearSCADA) had an identified vulnerability, they could not just tell their customers to download the patch for that component. While it might still mitigate the vulnerability, ClearSCADA would have no way of knowing how that update might affect the remaining elements of the bundle. So Serck had to take the ClearSCADA patch, apply it to their bundle and ensure that it didn’t cause any problems for the system. They also may have had to make modifications to make it (or their system) compatible with the newly patched bundle.

Who’s Responsible for Updating the Bundle?

In this case someone obviously stepped forward and notified ICS-CERT that Serck bundled ClearSCADA in their SCX package. It could have been Control Microsystems (ClearSCADA), knowing that they sold ClearSCADA to Serck. It could have been Serck once they became aware of the ClearSCADA vulnerability. It could have been the third party security researcher who identified the ClearSCADA vulnerability. Or it could even have been ICS-CERT that became aware of the interconnection and shared vulnerability. The updated advisory doesn’t tell us.

In this instance whom ever did the notification, the system apparently worked and fairly quickly at that. The original advisory was published on February 1st and this one just barely more than 2 weeks later; and with a working patch. Halleluiah, the System Works.

Or does it? How many other control systems include ClearSCADA in their software bundle? How many SCADA components that have been identified in the last six months as having identified vulnerabilities are included in bundles that have not been updated?

So the question is, who is responsible for initiating the chain of events that ends up with all bundled software being updated whenever a component vulnerability is identified and patched? Right now no one really is required to initiate that notification. So, who should be?

The easy answer is that the company with the initial vulnerability should be required to ‘push’ the information to all customers that use that software/equipment. That is what the automotive recall procedure attempts to do for car defects. Unfortunately, there is no legislation or rule that mandates a similar recall procedure for ICS systems. Perhaps that ought to be considered in any ICS cyber security legislation. At the same time it could make ICS-CERT the legal clearing house for the vulnerability information.

DHS Addresses Two Ecava IntegraXor Vulnerabilities

Yesterday evening the DHS Industrial Control System Cyber Emergency Response Team (ICS-CERT) took the unusual action of publishing two documents on vulnerabilities in the same SCADA system, the Ecava IntegraXor. The first is a follow-up to an earlier Alert and the second is a new alert about a newly reported vulnerability.

Directory Traversal Vulnerability

Last week ICS-CERT published an alert about a directory traversal vulnerability in the Ecava IntegraXor Human Machine Interface (HMI). At the time of the alert there were no specific mitigation measures available to respond to the vulnerability. Yesterday ICS-CERT published an Advisory on this vulnerability providing newly released information on a patch (along with a point of contact for additional support information) made available by Ecava Sdn Bhd, the Malaysia-based software development company that provides the IntegraXor product.

Additionally ICS-CERT makes their routine recommendation to “Minimize network exposure for all control system devices. Critical devices should not directly face the Internet. Control system networks and remote devices should be located behind firewalls and be isolated from the business network. If remote access is required, secure methods such as Virtual Private Networks (VPNs) should be used.” They also provided their standard risk assessment caveat for both this standard mitigation technique and the patch.

ICS-CERT notes that this vulnerability would allow an attacker with a low skill level to add an arbitrary path and files to the system and to read any file within the system. The vulnerability is exploitable using publicly available tools from a remote system.

DLL Hijacking Vulnerability

DHS published an Alert on a second vulnerability, this one dealing with a susceptibility to DLL hijacking attacks. The Alert reports that there are tools publicly available to exploit this vulnerability and that ICS-CERT is working with the Ecava on mitigation options. When more information becomes available, ICS-CERT will issue the appropriate advisories.

Stuxnet Updates

I hope everyone had a good holiday. I did take a short break from writing about chemical security issues, but not from reading about them. Over the extended weekend I came across two new Stuxnet reports. The first was an update of a previous Stuxnet report, “Stuxnet Under the Microscope, v.1.3”; the second is an analysis of the effectiveness of Stuxnet in attacking centrifuges in Iran, “Did Stuxnet Take Out 1,000 Centrifuges at the Natanz Enrichment Plant?”

I was pointed at the second report by a Christmas day blog by Ralph Langner which was appropriate since Ralph was one of the early proponents of Stuxnet being a specific attack on Iranian nuclear facilities. Ralph’s blog specifically points us at the concluding paragraph from that report. The last couple of sentences from that paragraph, reproduced below, provide an important segue to Ralph’s second blog of the holiday weekend.

“It is important for governments to approach the question of whether using a tool like Stuxnet could open the door to future national security risks or adversely and unintentionally affect U.S. allies. Countries hostile to the United States may feel justified in launching their own attacks against U.S. facilities, perhaps even using a modified Stuxnet code. Such an attack could shut down large portions of national power grids or other critical infrastructure using malware designed to target critical components inside a major system, causing a national emergency.”

Ralph and a number of other commentors have noted that the effectiveness of Stuxnet against a specific target is due in large part to knowledge about the specific equipment and programming protocols found at that target location. This requires, in addition to programming skills, some fundamental intelligence information and a detailed understanding of the process being attacked. Many have argued that the requirement for this level of information means that most process owners are, practically speaking, immune to such attack.

Ralph’s “Dirty Digital Bombs”

Ralph has come up with a descriptive new term for a concept that I have previously discussed here; the ‘Dirty Digital Bomb’. As I have noted before, random changes in PLC coding will be disruptive to modern manufacturing, including chemical manufacturing, processes. While they are unlikely to cause catastrophic failures, they are very likely to upset schedules or adversely impact product quality.

While I have looked at this type of attack as a method for industrial extortion, Ralph has taken a larger industrial view. He notes that: “The dirty digital bomb is a cyber weapon that inflicts low to medium damage to a large number of random targets.” He then goes on to explain that “small damage in many power plants may be worse than big damage in one specific power plant”. It would seem to me to be particularly true in a situation where there is a carefully crafted, interconnected power grid. A number of nearly simultaneous small failures could have a catastrophic effect on the grid as a whole.

He also points out that many modern industries are interconnected in a similar manner. Specifically he points at the German automotive industry with their interconnected web of suppliers. Again he points out that “small damage at many automotive suppliers may be worse than big damage at one specific car maker”.

I would further suggest that attacks of this sort, executed in fragile economic situations like we are currently undergoing, would have a further cumulative effect. The additional economic strain of having to recover from this type of attack could be catastrophic. Even in good times, having an entire industrial sector shut down for the weeks or months necessary to cleanse their control systems at the PLC level would have a cascading effect on the economy.

Chemical System ‘Dirty Digital Bombs’

To understand how easy it would be to adversely affect a wide variety of chemical processes with a single chem-stuxnet weapon all one has to do is to consider the ubiquitous valve in the chemical industry. Valves are used to control the flow of liquids, solids, and gasses and there are only a limited number of different PLCs that are used to control these devices. The timing of the opening and closing of valves is critical to the quality of chemical product manufacturing and, in many cases, the safe operation of chemical process facilities.

If the programming for one of the valve-control PLCs were changed by a chem-stuxnet worm to add even a 30 second delay in the execution of a ‘close’ command for that PLC there would be a wide variety of potential consequences ranging from improper raw material ratios (product quality impact), to overfilled containers (chemical spills), application of too much heat (or cooling, or vacuum or pressure) leading to a difficult to control process which could lead, in turn, to any number of quality and/or safety issues.

None of these would necessarily have dangerous consequences (though there certainly could be catastrophic consequences in some applications), but even the most benign result, an off-spec batch, can have significant economic consequences particularly if identifying the root-cause of the problem is complicated by the man-in-the-middle attack methodology used in Stuxnet.

Since the same type controller may be used in dozens, or hundreds of locations within a single facility, it would be inevitable that this single programming change would result in a complete stoppage of production at the facility while the problem was diagnosed. Once the problem was identified as being a programming issue, the process of completely removing the worm from the control system could entail weeks or months of work. If the worm was spread through a significant number of facilities, the shortage of trained experts to effect the removal and system restoration would cause additional delays in bringing all of the facilities back on line. Furthermore, once a multi-facility attack was identified, all facilities using the same controller would be taken off-line to ensure that they had not also been infected.

Such an attack would not require any specific process knowledge or any other kind of facility specific intelligence collection. All it would take is the knowledge of what type of PLC’s are used in valve control situations and an understanding of the programming of that particular PLC. All of that information is readily available.

Furthermore, valves are just one of a number of possible attack vectors that are found throughout a variety of chemical processes. Modules for the control of key process variables of weight, flow, temperature, and pressure could be similarly affected.

Cyber Security

To date the bulk of interest in the political community with cyber security has been focused on the information control side of things. This is because most people are at least partially affected by information systems in their everyday lives. It is easy to understand why it is necessary to protect personal, commercial or governmental information; we see the consequences of the failure to protect that information frequently in the news.

With the advent of the Stuxnet worms and its inevitable future variants and cousins, it is becoming increasingly clear that the protection of industrial control systems will be even more important to our industrial and economic safety. A systemic attack on the chemical process industry would have far reaching economic impacts on all other industries and our country as a whole.

When Congress returns in January, they are going to need to expand their interest legislating minimum standards for industrial control system security measures. To be effective those measures are going to have to span the entire gamut of the industrial control system community, from software and hardware vendors, to manufacturers that use such systems, and to the enforcement organizations that are going to become responsible for tracking down the perpetrators of attacks on these systems.

Effective legislation and regulatory action takes time to craft. The sooner that work is started the sooner our vulnerability to these types of attacks will become manageable.

Cyber Supply Chain Security

More and more companies are taking a serious look at the security of the supply chain for the raw materials that they need for their production, insuring that the suppliers maintain quality standards, will reliably deliver those materials, and can be trusted to do what they say they are going to do. The same attention needs to be paid to the suppliers of cyber hardware, software and services that are becoming an increasingly critical resource for manufacturers.

An interesting article over at SCMagazineUS.com looks at the issue of cyber supply chain security. It looks at a recent survey of security professionals at a number of critical infrastructure organizations, looking at their security practices related to their cyber supply chain. The results of the survey are disturbing, a solid majority of the respondents report inadequate procedures and processes to review cyber supply chain security.

Supply Chain Security

As a long-time process chemist for a manufacturer of industrial chemicals, a large part of my job was to provide information to customers that was used to assure them that we were following the necessary procedures to provide them consistently high-quality product that met all specifications and was manufactured by agreed upon processes. It was not enough to demonstrate that shipped products passed specific testing requirements, but manufacturing processes, key reaction parameters, quality assurance testing procedures, facility quality, safety and security programs were increasingly being evaluated before establishing, and audited during, our supplier relationship.

All of this was done because these customers realized that the materials that we produced for them were an integral part of the products that they sold. They held us, as a supplier, to the same high standards that they held their own manufacturing people, because the consistency of our production was an integral key to the quality of their production.

It goes without saying that industrial control systems are also an integral part of the quality chemical production process in most chemical manufacturing facilities. If one thinks seriously about that, it follows that the processes that bring the components of those ICS systems to the facility floor are as important as the processes that bring raw materials to the same location. But, how many companies apply the same high standards to the suppliers of their ICS components as they do to their raw material suppliers.

Cyber Supply Chain Security

Actually, I think that it can be successfully argued that the supply chain for our ICS components, including hardware, software and technical support personnel, may be more important to our modern chemical production processes. Particularly as it is becoming more evident every day that flaws in those cyber systems provide an opportunity for outsiders to gain access to those systems. That access could allow them to steal process information, adversely affect product quality or profitability, even to shut down the facility.

Chemical professionals are becoming increasingly aware that subtle differences in the manufacturing process may be as important a measure of the quality of a manufactured chemical as the specification testing done on the product. A similar awareness is becoming increasingly a concern for cyber security professionals. Small changes in component design or fabrication, substitution of counterfeit materials, and programming flaws can all have an adverse impact on cyber security. Proper examination of the manufacturing and programming processes to ensure that they are protected against manipulation, by outsiders as well as corrupted insiders, will help to ensure that the equipment and software installed at the manufacturing facility presents the minimum exposure to outsider attack.

Since ICS suppliers are not completely vertically integrated in their design, manufacturing and software processes, part of the vetting process must be the assurance that the ICS supplier is requiring the same sort of examination of their component and software suppliers. A single component that allows an unauthorized person access to the system potentially compromises the entire ICS system. This can include access through deliberate back doors as well as via inadequately documented communications protocols or the use of default passwords.

Personnel Surety

Another potential cyber supply chain security hole is the outsiders we routinely allow to access our control system equipment. Most chemical manufacturing sites do not have the on-site expertise necessary to install, update and maintain the control system hardware and software components. Only the biggest chemical companies can have a large enough staff of process control professionals to handle these requirements. Most facilities will have to rely on equipment/software producers, 3rd party venders, or consultants to handle these responsibilities.

Cyber security professionals need to be concerned about allowing these outsiders unfettered access to their industrial control systems. The cyber supply chain review process for the organizations providing these services needs to include adequate assurances that the people sent to the facility have been rigorously vetted and adequately trained on cyber security techniques. And procedures need to be in-place to verify that the vetting is current, each time one of these outsiders enters the facility.

High-risk chemical facilities covered under the CFATS program are required to ensure that background investigations of all personnel with unaccompanied access to critical or restricted areas have been conducted. It can certainly be argued that a vendor representative sitting at an ICS control computer has unaccompanied access to that system unless closely watched by someone with a detailed and comprehensive knowledge of that system. Anyone with this kind of access, or even just physical access to an unprotected USB port on any device connected to the system, needs to be appropriately vetted.

ICS Cyber Security

All facilities using industrial control systems have a responsibility to the owners and customers to ensure that those systems are protected against attack. Intellectual property protection and protection against directed process upsets are clearly of importance to all organizations. This is part of the fiduciary responsibility and contractual obligations of facility management.

High-risk chemical companies have an even higher duty to protect their industrial control system against attack. They have the same responsibility to customers and owners, but they also have a responsibility to protect their neighbors and communities from the potential affects of a terrorist attack via those control systems. The control systems could conceivably be used to turn the chemicals stored, used or produced at the facility, in some cases the very facility, into a weapon of mass destruction.

Close attention needs to be paid to industrial control system security by all using organizations. This security awareness needs to be applied to the whole cyber security supply chain.

Weiss ICS-Cyber Security Book

Late last night, Joe Weiss posted a notice on ControlGlobal.com that his new book on control system security was now available from the publisher. The book, Protecting Industrial Control Systems from Electronic Threats, according to the publisher, is designed to “help readers gain a better understanding of protecting ICSs [industrial control systems] from electronic threats”. Knowing Joe’s history, I would bet that ‘electronic threats’ would cover the whole gamut of threats from terrorist attacks, through criminal cyber hostage situations, to incompatible systems and human error. The Amazon.com web site today listed the book as ‘temporarily out-of-stock’ but they are still accepting orders and will ship (and bill) when the books become available. The publisher (Momentum Press) web site does not mention any shortage of the book and is actively soliciting orders for both the print and ‘electronic’ version of the book. I haven’t seen the book yet (no duh). When I get a chance to read it I will certainly let you know what I think. I suspect though, that I will recommend that anyone working in or around the security end of high-risk chemical facilities should get the book. There just isn’t enough information on ICS cyber security generally available to pass up anything written by someone as experienced as Weiss.