Wednesday, April 16, 2014

Coordinated Disclosure is Complicated

The problem of when to publicly disclose discovered vulnerabilities in control systems is a much debated topic. Theoretically, I think that everyone in the control system vender and owner world would prefer to see researchers coordinate their disclosures so that the vendor has a chance to correct the vulnerability and the owner/operators have a chance to fix their systems before the vulnerability becomes public knowledge. But even when a researcher is committed to coordinated disclosure things can get complicated.

Coordinated Disclosure or Not

Yesterday Adam Crain, a well-known and well documented researcher, posted an interesting blog entry on his web site. In it he discusses a method of dealing with recalcitrant vendors that apparently make no effort to correct a vulnerability in their product that has been identified by an independent researchers and coordinated with both the vendor and ICS-CERT.

Historically (an odd term to use is in this young field) one of the reasons given by many grey hat researchers for not coordinating disclosures is that vendors have ignored them or failed to take action to correct identified vulnerabilities. In general the control system industry has gotten much better at responding to coordinated disclosures. This has been a major contributor to the decline in the number of Alerts that ICS-CERT has had to publish recently.

Unresponsive Vendors

But for every trend, there is an outlier. In this case it appears that Adam has identified one such outlier. Now Adam and his partner Chris Sistrunk have been the poster children for the coordinated disclosure process with the series of DNP3 vulnerabilities in 28 products that they have reported over the last year or so. They have been active proponents of fuzz testing of control system components, but they have been scrupulous in coordinating the disclosure of their serious vulnerability discoveries with vendors and ICS-CERT. But, even Adam and Chris have their breaking point.

Hidden Components

What is particularly disconcerting in this case is that this is not a vulnerable device that can be exposed to the community and provide owner/operators with the choice of how to deal with their vulnerable systems. In this case the identified vulnerability is in software library that may have been used by any number of vendors in developing the software and firmware for a wide range of control system products.

And the system owner/operators have no way knowing if that library has been used in any of their devices, so there is no way for them to protect their systems or even know if their systems need protecting. Okay, I suppose there is a way; Adam has made his fuzzing tool available free of charge and an energetic and system savvy owner could probably use the tool to find the hidden vulnerabilities. I’ll have to ask Adam about how likely is that his fuzzer would find these particular vulnerabilities in an off-line control system (NOTE: Never fuzz test a live control system).

This is an ongoing problem in the control system community (okay in the entire Cybersecurity community) where few organizations have the necessary talent or time to produce every line of code in-house that goes into their control system components. When a control system device (or software) vender buys (or uses open source) software they also get any vulnerabilities in that software. Identifying those underlying vulnerabilities and tying them to all other uses of that code is complicated at best.

We, the ICS security community, need to develop a methodology for identifying and tracking down all of the vulnerable iterations of code that are identified in one place as being vulnerable and used in other systems and applications. The ICS ISAC SARA program is designed to address part of this problem, but I am not sure that it will be capable of going deep enough into the architecture of control systems to really help alleviate this problem.

Continuing the Fight

Adam, of course, is not going to rely solely on outsiders to get this problem fixed. While he has still not publicly identified the particular vulnerability he is doing more than just writing this blog post of his about this specific vender. He is also taking the information to the ICS community; outing the vendor as one of the uncooperative ones. Not only is he attempting to use community coercion to encourage cooperative compliance, but he is in effect warning other vendors that there may be a bug in their systems that needs to be addressed.

I wish him the best of luck, but I don’t expect to see much action, at least publicly. Very few vendors have gotten to the point that they self-identify vulnerabilities in their systems (Siemens is a major exception) even if the vulnerability comes in with outside code.

Tuesday, April 15, 2014

ICS-CERT Publishes Three Advisories – Two from HeartBleed

Today the DHS ICS-CERT published three advisories for vulnerabilities in industrial control systems applications from Siemens, Progea Movicon, and Innominate. Two of those (the first and last) are related to HeartBleed.

Siemens Advisory

This advisory provides a list of Siemens products that contain or are affected by the HeartBleed vulnerability. They currently only provide one updated to mitigate the vulnerability but do note that they are working on the other product updates. The unusual move to self-identify vulnerable systems before mitigation measures are available was almost certainly undertaken because tools to test for the HeartBleed vulnerability and exploit code for the bug both exist in the wild.

Siemens reports that the following products are affected:

● eLAN-8.2 eLAN < 8.3.3 (affected when RIP is used - update available)
● WinCC OA only V3.12 (always affected)
● S7-1500 V1.5 (affected when HTTPS active)
● CP1543-1 V1.1 (affected when FTPS active)
● APE 2.0 (affected when SSL/TLS component is used in customer implementation)
Siemens has an update available for eLAN (v 8.3.3) and recommends the following interim mitigation measures for the other products until the appropriate update is published:

● WinCC OA V3.12:
o Use VPN for protecting SSL traffic
o Use WinCC OA in a trusted network
●  S7-1500 V1.5:
o Disable the web server, or
o Limit web server access to trusted networks only
o Remove the certificate from the browser
● CP1543-1 V1.1:
o Disable FTPS, or
o Use FTPS in trusted network, or
o Use the VPN functionality to tunnel FTPS
● APE 2.0:
o Update OpenSSL to 1.0.1g before distributing a solution. Follow instructions from Ruggedcom [3] to patch APE 2.0
The VPN recommendations should have come with a caveat that the VPN should have its HeartBleed status investigated before it is used to protect a control system remote access.

Progea Advisory

This advisory is for an information disclosure vulnerability reported by Celil Ünüver of SignalSEC Ltd in a coordinated disclosure. Progea has developed an update that ICS-CERT reports has been checked and validated by Celil.

ICS-CERT reports that a moderately skilled attacker could remotely execute an attack using this vulnerability to gain access of OS version information.

Innominate Advisory

This advisory notes that Bob Radvanovsky of Infracritical notified ICS-CERT that Innominate has updated their mGuard product firmware to deal with the HeartBleed vulnerability included in versions of those devices. The advisory points at the Innominate advisory published last Friday. Bob also reported this last Friday on the SCADASec List.

HeartBleed Reporting

It will be interesting to see how many of these HeartBleed advisories get published by ICS-CERT. Most of these will end up being self-reported (even the Innominate advisory was essentially self-reported). I also doubt that there will be much more information in any of the upcoming advisories than we have seen in these two today.

It may be easier for ICS-CERT to just set up a HeartBleed page and updated it when necessary by listing the vendors that have published firmware or software updates that mitigate the vulnerability. I think it would be easier and more informative.

CSB to Approve Final Report on Tesoro Refinery Explosion

The Chemical Safety Board published a meeting notice in today’s Federal Register (79 FR 21208) announcing a meeting in Anacortes, WA to review the draft final staff report on the Tesoro Refinery Explosion of April 2nd, 2010. The public meeting will be held on May 1st, 2014.

An earlier meeting was also supposed to consider the review of the final staff report, but that was changed to just an informational exchange hearing because of apparent internal Board conflicts on the issue safety case programs.

This notice does not include the normal statement about the Board receiving public comments after the staff report is presented. This may be an oversight or they may just intend for the comments heard at the earlier meeting to be full extent of the public record that will be considered by the Board prior to voting on the acceptance of the staff report.


Yesterday the OMB’s Office of Information and Regulatory Affairs reported that the DOL’s Occupational Safety and Health Administration (OSHA) submitted a request for information (RFI) concerning a possible rulemaking to update the OSHA permissible exposure limit (PEL) regulations. This RFI has been on the Unified Agenda since the Fall of 2011.

The last time that OSHA attempted to update the PEL listings was in 1989 and that rulemaking was overturned by the Courts. Apparently OSHA has been trying to figure out since then how to overcome the 11th Circuit Court of Appeals’ objections to ‘deficiencies in OSHA's analyses’.

Given the controversial nature of this issue, I expect a lengthy OMB review of the RFI.

RHETORICAL QUESTION: Does ‘RFI’ really mean ‘We don’t know what to do so Punt?’

FRA Submits Risk Reduction NPRM to OMB

Yesterday the OMB’s Office of Information and Regulatory Affairs (OIRA) announced that it had received from the Federal Railroad Administration a notice of proposed rulemaking (NPRM) to implement the congressionally mandated (and a year and a half overdue) Railroad Risk Reduction Program (49 USC §20156).

The advance notice of proposed rulemaking was published in December of 2010 (75 FR 76345). Only twelve comments were received about that ANPRM (one from yours truly based upon this blog post). Interestingly no comments were received from any freight railroad or chemical shipper.

The Railroad Risk Reduction Program would be required to be implemented by all Class I railroads and any other freight railroad that has been identified by the Secretary of Transportation as having “inadequate safety performance". Two main components of the RRR Program would be a technology implementation plan and a fatigue management plan. Both would be spelled out in the railroad’s risk mitigation plan.

There is no telling how long this NPRM will remain under review at OMB; that is as much a political question as it is a regulatory question. Given the ‘high priority’ that rail safety appears to be in Congress and in the Administration due to the number of crude oil train derailments in the last year, I would hope that the OMB will act quickly on this NPRM. I won’t hold my breath though.

Monday, April 14, 2014

Rail Emergency Response Equipment

I read some interesting testimony this weekend from Tod Perlin, Fire Chief from the Town of Rangely, ME. He was one of the US first responders that aided the local Canadian fire department during the immediate aftermath of the Lac-Megantic crude-train derailment last year. His description of what happened is very moving and well worth reading.

There is one very important point that he made in his testimony at the THUD Subcommittee of the Senate Appropriations Committee hearing on rail safety. He mentioned that, in addition to pumping water out of a local lake (the ‘lac’ of Lac-Megantic) because the local fire hydrant system was compromised in the explosions following the derailment, fire fighters used ”8000 gallons of foam [that]was trucked in from the refinery in Toronto to help extinguish the burning rail cars”.

This is one of the major emergency response problems for crude oil and ethanol transportation emergencies; special fire-fighting foam is needed to put out these fires; water just spreads the flames. Fire-fighting agencies that have fuel terminals or refineries in their response areas will typically have access to these specialized foams (and the associated equipment to apply the foam), but most fire-fighters will not have access to this important tool. It is just too expensive and hard to justify on short budgets.

One way to ensure that first responders to rail accidents involving unit trains of crude oil or ethanol have immediate access to the essential (and correct type of) fire-fighting foam would be to require railroads to haul a specialized car at the rear-end of such unit trains that carries the appropriate foam and the necessary application equipment. This way fire departments along rail rights-of-way would not need to stock the required equipment; they would only have to pay for the training in its use.

If the equipment was properly designed and the operation clearly documented, the training costs could probably be almost eliminated. If a crew of fire-fighters had to read a 5 minute tutorial on the use of the equipment, it would still be much more readily available than a foam unit being trucked in from the nearest refinery or fuel depot.

To the best of my knowledge, these cars do not yet exist. But this is clearly a need that calls out for fulfillment. And if the railroads and shippers cannot figure that out, maybe they need some encouragement from the FRA and PHMSA.

Saturday, April 12, 2014

ICS-CERT Publishes Unusual Saturday Alert Update

Earlier today the DHS ICS-CERT published an update to their HeartBleed (okay OpenSSL Vulnerability) Alert that was issued and updated earlier this week. They have also provided a link to an FBI “Private Industry Notification” that provides Snort signatures for detecting exploits of the HeartBleed vulnerability.

The updated alert (Version ‘B’) points at the FBI Snort document. It also provides a link to the free ‘Snort Community Rules’ that were updated today (presumably with HeartBleed signatures).

The alert also reports that there are ‘additional indicators of compromise’ available on the Control Systems compartment of the US-CERT secure portal. You might want to check those out.

While I don’t have access to the Secure Portal, I certainly would recommend anyone running an industrial control system consider requesting access. There are too many times that the really good information (I hope it is really good) is kept under limited distribution for legitimate reasons.

“ICS-CERT encourages U.S. asset owners and operators to join the Control Systems compartment of the US-CERT secure portal. Send your name, e-mail address, and company affiliation to”
/* Use this with templates/template-twocol.html */