Acknowledging Uncoordinated Researchers

There was an interesting Twitversation last night that was started by my Tweet on the failure of ICS-CERT to acknowledge the researcher, Blake, who was responsible for the vulnerability disclosures that were the basis for the two latest ICS-CERT alerts. You can see the entire conversation by viewing Adam Crain’s Tweet. This is a perennial discussion in the ICS security community and I would like to take this opportunity to explain my point of view on the topic.

Uncoordinated Disclosures

First let me start off by clearly stating that I think the most effective (from the user’s perspective) form of vulnerability disclosure is a coordinated disclosure through ICS-CERT. This way the vendor has a chance to correct/mitigate the vulnerability before it becomes publicly available. I prefer the disclosure through ICS-CERT over direct disclosure through the vendor because ICS-CERT has a policy of publicly disclosing the vulnerability within 45-days if the vendor “is unresponsive, or will not establish a reasonable timeframe for remediation”.

Public disclosures of a control system vulnerability before the vendor has a chance to see/fix the vulnerability are generally a bad thing for the control system user community. It allows a much wider range of potential attackers to take pot shots at our devices. It is not, however, the worst option from a user perspective. That would be the sale of the vulnerability on the black/grey market to someone who would announce the vulnerability by actually using it to attack a live control system.

Encouraging Security Researchers

Because there will always be security vulnerabilities in any complex piece of software/firmware/hardware the ICS community needs to encourage independent security researchers to continue to look for new and innovative ways to compromise such systems. It is only through discovery and disclosure that these security holes will be discovered. In an era where it has become obvious that cyber-warfare is not only possible, but likely, it is obvious that the community is better served by public disclosure/repair policies than by allowing black-market sales to attackers.

As vendors get more proficient at making their products more secure, it will be harder and more expensive for researchers to find new vulnerabilities. We will find fewer and fewer researchers who will be willing to take on the vulnerability search just out of love of a challenge. They will need to have some sort of recompense for their expenses at the very least.

We have already seen a number of attempts made to turn security research into commercial enterprises, some with more success than others. All of these have some sort of draw backs as seen from the wider user perspective in that there is no guarantee that the discovered vulnerabilities are going to get directly back to the vendor so that corrective actions can be taken to eliminate the vulnerability. Lacking an effective bug-bounty program we are going to see even more of these attempts at grey-hat research organizations.

Independent Researchers

The age of the independent researcher as the main source of vulnerability discoveries has passed. That doesn’t mean, however, that they have disappeared from the landscape. There will always be the independent who loves to try his skills against corporate computing. This will also continue to be the breeding ground for young researchers who are trying to establish reputations that will ensure their access to jobs with the more formal research community.

The ICS community needs to encourage these independent researchers to move into the larger community. This cannot be achieved by ignoring their accomplishments. Failure to provide public recognition of the efforts will drive them to the black-hat community that thrives on notoriety or into the hands of the black-marketeers that will financially reward them for their efforts.

Intellectual Property Protection

As an independent blogger, I have a great deal in common with the independent security researcher. I write this blog because of a love for the industry and the challenge of changing society. The only compensation that I receive for this work (and there are lots of hours put into this, just ask my wife) is the recognition of my efforts and the knowledge that in some small way I am making a difference.

It is important to me that my ideas are shared, but it is also important that my work is acknowledged when it is shared. Most people, when they quote my ideas will give credit either to this blog or to me personally; that is all I ask. However, when someone quotes my work without giving me credit, they are stealing my work. They are misappropriating my intellectual property.

This is what ICS-CERT is doing when they publish a vulnerability discovered by an independent researcher yet fail to disclose either who that researcher is or from where they obtained the information. Just because they are a government agency does not make that misappropriation any less real or any less costly to the researcher. In fact, it can be argued that misappropriation by a government agency is even worse because it gives the appearance that such theft is socially acceptable or even legal.

Change the Policy

ICS-CERT needs to change their policy on disclosing the identity of independent researchers who publicly disclose ICS vulnerabilities without coordinating that disclosure with either ICS-CERT or the vendor. They are not going to stop such disclosures by failing to recognize their sources, they will only drive them firmly into the opposing camp and ensure that the zero day vulnerability will become public only when it is used as a weapon.

Needless to say, I will continue to give credit where it is due whenever I can discover who was behind the initial disclosure.

The Disclosure Debate – Basecamp Disclosures

I have been asked to weigh in on the ongoing debate about the recent PLC vulnerability disclosures by Digital Bond’s Project Basecamp. The apparent assumption behind the request is that since I am not a cybersecurity researcher, but rather a chemical facility security advocate, that I might have a different set of insights into the disclosure process. As I am almost always willing to provide my opinion on just about any topic, I could hardly turndown the request.

Ground Rules

First off I have to make clear that I have a professional relationship with Digital Bond. I periodically post on their blog about cybersecurity legislative matters. Dale Peterson has asked me to do so periodically, but he does not provide any remuneration beyond the access to a wider audience for my musings. He has personally made clear to me that I would have to really work hard to piss him off enough with any Project Basecamp criticisms to harm our professional relationship. That’s good to know, but it doesn’t really influence what I would write; people who know me well realize that I will express my professional opinions almost completely regardless of who will be upset by them or impressed by them.

Second, readers of this blog will almost certainly be aware that I generally come down on the side of full and open discussion of vulnerabilities. Over the last 4½ years I have described a number of potential physical vulnerabilities for chemical targets and discussed how they could most probably be successfully attacked by terrorists. I usually leave out critical details that only a well-trained terrorist or military man would be aware of so as not to encourage wannabes, but those details are not going to affect the response of defenders in any material fashion. And that is the key to the discussion of vulnerabilities on this blog; they are provided so that owners and operators of high-risk chemical facilities might better understand the risks they face.

Finally, I am not now, never have been, nor probably ever will be the owner of a control system. I have been a user as a process chemist, but I have never been responsible for the purchase, set up or protection of an industrial control system. It may be a subtle difference, but I don’t want anyone thinking that my musings in anyway represent the opinions of any portion of the chemical security community beyond the owner of this blog.

The Vulnerabilities Exist

The vulnerabilities that were discovered by Project Basecamp exist and have existed for some time. The Project Basecamp team went looking for these specific vulnerabilities because they exist in other PLCs, specifically the Siemens PLCs. And no one was really surprised that they were able to find these particular vulnerabilities.

The designers of these PLCs knew that these vulnerabilities were there. In many cases the vulnerabilities were apparently specifically designed into the equipment. The vendors could have corrected these vulnerabilities at any time.

Finally, Project Basecamp has been in the works for some time. Dale has been talking about what the team was going to be doing for quite some time. Nobody in the vendor community or the security researcher community or in the regulatory community should have been surprised by the results or the way in which they were communicated at the end of the Project.

Systems are at Risk

The facilities that use control systems that use these PLCs are at risk for potential attacks on their facilities employing the vulnerabilities that were reported by the Project Basecamp team. They have been at risk for such attacks since they first employed these devices. There has been some incremental increase in the level of that risk since Basecamp disclosures were made; how much of an increase no one really knows for sure.

The lack of surety is due to the fact that no one knows who else has been working on discovering the details behind these vulnerabilities and has already developed specific attack vectors using these vulnerabilities. In fact, using the Stuxnet model (or even the Duqu model) we don’t know how many facilities may have already been successfully attacked using these vulnerabilities.

Dale obviously selected a good team, but I would be extremely surprised if there weren’t hundreds of security researchers out there with skills at least as good as this team. Yes, I said hundreds. Do not forget that China and Korea (and probably Russia and India and Israel and …) have specifically gone about developing offensive cyber-warfare capabilities which would require developing thousands of cyber security research specialists; many of which would of necessity be focused on industrial control systems. And that’s not even considering the cyber-criminal underground that certainly exists.

The Upside

What has certainly increased is the awareness that these specific vulnerabilities exist and the methods to exploit them are now generally available. Any cyber-security contractor, ICS owner, or government regulator can use these tools to determine if a specific ICS installation is susceptible to attack using these vulnerabilities.

There will be some installations where other security measures already in place make an outside attack very difficult or perhaps impossible (I wouldn’t hold my breath waiting on that) to attack. There will be others where the local Junior High School computer nerd can own the facility. Most will fall somewhere in the middle between these two extremes.

Knowing the specific level of vulnerability and the mode of attack that could be employed, security controls can be put into place to mitigate (though certainly not eliminate) the risk of attack using these specific vectors. Most of these are well known and understood. ICS-CERT (and Digital Bond) have been talking about them for years.

Regulators should take specific note of the tools made available via the Project Basecamp disclosures. Any security inspection at a power transmission facility or high-risk chemical facility that does not use include the use of these tools to evaluate the security of the control systems employed at that facility cannot be called a real security inspection (Congress please note that this reality should be included in any ‘comprehensive cybersecurity legislation’ being developed in this session). ICS-CERT should immediately develop a training program for Federal, State and local government security inspectors in how to utilize these readily available tools to conduct such inspections.

The Downside

Sorry Dale. Your team has significantly lowered the knowledge threshold required to design and implement an attack on any control system using these devices. You have increased the number of potential attackers with the necessary skills to effect successful attacks using the tools that your team made possible. You are going to continue to catch some heat for that and it is certainly deserved. But you all knew that going in.

The Exception

Dale did slip a ringer in on us. Project Basecamp was advertised as a look at the vulnerabilities in PLCs. Including the Koyo ECOM100 was a bit of a surprise since it is not a PLC by any stretch of the imagination. I am surprised that no one has called Dale out on including this Ethernet connection device in the Project Basecamp investigation.

If they hadn’t found so many critical vulnerabilities in the ECOM100 I would have been one of the first to cry ‘Foul’. Realistically though, the communications between the PLCs and the control system are an important part of the operation of the PLCs. The wide spread implementation of Ethernet connections have made the modern use of the PLC possible; the older method of hardwiring each PLC was just too time consuming and the source of too much system downtime.

I only wish that Dale’s team had included a wireless server instead of an Ethernet device. These are becoming more widespread. In my opinion vulnerabilities in these servers potentially pose a much higher threat to the next generation of control systems as they may provide another undocumented link to the outside world.

The Way Forward

Cyber attackers will always respond quicker than system owners. But maybe we as a society need to have a public, very visible, successful attack on a modern control system. We need to understand that every tool has inherent risks associated with the tool. We require manufacturing facilities to have guards and safety devices in place to protect the workers from the inherent dangers associated with modern manufacturing equipment. Those guards and devices are now an integral part of the machine design, installation and maintenance process at modern manufacturing facilities. We really need to get to that same point with cyber-security tools.

So, maybe Project Basecamp disclosures will become the ICS version of ‘Unsafe at Any Speed’ or ‘The Silent Spring’ or even ‘The Jungle’; making the inherent vulnerabilities in modern industrial control systems more widely known. Industry never did appreciate Nader, Carlson or Sinclair, but society owes them all a large vote of thanks.

Thanks Dale.