Tweets and Comments – Pay for Patch

I’m really not trying to run a cybersecurity blog here, but it certainly does seem that cybersecurity posts seem to draw the most attention. I had two readers respond quickly, Joel Langill in a Tweet and Dale Peterson in a blog comment, to today’s blog post telling me that the practice of charging for security patches is fairly wide spread in the ICS vendor community.

 

I’m sorry to hear that, as one should be able to deduce from reading my blog post. Since I haven’t worked on an ICS since 2006 and didn’t maintain it then, it isn’t too surprising that I haven’t heard about this since it certainly hasn’t been discussed in any of my reading sources for the last couple of years.

Oh, well, I guess I could avoid some embarrassment and delete the last paragraph of my post since I obviously got it wrong (along with my tweet about the original posting), but I think I’ll let it stand. I’ve never been one to try to hide my mistakes.

It does make me wonder, if it is fairly wide spread, why ICS-CERT chose to mention the fact in their advisory about the EOScada system. I don’t recall seeing this mentioned in any other advisory that I have read over the last three years. Could it be that someone there in Idaho was trying to provoke a reaction to get a discussion started about the issue? We’ll probably never know, but I would like to think the issue deserves to be discussed, so let this be the forum. I’m not getting paid by either side and I don’t have a personal axe to grind in the matter.

So let’s hear what people have to say on the issue. I would like to hear from owners, and vendors, and integrators, and researchers, and even the politicians. I have a smattering of readers from all of the above. By all means hide behind the anonymous first name but please add your last name as vendor, owner, integrator, researcher or politician, that way we know where the comments are coming from.

And let us see if we can do this politely. On the day before our national election, we need to show the politicians reading this blog how adults discuss the important issues.

 

NOTE: Dale made a second, relatively unrelated point in his comment that I will discuss in a later blog posting.

LinkedIn Comments – Paying for Security Patches

I’ve had an interesting conversation with David Spinks, the owner of the Cyber Security in Real-Time Systems group on LinkedIn® concerning my comments last week about the EOScada advisory from DHS ICS-CERT. Concerning my comments about owners that did not have service agreements having to pay for security patches, David posted the following comment on the CSIRS group site:

“Personnally I think it is reasonable that any vendor charges to issue a patch to organisations that have purchased software and hardware but decide not to take out a maintenance contract. As I have said in other recent discussions the age of ICS on the cheap has gone .... organisations (in particular executive boards) need to beging to invest in security this includes spending on ongoing support and maintenance of the systems they are responsible for .... equipment and softwre refreash should be included in any new budget submissions for ICS systems.”

I replied that:

“I disagree, David. I look at security vulnerabilities in ICS much the same way that the US Government looks at safety defects in automobiles. They are something that the vendor is responsible for providing a fix for, free of charge. Now I would certainly agree that at some point in the life cycle of the product it makes more sense to update/replace than to patch, but given the long life-cycle of these products in the plant environment, this is probably at some significant number of years down the road.”

Now, both of those comments are rather brief, but they do address a very important issue in ICS security; an issue that deserves more than a paragraph describing each of the opposing sides of the disagreement.

To Pay or Not To Pay

First off, let’s clear up a common misconception; the customers are going to pay for patches. A company is going to expend time and materials developing patches. As with any expenses that a for-profit organization expends, the costs are recouped through sales; either directly as a service charge or indirectly through the cost of the products sold. If the costs are not recovered the company goes out of business.

A simplistic view of the above would lead one to believe that the service charge point of view taken by C3-ilex on their EOScada patch is a ‘fairer’ way to pass along the cost of the patch as the current owners are the ones receiving the benefit of the patch, so they should be paying the cost, either through an on-going service contract or a fee-for-patch.

This narrow view ignores the fact that future buyers will also benefit from the patch as it will, hopefully, be applied to future versions of the product. Another method could apply a portion of the current patch development costs to expected future product sales and allow the remainder to be charged to current customers. This may, in fact, be what C3-ilex is doing with this product; they haven’t commented on the effect of this patch on future product pricing.

Fitness for Use

So far this discussion has not looked at product liability issues. Let’s start here with the concept of ‘fitness for use’. This is a legal term that, according to BusinessDictionary.com, means:

“Effectiveness of a design, manufacturingmethod, and support process employed in delivering a good, system, or service that fits a customer's defined purpose, under anticipated or specified operational conditions [emphasis added]. Also called fitness to use.”

Product liability law generally holds that a seller guarantees fitness for use unless they specifically state otherwise. This is the reason that we see the terms ‘limited warranty’ or ‘no warranty implied’ on so many sales advertisements and contracts. The seller is legally proclaiming the fact that they are specifically limiting or declining to guarantee fitness for use.

The big problem here is the phrase ‘under anticipated or specified operational conditions’. Back in the bad-old-days when control systems were assumed to be protected-by-obscurity or air-gapped, no one made any pretenses that there was a need to offer security features on a control system beyond, perhaps, a password for access to work stations. After the twin tower attacks in 2001 we began to see a realization in more control system owners that they had a certain amount of responsibility for security their production systems, particularly in critical infrastructure industries.

Even so, Pre-Stuxnet (PS), there was still a general industry belief that control systems were protected by their complexity and isolation. Ante-Stuxnet (AS) a consensus is developing that there is much more that has to be done to protect control system security than just hide behind the myth of isolation and the faded hope of complexity. I think that a vendor would now have a hard time convincing a product liability jury that basic security measures were not covered under ‘anticipated operational conditions’ on systems sold AS; the question would be more open on PS systems.

Of course, there still remains the problem of deciding which security measures are basic enough to automatically be considered responses to ‘anticipate operational conditions’ or which are problems that could not have been anticipated by a reasonable person. This will have to be decided by case law or legislation.

The Court of Public Opinion

On the other hand, in the court of public opinion, the expectation of free security updates has already been established as the cyber-industry gold standard. In the IT side of the house that standard was clearly established by Microsoft. Their Tuesday push of security patches to the field has caused everyone else to fall into step in pushing free patches to customers; no one does it as effortlessly as MS, but they all silently follow suit.

On the ICS side the major players have all followed the MS example, except they are not actively pushing patches to the field for rather obvious reasons. Thus the standard of free patches has been clearly established in the control system market place. The current case of C3-ilex is an anomaly that will probably not be repeated.

Product Liability Legislation

Let us not forget that when Congress gets sufficiently agitated, they can wield a powerful legislative storm. Earlier I mentioned the special product liability rules that apply to the automotive industry. Largely because of the book, Unsafe at Any Speed, Congress set up the National Highway Traffic Safety Administration (NHTSA) to regulate safety recalls of automobiles. After the Ford-Firestone rollover fiasco, the rules about reporting of safety defects were further strengthened, expanding the rules of what was considered a safety defect.

The NHTSA rules set up a strict regulatory program for vendor reporting of manufacturing defects. Manufacturers are required to report any customer complaint or warranty repair to a wide variety of safety systems on motor vehicles. Additionally, there is a public hotline for direct to the government complaints about suspected safety defects while law enforcement and insurance companies also report suspected defects to NHTSA.

While the vast majority of automotive safety recalls are voluntarily initiated by industry, NHTSA does have the authority to order a manufacturer to initiate a recall. Furthermore, NHTSA receives quarterly reports on the progress of all recalls, voluntary and directed, to establish that the manufacturer is making a good faith effort to contact all owners of the affected models.

Imagine what an Industrial Control System Security Administration might look like if there is a major critical infrastructure incident that is traced back to security vulnerabilities in a control system. Congress may often be slow to act, but it is quick to over react. When its ire is raised, when the general public is grossly offended, Congress legislates with a broad and freely sweeping pen; frequently extending existing regulatory models into uncharted areas. If it is good enough for the auto industry it should be good enough for the control system industry; this is a response that is well understood by politicians.

No Fee for Patches

So no, I don’t see the C3-ilex fee-for-patching model catching on in the industrial control sector. Fortunately they are a small enough player that they are unlikely to attract the ire of Congress and cause legislation to be written to prohibit the practice. Unless, of course, one of the owner operators that has to pay for the current patch has the ear of an influential congresscritter. Then it would be easy enough to slide such a prohibition into a cybersecurity or spending bill. Similar things have happened before.

OMB Approves PCII Survey ICR

Earlier this week the Office of Management and Budget (OMB) announced the approval of an information collection request (ICR) submitted by DHS National Protection and Programs Directorate (NPPD) that would allow NPPD to collect information during a survey of participants of the Protected Critical Infrastructure Information (PCII) program. The ICR was approved without change.

The ICR

The original Federal Register notice for this ICR (76 FR 17935-17936) notes that:

The PCII Program helps government analysts, emergency responders, and other homeland security professionals access data about facilities and systems on which the Nation depends. The PCII Program is responsible for ensuring compliance with the regulation’s uniform procedures for the handling, use, dissemination, and safeguarding of PCII. In this capacity, the PCII Program oversees a community of stakeholders, including submitters of CII, authorized users of PCII and accredited Federal, State and local entities with homeland security duties.

The purpose of the survey covered by this ICR is to “gather information to improve relationships with stakeholders and maximize the value of the PCII Program” according to the abstract provided in the ICR notice. NPPD expects to have 100 responses to this survey and expects that it will take about 13 hours for a respondent to collect the data and respond. As is typical for NPPD submitted ICRs, they don’t expect this effort to cost the respondents any money; apparently they have never heard of the adage that time is money.

Interestingly, this ICR was originally submitted in February and then was withdrawn by NPPD in July; there is no word why the ICR was withdrawn at that time. It was resubmitted in September with no new Federal Register notices (ICR submissions require a 60-day notice and a 30-day notice before being sent to OMB), so one would like to assume that there were no significant changes made in the submission documentation.

The Questionnaire

There is a link to the approved PCII Stakeholder Survey provided in the ICR document; it is actually a Word® document version of the on-line survey. The version of the survey in the original submission is actually a series of web shots of the actual planned on-line survey. The only real differences between the two are differing sets of questions 11 thru 14. There is nothing startling in the differences in those questions other than the approved version appears to be asking for slightly less detail. Perhaps this was done to increase the anonymity of the responses.

While the new question 11 does include ‘Submitter’ as one of the responses to describe the person completing the survey, the responses to question 12 does make it clear that this survey is being targeted at government users of the PCII program, not the public portion that is actually providing the information. That may be why there is no cost associated with the 13 hours per response noted in the ICR.

Implications

I think that it is fair to say that the proper sharing of PCII is an important perquisite to ensuring that intelligence and security analysts have access to the necessary information necessary for doing their jobs. As such it is important for the PCII program managers to understand those things that are making that proper sharing of information more difficult.

The Wiki Leaks fiasco has also shown us what happens when it is too easy to share critical information. It would have been nice to see at least one question in the survey that would address the issues of inadequate information sharing controls.

Tidbits Heard at SCADA Meeting

One of the things that I have been missing by being unable to attend meetings and conferences is the side conversations that take place during the breaks in the meetings. I heard lots of interesting stuff at the meeting Thursday and I thought that I would mention a couple of them here.

Password Phishing Attacks

I was talking with a nice lady from the Coast Guard (I’m terrible about names and we both thought that she had given me her card, but I don’t have it) and the conversation turned to the DHS HSIN (Homeland Security Information Network) and Homeport. Both are semi-secure communications network that require some level of vetting and password access. I asked about the password change frequency and she told me it was 90-days, so this seems to be some sort of DHS (at least) standard.

I then asked her if the CG sent out emails reminding people about changing their password and they do. I didn’t get to ask any more follow-up questions, but it got me to thinking. Readers will remember that I have taken ISCD to task (most recently) for the emails they send out for updating the passwords for access to CSAT. These emails contain a link to CSAT where the password can be updated. I suspect that the Coast Guard and the folks running HSIN do the same thing.

This practice leaves a large part of the security community open to phishing attacks. A savvy attacker could send out an email like this and get system logon information by having the link go to a site they controlled. It’s not that difficult to set up an official looking site, or even a duplicate of the official site, that would allow for the collection of sign-on information; and even transmit the information to the official site so the password change would take effect.

It seems that DHS as an organization needs to re-think its password policy. I’m not sure how justifiable a 90-day reset requirement is, but the emails going out to remind people to reset their passwords should not include a link to the site where that takes place. I know, it sounds real customer oriented, but it just sets people up for failure.

A final word on this topic (well in this post anyway, I’m afraid I will return to it again at some future time). If you receive an email from a DHS agency about updating one of your passwords with them, DO NOT click on any links in the email to do so. Use your own list of links to get to the site.

Shamoon Attack Vector

You’ll pardon me if I don’t mention where this last tidbit came from. There were a couple of side conversations that went on about the control system implications of the recent big name attacks, including Shamoon. No one had any news about any direct attacks on control systems by these programs, but a number of people were concerned about the possibility of control system information being harvested by these attacks; nothing new there. One of these conversations, however, did provoke a comment about an idea floating around the counter-intel community that the Shamoon attack on Aramco was initiated via a thumb drive (no duh) inserted into a security system computer by a Palestinian security guard.

Palestinians perform a large number of low-level jobs in Southwest Asia including front line security officers. Okay, I know that security guards are an important part of the overall security plan and shouldn’t be considered low-level employees, but they certainly are so considered by most people; just look at their pay scales. It wouldn’t be hard for any national intelligence agency, terrorist group, organized crime syndicate, or even an oil-industry competitor to find and turn one of these security guards into a thumb-drive agent.

I’ll bet that every guard-house at active security gates have a computer or security terminal inside. I would bet that they don’t receive anywhere near the attention that computers do in the secure areas of the facility. But they are networked to the security office which is, almost certainly, linked to the enterprise system. This would be a nice attack surface.

And, if you were planning an attack, cyber or physical, wouldn’t it be nice to have a look at the security system controls before you started the attack? Quis custodiet ipsos custodes? Hopefully, not the attacker.

EOScada Alert from ICS-CERT

Yesterday DHS ICS-CERT published an advisory concerning multiple vulnerabilities in the EOScada application from C3-ilex based upon a coordinated disclosure from Dale Peterson of Digital Bond [Links added 11-03-12 06:30 EDT] (yep, it appears that even Dale will succumb to the temptation to coordinate a disclosure). Dale identified vulnerabilities on multiple ports related to improper access control, resource management errors (on two different ports) and data leakage.

The advisory reports that a low skilled attacker could remotely exploit these vulnerabilities. C3-ilex has produced a patch that is available to owners that have an up-to-date service agreement with the company. Other owners will have to pay for the patch. Yes, the advisory says that owners without a service agreement will have to pay to get these vulnerabilities that were due to design ineptitude corrected. I think that I would rather pay to replace the offending system and never do business with the vendor again.

I’ll bet that if Dale had known that this vendor would charge for patches he never would have been involved in a coordinated disclosure.

InfraGard SCADA Briefing

Today I had the pleasure to attend an SCADA Security Briefing sponsored by InfraGard, the Louisiana Governor’s Office of Homeland Security and Emergency Preparedness (GOHSEP) and Cimation. The presenters were Special Agent Will Hatcher (FBI), Devin King (GOHSEP) and Marc Ayala (Cimation). There were about 20 attendees from Louisiana chemical companies, and ICS vendor, and the US Coast Guard.

Presentations

The presentation by SA Hatcher was a good review of the change in the cybersecurity threat over the last 20 years or so (it was nice to hear someone talk about cybersecurity that remembers that hackers started out as phone phreakers, stealing service from Ma Bell). It was a fairly comprehensive review of changes in IT and ICS security issues over time. As one would suspect, SA Hatcher has had more experience with IT security issues, but he had a good understanding of recent ICS issues and looked at the DUQU-Flame-Shamoon as potential reconnaissance tools for future ICS attacks.

Devin gave an interesting presentation on the cybersecurity programs that he has helped develop for the Louisiana Fusion Center, one of the first cyber-fusion units in the country. Once again his main background is IT security, but, because of the large petrochemical industry in the State, there is a significant interest in developing ICS related cyber-security information sharing in the State Fusion Center. He noted in the presentation that he is getting significant information about cybersecurity incidents from State and local government agencies (about 50,000 reported cyber-attacks of all sorts per week), but nothing from the private sector. He solicited input from the audience noting that the unit was able to providing a variety of situational advisories and an extensive IP Blocking list.

Marc gave an interesting presentation on ICS security, having worked with ICS systems for a number of years. He included an interesting story about an ‘air-gapped’ control system that he had looked at that was based on an old-style pneumatic control system; the only problem was that the compressor supplying the control system air was a new-fangled, electronically controlled system complete with an internal web server.

Marc provided an interesting bit of information about the recent attacks on pipeline control systems. It seems that ICS-CERT updated their advisory (ISCA-12-136-01D) on their restricted server last week. The new version provides lists of files, versions and dates that have been found on affected systems; data that can be used to check computers for symptoms of attack. Marc pointed out that one of the files would look like it was a file for an Adobe file reader. This is a good reason for control system owners to have someone signup for HSIN access to that controlled server. (NOTE: I'm not signed up for this because of information sharing restrictions on their restricted information; not a good thing for a blogger.)

Demonstration

Marc also provided a demonstration of the results of a denial of service attack on an AB PLC. He had a nice HMI-PLC system setup that controlled a pump motor on the other side of the room. First he showed how he controlled the pump motor from the HMI via the PLC. Then he sent some random signals to an open port on the PLC simulating a DOS attack; it took just a couple of seconds for the pump to shut down. Even worse he showed that the DOS attack also resulted in the instruction set on the pump controller being erased so that it had to be reprogramed before the system would work again. Then he demonstrated how a firewall device protected the open port.

Future Briefings

This was one of a series of these briefings being conducted around the State. There is another one next month in Lafayette, LA  (watch Marc’s blog at Cimation for registration information). I would certainly recommend that facility owners and security officers consider attending. I would also recommend that other state organizations consider contacting Cimation or InfraGard to set up similar briefings.

ICS-CERT Late in Publishing Siemens Advisory

Yesterday the DHS ICS-CERT published an advisory for the Siemens SiPass Server. Siemens published this on their ProductCert web page back on October 10^th, a fact I noted in an earlier blog post about another Siemens product advisory from ICS-CERT. This advisory describes a buffer overflow vulnerability that was reported in a coordinated disclosure by Lucas Apa from IOActive.

According to the ICS-CERT advisory this vulnerability could allow a relatively low-skilled attacker to conduct a DOS attack or potentially execute arbitrary code on the system. Siemens has produced a hot fix (available through customer support) for three versions of the system, older versions should be upgraded. Additional protection can be established by configuring perimeter firewalls to block the affected port.