Reader Question – Which advisory?

 Yesterday, I had an interesting question asked me over on LinkedIn about my post last Tuesday on CISA’s control system security advisories. Taisia Berg asked: “Which of these advisories do you think will have the biggest impact on operators this week?” Quite aside from that fact that each facility is going to have its own unique mix of operational technology (and thus there will certainly be many facilities that are not affected by any of these eight advisories), manufacturing facilities generally are not able to respond to vulnerabilities on a daily or even weekly basis. Unless they can patch while operating (really not a good idea), they typically must wait for the next scheduled shutdown to patch any vulnerable equipment, and that could be months away. 

Having said all of that, this particular set of advisories presents a unique set of circumstances. The three ABB advisories relate to vulnerabilities that were all disclosed by the vendor back in January. The Hitachi Energy vulnerabilities were disclosed last week. One would like to think that owners of the affected devices/software would already have started their internal risk assessment and mitigation process for those vulnerabilities. 

So, what is the whole point about the CISA advisories, or gadflies like me writing about them? It is all about expanding the communications network that allows facilities to become aware of vulnerabilities in their equipment. In a perfect world (well almost perfect, vulnerabilities still exist) vendors would directly notify owners of their devices/software of each vulnerability as it was identified. But that is not practical because vendors frequently (usually?) sell through intermediaries and equipment frequently changes hands on resale markets. So, push notifications are not a total solution (probably not even a reasonably useful solution). 

Since many (certainly not all) vendors publish advisories for vulnerabilities in their products, one would expect owner/operators to watch the vendor's web sites for new advisories and updates. Since facilities may often have dozens (maybe hundreds) of different OT vendors to deal with, this could be a very time-consuming process (as I am acutely aware). So, CISA Advisories and blog posts like this are a shortcut to identifying new vulnerabilities (and updated information). 

Of course, CISA advisories also provide another important function, Security researchers who have not been able to successfully contact a vendor with a vulnerability notification can contact CISA to act as an intermediary. Even if CISA is similarly unable to coordinate with the vendor, they can issue an advisory based upon the researcher’s information.  

So, the unfortunate (and decidedly unhelpful) answer to the reader’s question is: “It depends.” 

Reader Question – CIRCIA Comments

Yesterday, a long-time reader asked me if I would be posting about CISA’s Cyber Incident Reporting for Critical Infrastructure Act (CIRCIA) notice of proposed rulemaking (NPRM). The question was asked because the Federal Register had ‘published’ the NPRM the day before on their ‘Public Inspection’ page. While normally this page lists the next day’s Federal Register publications, documents published in the ‘Special Filing’ section are published further in advance. In this case the CIRCIA NPRM will be officially published in the Federal Register on April 4^th, 2024

I replied to the question: “I am planning to discuss it on April 4th when it is published in the Federal Register because of the link capabilities.” I thought a little more detail might be appreciated.

First off, the early publication on the Public Inspection page does not contain the same information as provided in the Federal Register publication. Regulatory dates are typically calculated from the date of the FR publication and are noted in the PI documents as, for example, “[INSERT DATE 60 DAYS AFTER DATE OF PUBLICATION IN THE FEDERAL REGISTER]”. Additionally, some included tables may not be complete in the PI publication. Finally, there are provisions for agencies to make post PI publication changes before the official publication of the documents.

My personal reason, though, for not typically using the PI version for my blog comments is that there are no provisions in the PI version for links to paragraphs within the document. That is very important in a 417 page document like the CIRCIA NPRM. I really do like providing my readers with direct access to the regulatory language so they can see for themselves whether they agree with my interpretation of what is being said. I can do that with the FR version of the document, I cannot with the PI version.

There are also mechanical (as in writing mechanics) reasons for waiting for the Federal Register version of the NPRM to be published. There are tools available on the Federal Register Documents pages that make it easier to navigate lengthy documents and find supporting information (actual proposed regulatory code, for instance) that makes it less time consuming to prepare my analyses of regulations.

So, yes this is an important rulemaking, and an unofficial 417-page version is available for public perusal. I just do not intend to write an analysis of the NPRM based on that document. I will wait for the April 4^th publication of the official version. By the way, the 60-day comment clock starts from that publication.

BTW: That reader also commented that they did not always see my advertorial posts on LinkedIn. I reminded the reader that my Substack newsletter includes an almost daily post citing my recent publications here and other places and that post is available to free subscribers. So if you want to keep up with what I am writing go to CFSN Detailed Analysis and sign up today. You can also follow me on LinkedIn, Mastodon, and TWITTER.com.

Reader Comments - Hydrogen Facility Security

An interesting coincidence, I have received two comments on different blog posts about potential issues with hydrogen as a transportation fuel. One was a comment by a long time reader with a chemical safety background posted to a Short Takes post that included a link to an article about hydrogen fueling stations. Rosearray noted: “Widespread storage & use of hydrogen for transportation scares the sh!!!!t out of me as a safety hazard, not to mention potential for sabotage.”

I partially addressed those security concerns last month in a blog post about hydrogen fuel stations and the Chemical Facility Anti-Terrorism Standards (CFATS) program. In that post I talked about some uncertainties companies might face in planning for security measures that would comply with CFATS regulations. Those comments drew an email yesterday from my CFATS contact at CISA reminding me that the Office of Chemical Security already has programs in place to allow facilities to share preliminary information with CISA to help determine if a proposed facility might be covered under the CFATS program and what types of security measures might be required for regulatory compliance. This was not designed exclusively for hydrogen related facilities, but for the broader chemical facility audience that might be considering new facilities or expanding existing facilities. This is a good way to avoid unplanned expenses and regulatory headaches.

The general process is outlined in the CISA publication “Top-Screen Submission Considerations”.

As I discussed in my CFSN Detailed Analysis post (no longer behind paywall), if the hydrogen economy is to expand at the rate being considered by some congressional leaders, I think it would be more efficient for CISA to automate this process as part of their Chemical Security Assessment Tool (CSAT), but this early in the process, the tools currently in place would certainly work.

Reader Comment – Log4Shell Do Something Now

Yesterday I received a DM from a long-time reader and ICS influencer as part of an ongoing discussion about software updates for ICS products for the Log4Shell problem. It said:

“Patrick we need to shake up the ICS vendors they are not doing justice to the seriousness of this vulnerability .... (sic) they and their customers are all likely to be impacted .... (sic)”

My first inclination was to agree; this is certainly a problem, and something needs to be done about it now. But I recognized that response as being an emotional response and I have learned over the years that I need to let such responses simmer, so that I don’t say something that I later regret. And, this morning, I am glad that I did.

The Siemens Example

Siemens was one of the first ICS companies to respond to Log4Shell. This is to be expected. They have lots of experience dealing with newly discovered vulnerabilities. They published their advisory on December 13^th, and their first update on December 15^th. Part of the reason for the quick update was the discovery of a second Log4J vulnerability and the ineffectiveness of the first workaround. Then there was a second update on the same day. More affected products were listed, and one was removed. Now that there is a third Log4J vulnerability, I suspect that a third update is probably in the works.

Fixing Means Stopping Production

If an owner/operator started working on the first suggestions from the initial advisory, they may have had to start anew with the later information. The problem, however, is that to ‘fix’ and ICS system, you have to take that system out-of-service for some amount of time to apply the fix. That means that production must stop. If an owner/operator tried to do that for each of the Siemens’ updates, that would quickly begin to impact the bottom line.

Oh, and if you had taken the system down to ‘fix’ a device that was not really affected by Log4Shell after all, the cost would have come without any benefit.

Risk Notification

Okay, so maybe we really do want to have vendors to get a good fix made before the owner/operator tries to fix their systems. And remember, the fixes really need to be tested and evaluated in a mirror of the deployed system to make sure the fix does not cause even more problems. But we should be able to expect quick notification about the vulnerabilities from the vendors, right?

To be fair, the response that I reported upon earlier this week is the most comprehensive (far from perfect, but the best yet) set of vendor responses that I have seen since I started reporting on vendor advisories. It is far from complete, and many of the reports were of the ‘we are looking at the problem’ sort, but that is still much more than what we typically see for library vulnerabilities.

SBOM

The big problem here is that the Lib4J library is much more pervasive that first thought. This has grown beyond just a vendor or 3^rd party provider problem; we are starting to see that this is a 4^th or 5^th party provider problem. It certainly points out the need for software bill of materials, but it also points out how complex the SBOM issue is becoming in modern software development.

But SBOM is not a be all and end all. It must be accompanied by a vulnerability notification system that pushes those notifications down the line. This is the only way that vendors and end users will know to look at the vulnerabilities in the first place.

Reader Comment – Other RTOs Affected?

I had an interesting comment pop up over on LinkedIn about the RTOS advisory I discussed last night. Monty Grindy added:

“Interesting. Didn’t see QNX on that list.”

And he is, of course, correct QNX did not make the list. And, looking at the Wikipedia list of RTOS, there were an awful lot of other RTOS that were not listed. Does this mean that they were not affected? Not sure, but my guess is maybe???

If you read the Microsoft report on BadAlloc you run across some interesting comments. The first that impacts on the above question is:

“The vulnerabilities exist in standard memory allocation functions spanning widely used real-time operating systems (RTOS), embedded software development kits (SDKs), and C standard library (libc) implementations [emphasis added]. These findings have been shared with vendors through responsible disclosure led by the Microsoft Security Response Center (MSRC) and the Department of Homeland Security (DHS), enabling these vendors to investigate and patch the vulnerabilities.”

Thus, any RTOS developers that utilized the same SDKs or libc implementations could have ended up with similar vulnerabilities in their RTOS’s. This is not surprising from looking at the NCCIC-ICS advisory. Each of the 23 vulnerabilities listed use similar naming conventions for the affected operation.

Microsoft also noted that:

“These remote code execution (RCE) vulnerabilities cover more than 25 CVEs [emphasis added] and potentially affect a wide range of domains, from consumer and medical IoT to Industrial IoT, Operational Technology (OT), and industrial control systems.”

NCCIC-ICS only reported on 25 CVE’s (VxWorks and FreeRTOS both received two CVE’s). If we take that ‘more than’ statement seriously, and I do not expect that MS used it lightly, then there may be additional RTOS that MS found vulnerabilities in. If those vendors were able to convince MS and NCCIC-ICS that they were still vigorously working on correcting the problems, then NCCIC-ICS may have held off in their disclosure.

Finally, the Wikipedia entry on ‘Realtime Operating Systems’ lists a lot more than 25 entries into the category. I would be very surprised if Microsoft’s Section 52, the Azure Defender for IoT security research group, had enough time or incentive to try to test all of those RTOSs. They certainly made their point with the research that they did do.

I would also be surprised if they found this type of memory allocation vulnerability in all of the RTOS’s that they tested. Certainly, someone was using a different set of tools and libraries to set up their RTOS. It would have been helpful if Microsoft would have provided a list of systems that they tested that were not vulnerable to this particular type of vulnerability.

One final note. I am sure that now that MS has pointed the way to these common vulnerabilities in 25 different operating systems, that other researchers will be looking for other common cause vulnerabilities. Iot and OT cybersecurity is just going to continue to get more and more interesting.

Reader Comment – IOD from the Inside

An interesting comment over on LinkedIn about yesterday’s blog post on CISA’s Integrated Operations Division. The commentor is Wade W. Gough, a senior chemical security inspector with the Chemical Facility Anti-Terrorism Standards (CFATS) program. His insider-based feedback is always welcomed. He notes:

“Great discussion on our inner workings. Having worked in more complex environments under more competing command authorities, this hasn’t been an issue to me as IOD understands very well the regulatory nature of CFATS. With that & in my experience, working in a Regional office with IOD has plusses & some neagtives but nothing I would describe as a conflict or real concern & certainly nothing that conflicts w/ the CFATS program & its ability to do its job.”

I would not read too much into his ‘plusses and some negatives’ comment. There is no such thing as a perfect organization, and it is well known that DHS as a whole has had more than its share of negative feedback from its employees over the years. It is, however, heartening to hear that he has not seen anything that “conflicts w/ the CFATS program & its ability to do its job.” He obviously cannot publicly complain too much about agency operations while being publicly identified as a CSI, but there is no reason to question his unsolicited positive comments.

I do stand by my suggestion, however, that this is an organizational situation that is ripe with potential for conflicts. While good people with honorable intentions will certainly be able to make the system work, a single person with a conflicting agenda or a need for personal power could cause all sorts of problems in this type of situation. Again, someone outside of the two agencies needs to keep a periodic eye on the situation to ensure nothing untoward happens. The CFATS program had enough management problems in its early years, it does not need any new organizational blemishes.

Reader Comments – SSI and Contractors

Long-time reader and frequent cogent commentor Laurie Thomas left a comment on my recent blog post about the introduction of HR 1871. That bill would bill would require the TSA to review procedures and guidelines for the use of the Sensitive Security Information (SSI) designation of information. Laurie points out that government contractors would certainly be affected by changes in the SSI program. Her comment is certainly worth reading, particularly by the staff at the House Homeland Security Committee responsible for this legislation.

Just a reminder to anyone concerned about changes to the SSI program, TSA is VERY slow to make changes to the program. There has been a rulemaking in place since 2004 when TSA published the interim final rule establishing the program. According to the Fall 2020 Unified Agenda, the final rule was expected to be published in August of this year, but that has slipped so many times it is no longer an even aspirational forecast.

Reader Comment – Move CFATS to EPA

I received an interesting comment on a post from last week about the passage of HR 251 in the House. The comment was a short question: “Why not move the entire program under EPA?”

This question has been asked many times in the 10-year history of the Chemical Facility Anti-Terrorism Standards (CFATS) program. The short (and less than satisfying answer) is that security is the purview (for better or worse) of the Department of Homeland Security while the EPA is tasked with helping to prevent accidental discharges of hazardous chemicals. The reason that this answer is less than satisfying is that there is a great deal of practical overlap between these two missions.

The Differences

The better way to explain why CFATS should remain in DHS and not the EPA is to look at the differences between the CFATS program and the EPA’s Risk Management Program (RMP). While there is certainly a degree of commonality in the chemicals of concern between the two programs, there is a significant difference in the function of the two programs. The CFATS program is a risk management program and the RMP (despite the name) is a chemical management program.

If a facility has a designated minimum inventory of a covered chemical under the RMP program they are required to institute a number of internal programs to protect that chemical from accidental release as well as measures to coordinate with the local community to allow for an adequate response if those protections fail. The EPA will probably not get around to inspecting that facility for RMP program compliance unless there is a reportable release of a covered chemical. The result of that after-the-fact inspection will be a notice of non-compliance with one or more of the requirements of the program, a fine, and then a resumption of official ignorance until the next reportable release occurs.

That same inventory amount of the same chemical under the CFATS program triggers a reporting requirement to the DHS Infrastructure Security Compliance Division (ISCD). ISCD takes the required elements of that report and evaluates the risk that that facility might be a target of a terrorist attack. IF ISCD finds that the facility is at high-risk of such an attack, the facility is notified and is required to develop a site security plan to substantially reduce that risk. CFATS Chemical Security Inspectors will inspect the facility during the development process to help ISCD to determine if the SSP provides an adequate level of security for the facility in question. Once the plan is approved, ISCD will conduct periodic compliance inspections to ensure that the facility maintains their security program to the agreed upon standards.

Clearly, the CFATS program is a much more regulatorily hands-on program with a lot more interaction between inspectors and facility personnel. This is only possible because of the relatively small number of facilities covered by the program. Thus, 160 CSI can cover the 3,300+ facilities in the CFATS programs. The EPA would require thousands of inspectors to provide similar levels of coverage for the facilities covered by the RMP.

More Chemicals Covered

The CFATS program chemicals of interest (COI) is similar in many ways to the RMPs list of covered chemicals. The most toxic and most flammable chemicals are found on both lists with similar inventory levels triggering regulatory interest. The CFATS program, however, also includes chemicals in their COI list that could be used to make chemical warfare agents or improvised explosive devices.

This provides for some interesting differences between the two programs. Chlorine, for example, is covered by both programs as a toxic release hazard at similar inventory levels. Under the CFATS program it is also covered as a potential theft/diversion risk for use as a chemical weapon away from the covered chemical facility at much smaller inventory levels when packaged in portable containers. Again, this is a security risk (and an off-site security risk at that) not an environmental hazard.

Emergency Response Planning

Another area where there is some apparent overlap between the two programs is in the area of emergency response planning. Both programs contain an awareness that they will eventually fail in preventing an incident with serious off-site consequences. This will require that local emergency response personnel take some sort of action to mitigate the harm to local neighbors of the facility.

To date, neither program has a real strong history of ensuring that the covered facilities are providing local emergency response planners with all of the pre-incident assistance that would be needed to plan for an effective response to a hazardous chemical release. There are similar reasons for that failure. First, neither the program officials nor the facility management have any control over the local emergency planning process. Secondly, neither program has the congressional funding to provide the financial resources that the planning process requires.

Again, the CFATS program does have an advantage over the RMP; the CSI should be insuring as part of their inspection process that the facilities have at least provided the necessary information to the local emergency response folks. Again, the RMP only really provides for checking on this post-incident when it is too late to correct the problem.

Why?

The big question for me is not the ‘why not move it’ question provided by this reader, but why bother to try? Before the CFATS program was started, one could make the argument that at least the EPA had people with chemical safety knowledge that would be useful to setting up the CFATS program. The problem was (and remains) that the CFATS program is not mainly a safety program, it is much more about security than safety. If the CFATS program had been started in the EPA, the agency would have had same type initial problems with security that the folks at DHS had with chemical safety.

At this point, however, those types of safety vs security problems have been pretty much overcome. ISCD now has a pretty good mix of security and safety expertise in its CSI force. Their major problem now seems to be a lack of computer security (particularly for control systems) expertise, but that problem would be even worse in the EPA. ISCD is part of CISA and the control system security experience found in parts of that organization could be valuable for correcting the current ISCD cybersecurity shortcomings.

No, the CFATS folks need to remain as part of DHS; a move to EPA would solve almost no problems and create too many new ones. What we need now is for Congress to take a realistic look at the current program and decide what needs to be fixed and how best to take care of those issues. Unfortunately, in the current confrontational political environment we are working under, I expect that it will take longer than 15 months to accomplish that. I hope that I am wrong.

Reader Comments 11-17-14

It is rare for me to get a reader interchange in the comments section of this blog and rarer still for those comments to be about my politics. But, an anonymous reader (apparently the owner of a small CFATS covered chemical facility) responded to my Sunday post about the chemical accident this weekend in Texas. Actually I think that maybe his comments were to/about me and were merely connected to the most recent blog. In any case, he noted:

“People like you have made America a regulatory nightmare. You're an apologist for a CFR police state, and when confronted about this you drop fear bombs to justify your point of view.”

He then goes on to complain about “Soviet-style bureaucrat from DHS” and the unconstitutionality of the CFATS program under the 4^th, 5^th, 9^th, and 10^th Amendments. I have some objections to both claims. First and foremost, the CFATS inspectors that I have talked with and corresponded with were not bureaucrats in any form, but were rather hardworking, caring and professionals who were primarily interested in helping facility owners keep their facilities safe from terrorist attack in the most efficient manner possible. There may be bureaucrats in ISCD Headquarters (though the ones that I have met certainly do not meet the classical soviet stereotype), the folks in the field are so far from that description to bear comparison by the most near-sighted citizen.

As to the second, [STADARD LEGAL CAVEAT: I AM NOT A LAWYER] that does not bear close scrutiny either:

4^th Amendment – Search and Siesure – The Supreme Court has often held that regulatory inspections do not constitute searches.

5^th Amendment – Self-incrimination – CFATS regulations are not criminal in nature so there can be no self-incrimination. Upheld in numerous instances for all sorts of regulatory programs.

9^th Amendment – Rights not enumerated are protected – See next comment.

10^th Amendment – States rights. Commerce clause and common defense clause of the constitution both provide the legal framework for the CFATS program.

The second anonymous commenter (with the nom de chem of Common Sense from the South) was rather blunt in responding to the first commenter and certainly more personally directed than I would have done. I do appreciate the characterization of this blog as “a respected professional blog”.

Reader Comment – AWWA Guide Available to Non-members

Kevin Morley, the Security & Preparedness Program Manager for the American Water Works Association, left a nice comment on my post from last week about their control system security guide. He noted that:

“Access to the AWWA guidance and use-case tool do not require membership in AWWA. These resources are freely available to everyone. Access does require creation of a user account, which simply confirms that the user accepts the terms of use.”

This is certainly good news for water systems that are not members of the AWWA. It also means that folks with control systems in other types of critical infrastructure have a tool that can be used to look at their control system security.

There will be very few things in the AWWA tool that are not applicable to other organizations. It might not look at all aspects of control system security for other types of industries, but lacking this kind of detailed guide from anyone else, it would certainly be a good start.

Reader Comment – Component Security Risk – 01-09-14

Early this morning there was an interesting response posted to a LinkedIn discussion group about my earlier blog post on the Raven X EV-DO ICS-CERT Advisory. Michael Thibodeaux commented:

“I love the Raven X EV-DO advisory the best as it is a OEM issue and there is no good way to get a list of the companies that implement this device in their product. A good thesis for an undergraduate would be on this theme. To gather the who and what uses this device take lots of research and time that Undergraduates are willing to put to work for such a theme.”

This is an ongoing issue for a large number of the vulnerabilities that are reported in the ICS arena. It is bad enough when there is a patch or firmware upgrade to apply to fix the problem, but when the mitigation strategy selected by the equipment vendor is hardware replacement (especially when there is inadequate communication of that recommendation as in this case) it becomes much less likely that the fix will take place.

Since the problem here involves a wireless communications device it is particularly vexing that better solutions are not forth coming. These devices are, almost by definition, outside of the physical security protections of an installation. This could allow the access to the ‘isolated’ control system network that too many other vendor’s security vulnerabilities are relying upon for protection.

Blog Questions

I got an interesting email from a new reader yesterday that among other things posed a question about how to post questions to this blog. This isn’t something that I have specifically set up this site to do, but I suppose that over the last five years or so I have developed a process for dealing with reader questions, so maybe I ought to formally explain it.

Submitting Questions

There is no formal method for submitting questions on this blog, no specific tool or box to use. If you have a question about a specific post there is a Google comment tool at the bottom of every post. It is a link that either says “No Comment” or “X Comments” (X being the number of comments already posted). Click on that link and there will be a listing of the current comments for that post and a comment form.

Completing that comment form will result in Google sending me an email about the comment providing me with the opportunity to ‘moderate’ the comment. I have two moderation options, publish the comment as is or not. I only delete comments that are obvious spam or comments that are offensive (my standards usually deal with flaming attacks on other readers, but from time to time grossly offensive language peaks my ire).

Another, more direct way of submitting questions is simply sending me an email. I have my email address posted in the ‘About Me’ section of the blog on the right hand column below the blog roll (PJCoyle@aol.com). Tweeting me will also generally work (http://twitter.com/pjcoyle) as will sending me a LinkedIn message (http://www.linkedin.com/in/patrickcoyle/); both require that we are linked according to site protocols (note: I’m always interested in expanding my network on either site).

Responses

I now have a full time job as a QA Manager and R&D Manager for a specialty chemical company (that remains nameless so I can continue writing this blog without offending corporate sensibilities) so I don’t guarantee that I will respond to all questions/emails. If questions or comments catch my interest I will typically respond with a blog post (as most readers well know), but I do respond privately via emails as well when I feel that is appropriate and I have the time.

Privacy Protection

Please, if you have some particular reason for me to protect your privacy, please let me know that specifically. I am usually able to guess that, but I have gotten one reader in some minor amount of problems with their corporate supervision because I was under the impression that he was speaking in an official capacity. I try to be careful of that (particularly for people that are obviously not in position to speak for a company or agency), but I am mostly human and am thus prone to errors of judgment.

I will do my best to protect confidences with one exception; if I believe that the information provided to me indicates that any individual or high-risk facility is in imminent danger of physical attack or compromise, I will immediately share that information with the appropriate authorities. My call, my responsibility.

Special NOTE: Anonymity is fine, I respect that it is frequently necessary. But please, let’s see a little more creativity than the generic ‘Anonymous’. Please give me some sort of handle to hang a response on.

I hope this helps and I always look forward to hearing from readers.

Tweets and Comments – Pay for Patch

I’m really not trying to run a cybersecurity blog here, but it certainly does seem that cybersecurity posts seem to draw the most attention. I had two readers respond quickly, Joel Langill in a Tweet and Dale Peterson in a blog comment, to today’s blog post telling me that the practice of charging for security patches is fairly wide spread in the ICS vendor community.

 

I’m sorry to hear that, as one should be able to deduce from reading my blog post. Since I haven’t worked on an ICS since 2006 and didn’t maintain it then, it isn’t too surprising that I haven’t heard about this since it certainly hasn’t been discussed in any of my reading sources for the last couple of years.

Oh, well, I guess I could avoid some embarrassment and delete the last paragraph of my post since I obviously got it wrong (along with my tweet about the original posting), but I think I’ll let it stand. I’ve never been one to try to hide my mistakes.

It does make me wonder, if it is fairly wide spread, why ICS-CERT chose to mention the fact in their advisory about the EOScada system. I don’t recall seeing this mentioned in any other advisory that I have read over the last three years. Could it be that someone there in Idaho was trying to provoke a reaction to get a discussion started about the issue? We’ll probably never know, but I would like to think the issue deserves to be discussed, so let this be the forum. I’m not getting paid by either side and I don’t have a personal axe to grind in the matter.

So let’s hear what people have to say on the issue. I would like to hear from owners, and vendors, and integrators, and researchers, and even the politicians. I have a smattering of readers from all of the above. By all means hide behind the anonymous first name but please add your last name as vendor, owner, integrator, researcher or politician, that way we know where the comments are coming from.

And let us see if we can do this politely. On the day before our national election, we need to show the politicians reading this blog how adults discuss the important issues.

 

NOTE: Dale made a second, relatively unrelated point in his comment that I will discuss in a later blog posting.

Reader Comments – Tofino Technical Discussion

We have had quite a discussion started about my post last week about a new technique being offered by Tofino Security to help secure communications within a SCADA/ICS system. Apparently I didn’t get my description of their product/service too far from correct, but there has still been a lot of discussion back and forth about what it can do and what it can’t do. To be fair, Eric Byres doesn’t claim that this is the be-all-end-all security device, but that it does provide another level of security in a properly designed defense-in-depth security plan.

Readers of this blog that truly understand the ins and outs of ICS security, please read all four posts; two by an Anonymous reader, one by Joel “the SCADAHacker” Langill, and one by Eric. There is lots of good information about how this system works and how it can be integrated into an effective ICS security system. And by all means, feel free to join in the discussion; I love learning new things about ICS security.

If you are a reader that thinks that you have a pretty good understanding of the ins and outs of ICS security, take a read through the comments. If you get lost in some of the terminology in the first couple of sentences, but understand the general gist of what they are saying (kind of like me, in other words) stay on the outskirts of any serious SCADA security conversation and nod your head from time to time and you’ll look pretty smart (works for me). But please, get some professional help in designing, implementing and maintaining your ICS security system.

If the discussion is in archaic Greek as far as you are concerned, then run, don’t walk, to the nearest introductory course on ICS security before you even talk to an ICS security contractor. Otherwise, you are likely to pay too much for hardly any security.

In any case, read the discussion, it is very educational.

Reader Comment – 08-21-12 – More Sen. Grassley Disclosures

I just allowed the posting of a real interesting anonymous comment to my earlier blog posting on Sen. Grassley’s (R,IA) letter to Sec. Napolitano about mismanagement in ISCD. This new comment maintains that Grassley has uncovered “yet another egregious abuse of authority in the DHS Office of Infrastructure Protection”. As of midnight EDT there was nothing on the Grassley web site about these claims.

This anonymous reader maintains that:

“The political assistant secretary illegally removed his career deputy from office just before the 2008 election, enabling him to move his unqualified crony into the CFATS Director position, where she proceeded to practice more of the cronyism that got her there and to waste hundreds of millions of dollars, all while falsely reporting compliance with the law.”

Now if my memory (and a brief Google search) serves me correctly, the Assistant Secretary for the Office of Infrastructure Protection in 2008 was Robert Stephan, a Bush appointee. The ISCD Director would have been Sue Armstrong.

It will be interesting to see if Sen. Grassley actually has any information beyond the political cronyism charge. While we are supposed to have mechanisms in place to ensure that such things as are being alleged here never take place, no one with any experience in politics would be surprised if such things actually did take place. What is interesting here is that the cronyism started in the Bush Administration and continued into the Obama Administration. That doesn’t seem to be strictly ‘political cronyism’.

I will make one comment on one of the charges here; that of “falsely reporting compliance with the law”. If the anonymous reader is talking about Ms. Armstrong’s testimony about the CFATS program before various Congressional committees during her tenure as Director of CFATS, I don’t recall any of those appearances where she claimed anything about the program that wasn’t subsequently verified. The serious problems with the SSP authorization process didn’t really happen on her watch.

In any case, if anything more comes of this it will be just one more problem that the CFATS program will have to overcome in the next year.

Reader Comment – NIST ICS Security Guide

A reader of this blog, Ragnar Schierholz, added a comment to my recent post about the publication of the NIST ICS Security Guide. He noted, as have some bloggers, that the recently published version of the Guide is little different from the draft of the document that was published about three years ago. He then asked if I had noted the very close similarity in the two versions.

I’m sorry Ragnar, I didn’t. The reason is that I never saw the draft document. Three years ago my understanding of ICS security issues was much narrower than it is today. Like much of the user community, I was essentially unaware of the multitude of cybersecurity issues that we recognize as being important today. Three years ago readers of this blog would have read about physical security measures for control rooms and vague suggestions that complete isolation of control systems was ‘becoming difficult’.

My appreciation for the complexities of control system security issues has changed over time and this has led to increased coverage of those issues in this blog. Hopefully, this has helped to increase awareness in the user community on these issues.

As a number of bloggers have noted, if the user community does not demand increased security in their systems, vendors will likely remain reactive to security vulnerabilities rather than proactively designing more secure systems (I almost wrote ‘secure systems’ there, a misnomer if ever there was one). That demand can only be made if there is an increased understanding of the problem.

So, while the newly released security guide may be little changed from the draft, it is a new document to many of us in the chemical security community. That makes it a valuable addition to any chemical security library.

On a personal note, questions like this one posed by Ragnar make me wonder what security issue I’m overlooking today due to the limits of my knowledge. Hopefully my readers are standing by to help identify those issues for me.

Reader Comment – ICS and GPS

Earlier today Brad Calbick left a comment on today’s earlier blog about the ICS-CERT Alert on the GPS outage. While generally supportive of my comments about the late reporting, he took objection to my negative comments on using the GPS for critical system support; writing: “What is your proposed alternative to utilizing GPS for critical systems? I'm not aware of a solution that improves on the reliability and cost of GPS.”

I must admit that I don’t have an alternative, particularly since I don’t really understand what the GPS timing signal is used for. I would guess that it is used in synchronizing operations separated by enough distance that transmission time lag can cause problems. I am not a systems engineer so any thoughts that I would have on a substitute would be suspect at best.

GPS is a Military System

What I do know is politics and government. The GPS system was designed for and by the US Military for their use. A number of entities have complained over the years about the DOD control of the system, but since DOD paid for the system it is theirs to operate as they please.

I remember when civilian GPS devices were first introduced. Most people focused on the military’s initial refusal to allow civilian users full access to the system, access that would have allowed for more accurate position data. What most people failed to notice, or certainly remember, was that the military made a point of telling everyone that they would not guarantee future access to the signals.

The problem with ICS vendors using a pirated (I know, you can’t really consider receiving a broadcast signal as pirating) GPS signal for their own purposes is that they are using the signal without the military’s knowledge or ‘approval’. Thus, when the military plays with the signal for their purposes there is no mechanism (or reason) for the military to inform the ICS users.

Now, obviously the FAA has an agreement with DOD about their use of the GPS signals (almost certainly an memorandum of understanding). That agreement required DOD to provide the FAA with advance notice, and the FAA forwarded that advance notice to their GPS system users.

Protecting ICS use of GPS

If the ICS community is widely using the GPS signal for control purposes (and I don’t know how wide spread the use is) then someone is going to have to ensure that the military informs that community in advance of any modifications to the signal. I suppose that individual vendors could try to negotiate that agreement, but I doubt that that would be the most successful way of dealing with the situation.

I would suggest that DHS ICS-CERT would be the logical government organization to negotiate an MOU requiring advance notification of GPS signal modifications on behalf of the ICS community. Just like the FAA did in these two situations, ICS-CERT would then notify the community of the modifications so that facilities could take appropriate actions to protect their systems that use that signal.

Of course that brings up another problem, ICS-CERT is a passive communicator. They post information on their web site, but have no mechanism to point people to the new information. There are a few people like my self that actively monitor the ICS-CERT web site and then broadcast that information on blogs and tweets.

While that is more active than what ICS-CERT does with its information, it still doesn’t ensure that the information gets out to all affected parties. Someone is going to have to develop a communications system for these alerts that pushes the information far enough down the communication’s tree that everyone who needs the information will get it. At the same time the system would have to be careful about not blasting information out to people that don’t need it.

Again, I am not an engineer (a communications engineer in this case) so I can’t begin to describe how such a system might work. My part of the puzzle is to identify the problems and prod people into taking care of them.

CFATS Reader Communications

I had an interesting email conversation with a reader yesterday. She sent me an email to let me know about a typo in my post about HR 3082. Along with her polite question about the extension date on CFATS she thanked me for my CFATS reporting. She works at an industry representation organization and was responsible for tracking programs like CFATS that were of interest to some of the facilities in their industry; an industry that is not typically considered to be a chemical industry.

I am always interested in hearing about how CFATS affects these non-chemical high-risk chemical facilities, so I asked what is for me is a fairly standard request. I told her that I would like to hear about any problems facilities in her industry were having with CFATS implementation or enforcement. She vaguely promised to see what she could do about getting me the information, but you could tell from the tone of the last email that she had some legitimate concerns about disclosure rules.

We’ll see how that particular request goes, but I it got me started thinking about making the request to the general readership. It would save me some time waiting to be contacted by people like her in the various industries that are affected by CFATS. So here goes.

CFATS Covered Facilities

When one hears the term chemical facility security, one has the immediate tendency to think of classical chemical manufacturing and distribution facilities. While CFATS is applicable to these types of facilities, the regulation drafters at DHS knew that a great many of the facilities handling chemicals of potential interest to terrorists were far removed from that classic model. So they wrote the CFATS regulations focusing on the chemicals of interest rather than on the type of facility.

The variety of industries potentially affected is astounding. It includes meat packers (anhydrous ammonia for refrigeration systems), water parks (chlorine gas for water disinfection), agricultural facilities (a variety of pesticides and fertilizers) and even electrical power generation facilities (ammonia for smoke stack scrubbers). All of these industries have security programs of varying sorts, but few of them were focused on potential terrorist attacks on their chemical use/storage facilities.

Unique Issues

Many of these facilities are going to have their own unique problems with CFATS implementation and enforcement activities. Commercial facilities might have to worry about how to deal with personnel surety programs applicability to their customers. Distribution facilities might have to worry about meeting a variety of requirements with a very limited manpower pool. Many of these facilities have space issues that make it difficult to comply with detect and delay requirements.

Each of these facilities is going to have to consider their own situation and how they are going to apply the CFATS risk-based performance standards in formulating their security response to the threat of a potential terrorist attack. In many instances the ‘normal’ security responses are not going to work and the facility security management team will have to come up with unique and creative responses of their own.

I really want to hear about these security challenges (and potential solutions). One of the reasons is that I love an intellectual challenge. I enjoy thinking about these situations and coming up with my own ideas of how they can be dealt with. And I obviously enjoy sharing my thoughts and observations (or else I wouldn’t be writing this blog). I am not vain enough (not quite) to think that my thoughts are going to solve all of the problems of the industry, but they might spark some thoughts or observations that had not been considered by the facilities having problems.

More importantly, my readers include a large number of people with a wide variety of security experience and backgrounds. Many of them are willing to comment on my expressed thoughts and observations with corrections and new ideas of their own. Again the additional eyes may see or know something that might be helpful in solving the problems.

Another important part of the audience of this blog comes from Washington, regulators and legislators (and their staffs). These folks are typically focused more upon the large picture, but they need to hear the unique problems (and solutions) that the CFATS program engenders. It may come as a surprise to people in industry, but most of these beltway folks do really care about what happens with the regulated community and want their laws and regulations to work.

CVI Issues

I am well aware that individual CFATS facilities have to follow the DHS rules on the protection of Chemical-Terrorism Vulnerability Information and I certainly would not advocate sharing protected information with me. While I have completed the original CVI training program and have my authorized user number, I would have a hard time convincing a DHS Chemical Security Inspector that I generally have a need to know (This of course does not apply if I am doing specific consulting work for a facility, but there I would also be covered by non-disclosure agreements as well as CVI rules).

I also know that, according to Google, about 20% of my readership is from outside of the United States. Most of these readers have legitimate chemical security concerns of their own and are interested in seeing how we take care of our similar issues. Others are people that have to do business with CFATS affected facilities and need to understand the security constraints on their business. I must, however, consider the possibility that there may be a few readers that might want to use information from this blog to do harm to CFATS protected facilities.

So, generally I don’t want readers to send me information that might be covered by CVI rules. More over, I will be very careful that anything that I reveal will not be able to be used to be used to identify a specific facility, or provide information necessary to formulate a successful attack on a facility.

This is one of the reasons that I typically make this request to organizations that represent industries rather than individual facilities. This typically makes information that I receive once removed from the facility level and thus that much less likely to involve covered or sensitive material. That doesn’t mean that I don’t want to hear from facilities (I certainly do), but I don’t want to make anyone uncomfortable by seeming to want them to violate CVI rules.

Self-Censoring

There is one last point that I would like to make on this issue. While I enjoy writing about chemical security issues, I do not write about everything that I hear. I have had interesting discussions with a number of people in both the chemical facility and security industry side of the issue that I will never be able to include in the blog. Security issues and CVI issues are a concern in some instances. In other cases legal, commercial or political concerns restrict my ability to share the information. That is the way of the world.

I am certainly not going to restrict who I listen to because of these concerns. The information that I gain from these exchanges helps to inform me about a wide variety of issues and concerns. The information helps me to formulate my responses to a number of the other issues that I address in this blog.

I would like to note that if a reader wants to share information in confidence that they should make that clear up front. While I can usually spot CVI information fairly easily, the legal, political or commercially sensitive information is quite frequently harder to spot. But remember a blogger is at heart a reporter and a commentator. We exist to share and expound on information. I have done my best to respect confidences, but I do have to know about them to properly respect them.

So, if you have a unique concern about CFATS implementation or enforcement, please let me know. I or my readers may have ideas that might be of help to you. Or just explaining the problem to me may allow you to look at it in a slightly different light that allows you to come up with your own solution. And that’s the whole point, lets solve these problems.

Reader Comment 10-20-10 Safety as Security

Andrew Ginter, who writes the Control System Security Blog, posted an interesting comment on yesterday’s blog on how Stuxnet could be used against high-risk chemical facilities. The comment addresses a number of good detailed points, all worth reading, but one of the most interesting is summed up in one very important sentence:

“Safety systems need to start being evaluated from an adversary's perspective: is there a set of components, which if destroyed or caused to malfunction simultaneously, can cause a catastrophe?”

As facilities take detailed looks at their security plans and procedures it will quickly become apparent that many of the security actions that facilities take will be closely related to safety procedures that are already in place. The reason for this is fairly obvious, the release of hazardous materials into the environment is the ultimate goal of both programs. Safety programs are designed to prevent accidental releases and security programs are designed to prevent deliberate releases.

From a control system perspective, Andrew points out a very important point; “The worst consequence that a worm, or even an insider with access to the control system, should be able to produce is denial-of-service: triggering a safety shutdown.” But, this will only be true if it is not possible for the same cyber system attack to affect both the safety systems in place in the facility and the control systems.

Upper management needs to have this clearly explained to them. At one facility where I worked the computer safety systems and the standard control system were installed on the same computer, using the same control system software. Engineering had requested a separate control system for the safety systems, but corporate management deleted that system from the budget, noting that the current control system had more than enough capacity to handle both systems.

We designed in some manual safety protections into our processes, including manually locking out valves for chemicals that could present reactive hazards for the process. But it is hard to keep such operator-centric safety processes operational; it requires aggressive auditing. Unfortunately we could not come up with such manual processes to back-up all of the cyber safety controls.

As many writers have pointed out safety is an attitude as much as it is a program. The same is true for security. All of the safety programs or security programs mean little if there is not a facility wide attitude that these programs are important and should not be shortcut under any circumstances. When the twin attitudes of safety and security are present in a facility team, the interactions between these two programs will reinforce both.

Reader Comment 09-21-10 Tracking FAQs

Yesterday an anonymous reader commented on an older post where I described the new CFATS Knowledge Center web page. Along with cogent comments about my blog post (read – agreed with me) Anonymous had a valuable suggestion that I would like to pass on since I know that there are some DHS-ISCD readers of this blog.

Question Sorting FAQs

Anonymous wrote: “. I'd sure like to see the FAQs in the future contain a reference to the specific numbered question one would find in the information gathering processes of Registration, Top-Screen, SVA, and SSP; eg [Q:7.005-14425] from the SSP questions.”

The average CSAT user looking for information will typically be working on one of the many submissions that are the heart of the CSAT system. Their questions will most likely be keyed to a specific question that they are being to answer in those submissions. Being able to enter that Question Number as a search term will make the search for the information they seek that much easier.

Now I understand that the FAQ listed on this page do not cover all of the questions. Since there are only 422 FAQ currently listed and there are more like a bazillion questions in all of the CSAT tools, it is obvious that most questions don’t have an associated FAQ. This is actually a good thing; it means that the vast majority of the questions are written clearly enough that there are no questions about the information that ISCD is looking for.

Question Numbers on Questions

We can extend that suggestion to the web form that DHS provides for CSAT users to submit questions to the Help Desk. If the Help Desk Web Form were to be slightly modified to include a field for CSAT question numbers, it would help to facilitate question searching of the FAQ.

More importantly it would also make it easier to actually address the information need of the questioner. Very few people are trained to write effective questions. When those questions are asked live to Help Desk personnel on the phone, clarifying questions can be asked in return to allow for a more effective answer to the CSAT User. This is more difficult to do when the question is submitted via the web form. With the CSAT Question identified on the form, it will make it easier for the Help Desk personnel to accurately answer the question that is really being asked.