Reader Comment 08-13-09 No Weiss PhD

A reader from SCADASEC-L, an on-line SCADA Security discussion group, corrected my awarding of a Ph. D. to Joe Weiss in my blog posting yesterday. As the reader noted, I remembered him being called (or described as, I can’t remember for sure) Dr. Weiss at a Congressional Hearing, but that was not the sole reason that I used the title. I intended it to be a measure of the respect that I hold for Joe’s learned opinions on ICS security issues. Even so, I apologize for inappropriately using the honorific ‘Dr’ in attributing Joe’s comments. While I certainly appreciate the friendly, constructive tone that was used in this correction, I recognize that there are people in the world who get very upset about the misuse of such honorifics. I certainly don’t want to get Joe Weiss in trouble with such people. He already has his work cut out for him fighting the demons of inept security practices; he doesn’t need the added burdens of fighting the honorifics police (which certainly does not include the reader who made this comment). As always, I appreciate the constructive criticism and urge readers that note mistakes in my posting to let me know about them.

Weiss on Primer

Dr. Joe Weiss has a short piece at ControlGlobal.com on the DHS Primer Control Systems Cyber Security Framework and Technical Metrics report that I discussed in yesterday’s blog. While his piece is not a reply to my blog, it does make some interesting observations that bear repeating. Since he is much more knowledgeable on control systems security matters than I ever hope to be I thought that I would use the benefit of his knowledge to extend yesterday’s discussion. Accidents not Attacks First he points out that, while “most control system cyber events are incidents not attacks”, the “the Primer is focused on malicious IT-type attacks”. This is a common problem that DHS faces in many of its endeavors. Its mission is to protect the nation from ‘malicious’ attacks from within or without our borders. The fact that more people and infrastructure are damaged every year from incidents due to stupidity, poor design, inadequate attention to detail, or just plain sloppy execution than have been hurt by terrorists (foreign or domestic) in the last 10 years, has little bearing on the DHS outlook. Their job is to squish terrorists. Most industries in this country look to OSHA or EPA to regulate the prevention of ‘accidental’ injuries in the work place. Unfortunately, neither organization has done much to look at the effects of inadequate regulation of industrial control systems on personnel injury rates or unintentional environmental releases of hazardous chemicals. Given the speed (or lack thereof) with which these two organizations have responded to recommended changes in dust control or control of reactive chemistry, it will be decades before they address cyber control issues. Security Knowledge Joe then goes on to say: “There are very few people that actually understand control system cyber and most are not in the security group.” He makes a good point here. If the corporate security people have an IT background not a control system background (the normal course of events) the problems that they will project into the system by employing typical IT security procedures will be tremendous. Any high-risk chemical facility that uses an ICS must have a control systems engineer on the security team. Someone who truly understands the ins and outs of the control system will be very valuable to the security team even if that engineer has little or no IT security training. What would be even more valuable would be to have someone trained in control system security, but as Dr. Joe has mentioned in a number of his blog postings, there is no such training program yet in existence. Maybe CERT should start providing grants for a cyber security chair at Cal Tech, Ga Tech, and MIT. That would start catching industry attention. Legacy Control Systems Joe’s last point is that the Primer “does not recognize the unique issues with legacy control systems”. Control systems are expensive and have steep learning curves, so they are not replaced just to keep up with generational changes in the marketplace. Only complete system failures, new production requirements, or changing enterprise software requirements will typically lead to installation of new systems. Even then, while the computers used to host the ‘system’ are likely to be updated to handle the capabilities of new software, the outlying sensors and controls usually remain in place until they fail. The ‘legacy control systems’ issue is related to the fact that most of these older systems have only limited security capabilities. There was little need for security programming when these systems were developed because there was no intention to connect them to the Internet or to allow wireless communications connections. Joe sums up the problem by noting that “Many systems cannot take complex passwords. Many systems simply cannot be patched expeditiously, if at all.” Better than Nothing While Dr. Weiss makes many important points in this short piece, and would certainly be able to provide more examples of short comings with this Primer in a longer piece, I still think that Primer provides an important start in establishing the idea that cyber security is a process. As with any process that industry uses, it should be subject to continuous improvements and that requires a measurement system to evaluate if changes made to the system actually result in improvements. This Primer provides a start for establishing such a measurement system. As we inevitably begin to churn control system security specialists out of our educational facilities more work will be done on developing and improving the metrics for tracking system security issues. The widespread adoption of a system like that outlined in the DHS report will be an important first step in that improvement process.

Public Health Response to Chemical Incidents

The Rand Corporation recently published a technical report that they prepared for the Department of Health and Human Services (DHHS), Public Health Preparedness and Response to Chemical and Radiological Incidents. The report looks at potential requirements for responses by local public health services to incidents involving the release of chemical or radiological contaminants. The report notes that these releases can come from a number of sources, including: chemical and nuclear plant accidents, train collisions, product tampering, and chemical terrorism” (pg xi). The authors conducted literature reviews, looked at chemical and radiological incident after action reports and conducted interviews with public health professionals from around the country. They identified a number of public health actions that formed key responses to these types of incidents. They then looked at these actions to see which ones are supported by developed, published and well understood ‘practices’. The report identifies four practices that need further development work. They are (pg xii):

“Conduct initial epidemiological investigation; “Provide public information; “Establish a victim registry and monitor long-term health; and “Monitor health conditions at shelters and mass care centers.

Conduct initial epidemiological investigation The report notes that in “the case of a chemical or radiological release that is initially unrecognized or poorly characterized, public health departments will need to use epidemiological techniques to deduce information for understanding the type, time, and/or location of the release” (pg 10). This would be most important where a deliberate release was made covertly, but it could be necessary where there was a non-catastrophic accidental release that was not detected before the medical affects were noted. The problems in this area for chemical releases deal with the initial identification that the medical problems are caused by a chemical exposure and determining which chemical is responsible for the exposure. This may be further complicated by slow on-set symptoms and the similarity of the symptoms to normal disease processes. Provide public information Once a chemical or radiological release event has occurred the public health services may be responsible for risk communication functions, including “providing instructions to individuals about sheltering or evacuating based on where they are located, how to decontaminate, what symptoms to monitor, how to determine whether medical attention is necessary, and potential treatments that may be provided” (pg 10). While actual communications may be handled through the incident command center, it will be input from public health professionals that guide those communications. Establish a victim registry and monitor long-term health Medical care for acute trauma or medical affects of chemical or radiological exposure during the incident is the responsibility of the emergency health care system. The chronic affects of the exposure will be more difficult to manage. This will require the “the creation of a registry about potentially exposed individuals including, for example, contact information, location at time of incident, duration of the exposure at that location, and symptoms” (pg 11). Monitor health conditions at shelters and mass care centers Public health professionals already have a long list of responsibilities for monitoring conditions at evacuation shelters and emergency care centers during any large scale evacuation situation. For chemical and radiological exposure events an additional concern “is monitoring contamination levels at shelters to ensure that they remain safe” (pg 12). This includes meteorologically mediated exposure (normal drift of the contaminant cloud) and contamination inadvertently brought into the shelter by the evacuees. Additionally, exposure symptoms with a lengthy time delay require constant monitoring of the shelter population. CFATS Implications High-risk chemical facilities with toxic release COI have a special responsibility to coordinate in advance of an incident with public health authorities about the chemicals on-site that could have significant off-site affects in the event of a successful chemical attack. Public health officials would then be able to plan for their response to such a terrorist attack. These plans would include:

Identification of symptoms of exposure. This would allow for education of emergency medical technicians and emergency room personnel on the appropriate response required for exposed patients, including initial support and stabilization needs and decontamination requirements.

Provisions for chemical monitoring. This would allow for pre-stocking emergency rooms, ambulances and shelters with appropriate devices for monitoring for chemical contamination of patients and evacuees.

Identification of appropriate treatment options. This would allow for appropriate training of local medical professionals about the most effective diagnostic and treatment options for chemical exposures that result from the successful terrorist attack.

Identification of chronic health affects. This would allow the public health agencies to plan for the establishment of exposure registries and identify what follow-up actions would be necessary.

Reader Comment 08-10-09 QHSR

Monday evening Laurie, a reader from University of Findlay (Ohio) and active in the maritime security field, added her comments to my blog on the completion of the first QHSR Dialogue. She made three interesting points: #1: The “website was evidently not sophisticated enough to accept a cut-and-paste from Word.” There are a number of people that routinely use off-line word processing programs (most commonly Word® as I am sure the people in Seattle are proud to note) to formulate postings to public web sites. I certainly count myself in that number. It aids in the processes of refining and editing ones work and making sure that the idea is being recorded in a recognizable manner. The lack of editing tools like ‘spell check’ and ‘grammar check’ certainly contributed to some of the more confusing ideas posted to the site. It is really a shame when people, myself included, think that a web site managed by the National Academy of Public Administration is ‘not sophisticated enough’. But, there were so many operational problems associated with this website, that DHS needs to reconsider taking the web site in-house. The team that does the work on the CFATS tools (for example) could do a much more professional job. I know that there was some initial concern that sharing even marginally personally identifying information with a DHS registration could stifle some of the participation. The amount of information on each poster openly displayed on the NAPA run site, however, would certainly allow for ‘swift DHS retaliation’ if such were actually in the plans. #2: “I wish we could have had some input at an earlier stage - I'm not sure the mission areas reflect the depth of DHS' many functions.” Laurie was not the only person that I heard this complaint from. The six discussion topics were overly broad and provided a great deal of overlap between the many functions found in DHS. Furthermore, there was nothing to tie the topics back to specific functions within the Department. Thus, Laurie could find nothing concerning MTSA issues and I could find no reference to chemical facility security. I understand that the focus of the Dialogue is on ‘mission, goals and priorities’, but according to the Dialogue Flyer the QHSR is a “congressionally-mandated, top-to-bottom review of Homeland Security policies and priorities will guide the Department and the nation for the next four years.” If the participants in the Dialogue cannot not find references to the programs that they work with on a daily basis, then this is not a ‘top-to-bottom review’. At best it is high-level review of the guiding philosophy of the Department. That would be better than no public participation, but it is not what was advertised. #3: “I thought it amazing that no definitions were offered for key terms.” I’ll defer to Laurie on explaining the importance of this comment:

“’Operational’ can [mean] one thing to a federal agency and quite another to local law enforcement. And in a profession where it's a fun ice breaker to ask people what they really mean by ‘homeland security’, and the regulations we all live and die by are incomplete without definitions sections, I am deeply uneasy relying on what I THINK a term means. Definitions don't impede the conversation; they give us focus and clarity.”

While I am generally supportive of the QHSR Dialogue, I do hope that the Department and NAPA take a good hard look at the format of their discussion and how that plays into the interaction of ideas that they are looking for. The Study Groups will go back and look at the details of the ‘ideas’ presented and discussed in the first Dialogue. I hope that the site managers use the same type of ‘iterative process’ to refine the background for the Dialogue.

Control System Security Metrics

A key component in any attempt to ‘improve’ a system is the ability to measure change in system characteristics. It is only thru such measurements before and after a system change that an objective analysis of improvement can be made. The DHS Control Systems Security Program (CSSP) recently published their latest version of their suggested metrics for measuring changes in relative security of industrial control systems (ICS), the Primer Control Systems Cyber Security Framework and Technical Metrics (Primer). Dimensions of Cyber Security The CSSP Primer describes a ‘framework’ for making decisions about ICS security. It identifies seven control systems cyber security dimensions that “capture many of the system attributes, which correlate with a control system’s risk exposure” (pg 1). The CSSP uses the term ‘dimension’ because they describe “an important aspect of the control system’s cyber security posture at a given point in time” (pg 2). Those seven dimensions are:

“Security group knowledge “Attack group knowledge “Access “Vulnerabilities “Damage potential “Detection “Recovery”

Detailed definitions of the terms can be found in the Primer (pg 2), but most of the dimensions are relatively easy to decipher just from their titles. The two ‘group knowledge’ dimensions are a little less clear than the others. The ‘Security group knowledge’ looks at looks at how easy it is for changes to be made to the ICS without the knowledge of those responsible for the security (the ‘Security Group’) of the ICS; supposing that such surreptitious changes could be used to gain control of the system. The ‘Attack group knowledge’ dimension looks at how easy it would be for an attacker to gain information about details of the system that would allow for a deliberate, knowledgeable attack on the system. ICS Security Metrics The security ‘dimensions’ help to define the framework for ICS security measures, but they do not provide any direct measures that will allow a facility to track the ‘performance’ of their security systems. The Primer notes that there are a number of different metrics that have been developed by people in the industry and provide the following list of references (pg 24) as examples:

“E. Chew, A. Clay, J. Hash, N. Bartol, and A. Brown, Guide for Developing Performance Metrics for Information Security, NIST Special Publication 800-80, May 2006.

“R. Ross, S. Katzke, A. Johnson, M. Swanson, and G. Rogers, “System Questionnaire with NIST SP 800-53: Recommended Security Controls for Federal Information Systems,” Technical Report, NIST, References and Associated Security Control Mappings, Gaithersburg, Maryland, March 2006.

“M. Swanson, N. Bartol, J. Sabato, J. Hash, and L. Graffo, Security Metrics Guide for Information Technology Systems, NIST Special Publication 800-55, National Institute of Standards and Technology (NIST), Gaithersburg, Maryland, July 2003.

“M. McQueen, W. Boyer, S. McBride, M. Farrar, and Z. Tudor, "Measurable Control System Security through Ideal Driven Technical Metrics", S4: SCADA Security Scientific Symposium, January 23, 2008”

The Primer authors note that to be effective there should be at least one metric for each of the ICS security dimensions listed in the Primer. They recommend (pg 5) that the following 10 metrics (and associated dimension) should be used to track changes in the security posture of ICS:

“Rogue Change Days (Security Group Knowledge); “Security Evaluation Deficiency Count (Security Group Knowledge); “Data Transmission Exposure (Attack Group Knowledge); “Reachability Count (Access); “Attack Path Depth (Access); “Known Vulnerability Days (Vulnerabilities); “Password Crack Time (Vulnerabilities); “Worst Case Loss (Damage Potential); “Detection Mechanism Deficiency Count (Detection); and “Restoration Time” (Recovery).

According to the authors, each of the metrics listed above is an answer to one basic security question: “What can be objectively measured on the system that is a reasonable representation of how nearly the system approaches the ideal of its associated control systems cyber security dimension?” The Primer provides a detailed technical discussion of each metric including a range of possible values and an ‘ideal’ value. While ‘ideal’ value may not be achievable, the measurable values do provide a tool for tracking progress or assessing the efficacy of security measures. Case Studies The Primer does provide two examples of how the security dimensions and metrics were used in actual practice in two different case studies. The first such study will be of primary interest to the chemical security community since it took place at a chemical facility with a DCS ICS. Unfortunately the details associated with the case studies are minimal and there is little discussion of how the metrics were applied. There is more detail for the second case study which looks at the more politically sensitive case of an electric power distribution SCADA system. Standards for Metrics What is missing from the discussion of the metrics is the definition of an ‘acceptable’ value for the metric signifying that the security measures are ‘adequate’. There is a brief mention of ‘target’ values in the two case studies. The authors describe the suggested target value as “the value that could be obtained by changing the system configuration to improve cyber security while retaining required functionality” (pg 20), but provide no information on how the value was set and use different values in the two different case studies. While not stated in the Primer, the reason for the lack of definition of acceptable values is at least partially based on the fact that such a decision is at its most basic a management evaluation of the risk and the cost of security. Where legal standards are set (and that has not yet been done for ICS) for acceptable values then achieving those standards becomes a matter of compliance not security. Recommendation The CSSP should be commended for developing this document. It is a valuable addition to the cyber security debate. It is, however, probably not an adequate amount of information to develop and implement a metric based ICS security monitoring program. It assumes a relatively high level of knowledge about industrial control systems and computer networks. If CSSP provides some training sessions (on-line and on-site) to support this framework, it will be much more accessible. Having said that, facilities that use an industrial control system need to get a copy of this document and review it in detail. High-risk chemical facilities in particular should pay special attention to these metrics in developing their cyber security programs.

Reader Comment 08-05-09 SSP Webinar

Last week a reader, Jesse Maruschak, posted a question in response to my blog on the DHS/ChemITC webinar on the Site Security Plan. Jesse wrote: “Could you please post information on how we could become a participant in the next informational webinar, I would appreciate it”. It took a little longer than I had hoped to confirm some information, but here is the answer to Jesse’s question. First, DHS is giving priority for the webinar slots to facilities that have been notified to complete the SSP. The facilities that have received their notification, should receive an email, through their designated Submitter, at about the same time that they receive their notification letter. That email will include, among other things, an explanation of how to sign up for the webinar. If that email has not arrived within a couple of days of the notification letter, the Submitter should contact the CSAT Help Desk (866-323-2957). Other requests for webinar participation on a space available basis should be made by email to CFATS@DHS.Gov. The email should probably include a brief explanation of why participation is being requested and point of contact information (phone and email) of the person making the request. I would be willing to bet that DHS would be willing to work out special presentations (similar to the ChemITC presentation that I participated in) for industry groups. That does not mean that the presentation would be tailored to the particular industry (but you could certainly ask), but that the industry group could coordinate for the registration of a number of their members, instead of everyone trying to make that connection on their own. Finally, I would like to reiterate something that I suggested in the earlier blog. If a facility is getting ready to do their SSP, having the entire team watch this presentation in a facility conference room with a single log-in would probably be the way to go. I’ll add to that now that there is apparently a copy of the slide presentation now available on the Chemical Sector Security Summit web page (though I certainly don’t expect that DHS will update that set with any changes that might be made to the presentation). Making copies of that slide set for each participant might make it easier to take better notes. Hope that helps, Jesse. Again, sorry it took so long to get back with the information, but as I understand it DHS is doing these presentations almost weekly.

Common Control System Vulnerabilities

Last month the Department of Homeland Security (DHS) National Cyber Security Division’s Control Systems Security Program (CSSP) released a report on the results of 15 control systems assessments that the CSSP has conducted since 2004. With only 15 assessments this is hardly a comprehensive look at industrial control systems (ICS) cyber vulnerabilities, but it does provide a window into the typical problems found in control system applications. These assessments were not done on live installations of control systems. The assessments were done using techniques such as “reviewing the production system network diagrams and firewall rules, and performing a hands-on assessment of a duplicate nonproduction installation of the system” (pg 1). “This controlled environment allows realistic assessments of systems and components without the adverse consequences resulting from potential system failures.” One of the scary aspects of this report is found on page 5 in the “Impact on ICS Security” section of the report:

“Although not all findings have been addressed, most systems have been modified to improve security based on assessment reports. After-action validation of mitigations to identified security flaws are performed by the CSSP assessment team to help ensure the security assessments are successful in increasing critical infrastructure security. Some of the vendors have been forthright in sharing the results with their customers, and some have felt that any disclosure of vulnerabilities could lead to exposure of their customers to potential cyber attacks.”

I’m not sure what disturbs me more the ‘not all findings have been addressed’ or the ‘some of the vendors have been forthright in sharing’ comments. The combination of the two should be enough to convince people in the chemical security community that there is a significant problem with the security of industrial control systems that are such an integral part of so many high-risk chemical facilities. Types of Vulnerabilities The report loosely groups assessed vulnerabilities into “nine general security problems that sum up the main weaknesses that ICS products and installations are prone to have due to legacy code, lack of security training and requirements, and ICS operational requirements” (pg 9). The top three (by % of assessment findings) problem areas are (pg 7): poor network protocol implementations (26%), information disclosure (21%), weak authentication (18%). The common vulnerabilities found in the network protocol implementation category include (pg 8, Table 1):

“Lack of input validation: Buffer overflow in ICS service “Lack of input validation: Lack of bounds checking in ICS Service “ICS protocol uses weak authentication “ICS protocol uses weak integrity checks “ICS product relies on standard IT protocol that uses weak encryption”

The common vulnerabilities found in the information disclosure category include (pg 8, Table 1):

“Unencrypted proprietary ICS protocol communication “Unencrypted nonproprietary ICS protocol communication “Unencrypted services common in IT systems “Open network shares on ICS hosts “Weak protection of user credentials “Information leak through unsecure service configuration”

The common vulnerabilities found in the weak authentication category include (pg 8, Table):

“ICS uses standard IT protocol that uses weak encryption “Use of standard IT protocol with clear-text authentication “Client-side enforcement of server-side security “Improper security configuration “No password required “Weak passwords “Weak password requirements”

Control System Security Debate This document is a valuable addition to the cyber security debate. Industrial control systems are used in a wide variety of settings, but they are of special interest to the chemical security community. ICS are a key component of most chemical manufacturing systems and even purely distributional facilities frequently use these systems to control loading and unloading operations. Security vulnerabilities in the ICS must be seen as potential avenues for attacks on high-risk chemical facilities using those systems. The current Site Security Plan implementation does address the issue of cyber security, including security issues associated with ICS. It does not include the same level of ICS assessment described in this assessment. This is certainly reasonable given the high level of complexity of the existing SSP, the limitations imposed by the authorizing legislation, and the level of expertise required to do these assessments. But, sooner or later, detailed assessments of security of cyber control systems will need to be done at the high-risk chemical facilities covered by CFATS.