Reader Comment – AWWA Guide Available to Non-members

Kevin Morley, the Security & Preparedness Program Manager for the American Water Works Association, left a nice comment on my post from last week about their control system security guide. He noted that:

“Access to the AWWA guidance and use-case tool do not require membership in AWWA. These resources are freely available to everyone. Access does require creation of a user account, which simply confirms that the user accepts the terms of use.”

This is certainly good news for water systems that are not members of the AWWA. It also means that folks with control systems in other types of critical infrastructure have a tool that can be used to look at their control system security.

There will be very few things in the AWWA tool that are not applicable to other organizations. It might not look at all aspects of control system security for other types of industries, but lacking this kind of detailed guide from anyone else, it would certainly be a good start.

Control System Security Guide

There is an interesting blog post by Bridget O'Grady over at SecurityNotes.asdwa.org about a new control system security program being introduced by the American Water Works Association (AWWA). Based at least in part on the recently published Cybersecurity Framework (CSF), this voluntary program for water treatment facilities looks like an interesting attempt at making the CSF usable.

There are two main components of this program, a Cybersecurity Guide and an on-line Cybersecurity Guidance Tool. Unfortunately for most readers of this blog, the tool is only accessible to members of the AWWA.

Cybersecurity Guide

There are three main parts to the publicly available guide:

• Recommended Cybersecurity Practices;

• Cybersecurity Guidance Tool; and

• Cross Reference to NIST Cybersecurity Framework

The recommend practices section gives a overview of the broad sweep of cybersecurity practices including definitions of some key terms. It addresses twelve important areas of cybersecurity:

• Governance and risk management;

• Business continuity and disaster recovery;

• Server and workstation hardening;

• Access control;

• Application security;

• Encryption;

• Telecommunications, network security and architecture;

• Physical security of PCS equipment;

• Service level agreements;

• Operations security;

• Education; and

• Personnel Security

Table 2-1 in the Guide provides a slightly more detailed listing of the various components of the above listed category. All of this is written in the broadest language and is hardware and software non-specific. While some of the wording used applies specifically to water treatment systems, there is nothing here that could not generally be applied to any industrial control system.

Cybersecurity Guidance Tool

While the tool itself is not available to the public, there is a good description of how the tool works and how to use it in the Guide. It employs a check-list type approach to allow a facility to describe its control system. For example, under system architecture there are three check boxes (and more than one box can be checked):

AR1: Dedicated network: All network and communications infrastructure is dedicated exclusively to SCADA. No connections to enterprise networks.

AR2: Shared WAN: Wide-area network communications infrastructure is shared (controls: physical (media) separation, VPN, VLAN, firewall).

AR3: Shared LAN: Local-area network communications (within facility) is shared (controls: VLAN, firewall).

Each of these selected boxes is described as a Use Case. Once the system architecture is described, the tool provides a list of Recommended Controls for each of the selected Use Cases. Readers who are familiar with the CSF will recognize the general format of these Recommended Controls as it references back to various established standards using both the standards listed in the CSF and some additional standards more directly applicable to control systems (DHS DID: DHS Recommended Practice: Improving Industrial Control Systems Cyber Security with Defense-In-Depth Strategies) or water treatment facilities (ANSI/AWWA G430-09: Security Practices for Operations and Management).

The Recommended Controls are provided in four different priority levels starting with the minimum accepted levels of security for SCADA/PCS (Priority 1 Controls) and ramping up to the most complex controls that are targeted at preventing the most sophisticated attacks (Priority 4 Controls). The description of the use of these various priority levels seems to be more targeted on an implementation.

Cross Reference to CSF

Appendix A provides a tabular cross reference of these suggested security controls back to the Appendix A table in the CSF. Unfortunately they used the August 28^th, 2013 draft version of the CSF for their table so it does not exactly match up with the table in the final version of the CSF. Given that this was published within a week of the final version of the CSF I can understand why this choice was made. It would have been nice, however, if the authors had been able to access a more up-to-date version of this table, but such is life.

Commentary

This actually looks like a very useable process and the AWWA is to be commended, not only on the thoroughness of the effort, but on the speed with which it was done. They obviously relied on a lot of the public work that was done by NIST during the development of the CSF.

There is one slightly negative thing that I do have to say about this effort. This program is a management program not a technical program. It is a valuable tool to provide management with a set of techniques to oversee the establishment and maintenance of a control system cybersecurity program. It is not, however, an actual guide on how to secure a specific control system.

Granted it would not be possible to write a single useable document to the security of the wide variety of control systems in use even in the relatively limited area of water treatment. But management must realize that they are still going to have to rely on the judgment and skills of their control system staffs and contractors to actually put the controls into place and make them work on a day-to-day basis. And if management is not willing to ensure that those employees and contractors have the necessary skills and tools to accomplish those tasks, no level of ‘compliance’ with a tool such as this will provide any kind of cybersecurity for their organization.

Water Security Congress

On Monday, the ASDWA Security Notes Blog posted information on last month’s 2010 Water Security Congress conducted by the American Water Works Association (AWWA). While most of this meeting was focused on security issues that did not specifically deal with chemical issues at water facilities, there were a couple of presentations that might be of interest to the general chemical security community. For readers from the water community may want to look at the general AWWA page on the meeting.

Water ISAC

There was an interesting presentation made by Aaron Levy discussing the Water Sector Information Sharing and Analysis Center (WaterISAC) that he heads. This is an organization run by the Water Sector Coordinating Council, part of the National Infrastructure Protection Program. It provides an information sharing environment for all sorts of water sector protection activities including a secure portal for sharing intelligence information. I have not seen a comparable organization coming out of the Chemical Sector Coordinating Council.

One of the interesting issues raised in Levy’s presentation is the problem of financially-motivated vandalism. Water facilities have a security issue that is much more prevalent than terrorist attacks, the theft of metal (pipe and wire) from their extended facilities; a problem aggravated by the poor economy.

Stuxnet was briefly addressed in the presentation both as a threat to water system control systems, but also as a threat to the electrical power supply critical to the operations of these systems. This is an issue that should also be of concern to chemical facility operators.

I want to take an opportunity to commend Mr. Levy for the format of his presentation file. Most presenters provide copies of the presentation slides for these type files. The WaterISAC presentation not only includes the slides, but the notes that he used in his presentation. This provides a lot more information and provides more of a flavor of the actual presentation. Still not as good as a video file, but Levy is to be commended for taking this innovative step.

Utility Security

This presentation by John W. McLaughlin, from Jacobs-JJG (engineering firm), looks at the switch from security being a counter-terrorism matter to a more inclusive ‘all-hazards’ approach. He notes that many utilities do not see themselves as a terrorist target and may see counter-terrorism measures as a diversion of funds that could be better spent elsewhere.

He notes the enlarged focus of security at the national level is now looking beyond just the possibility of terrorist attacks and is addressing all sorts of security issues. A major new focus at the national level is ‘resiliency’, the ability to get the water system up and running after an attack, vandalism, systems failures, or natural disaster.

This discussion about the need for resiliency is certainly an increasingly important focus at DHS. Prevention of production disruptions from all causes is important, but resilient facilities recognize that all disruptions cannot be prevented. I’ve noted for some time in this blog that recognition of the practical inability to prevent all terrorist attacks requires planning for emergency response to deal with resulting chemical releases, fires and explosions. Resiliency looks even beyond that, at what happens after the emergency response is done.

While chemical facility management certainly has an interest in getting their facility back on line, they don’t have the same urgency in this matter that public utilities have. An off-line chemical facility will not typically cause life changing problems for all of their neighbors and customers. A shutdown water plant, on the other hand, adversely affects the entire community until it gets back on-line.

Water and CFATS

Bryon O. Elwell, from ABS Consulting, provided an overview of both the current CFATS regulations and the pending legislation in Congress that might affect security operations at water and waste-water treatment facilities. The information in the slides was well put together, especially the summary data on S 3598, the Secure Water Facilities Act.

Elwell presented some facts that I haven’t seen pulled together before. He noted (slide 18) that there were 1,200 waste water treatment plants that had more than 2,500 lbs (CFATS SQT for Chlorine) of Chlorine gas on-site and 1,700 water treatment plants that were covered by EPA’s risk management plan rules (and presumably would be covered by CFATS rules). He also provided (slides 19 and 20) a more complete listing of water treatment chemicals that are DHS chemicals of interest (COI) than I have seen to date.

Water Cyber Security

I was pleased to see that there were three separate presentations on control system cyber security (CSCS) matters; all of them should be reviewed by any organization with an automated industrial control system. The first was by Candace Chan-Sands, from EMA, Inc. who’s presentation looked at the DHS ICS-CERT Cyber Security Evaluation Tool (CSET). This is a service provided by DHS ICS-CERT to evaluate the security environment for a facility’s control system. Some how I have overlooked this service listed on the ICS-CERT web site; I’ll cover this in more detail in a separate blog post.

The second presentation was by W. Michael Sutton, an engineer with Malcolm Pirnie Inc. He looked at both the development of the ISA-99 cyber security standards under development. He pointed the audience at ANSI/ISA-99.02.01-2009, the Security for Industrial Automation and Control Systems: Establishing an Industrial Automation and Control Systems Security Program, published in 2009. He did not provide a link to the ANSI site for obtaining the document; here it is. He also provided a brief look at control system design and technologies available to help secure cyber assets.

The third cyber-security presentation was made by Jeff Mills, from Coalfire, an IT Audit and Compliance Management firm. This is a pretty good, if high-level, review of the control system security problem, but it does not address the peculiar ICS issues with implementing many IT security measures. There is, however, a very good description (slide 25) of general control system security measures that would apply to any facility using ICS systems.

Posting Presentations

I always appreciate it when organizations like the AWWA post their presentations on-line. I don’t have the travel budget to get to these meetings and there is a wealth of good information presented at meetings like this. Most people in the public water supply industry have similar budget constraints making the posting of these presentations a valuable service to the industry. I would like to make my standard presentation recommendation to AWWA; next year, how about providing videos of the presentations on-line.

Personal Complaint Warning: I am always upset when an organization posts .PDF documents with excessive security settings. Why would anyone be concerned about someone copying and pasting from the document. It makes the job of reviewers like myself that much more difficult. This is especially true when the document includes URL’s, its bad enough that the listed URL’s were not live links, but then they were protected against copying. What a PAIN! Please re-think your document security policy.

Water System Lobbying

I ran into an interesting blog post at ISAQQA-WUC.Blogspot.com about efforts of member utilities of the American Water Works Association to oppose passage of legislation that could allow the imposition of changes in disinfectant chemicals on water treatment and waste water treatment facilities.

The AWWA has opposed bills like the House passed version of HR 2868 and the recently introduced S 3598. They are now calling for members to contact their Senators to ask them to support the version of HR 2868 ordered to be reported by the Senate Homeland Security Committee. This very long blog post provides an interesting and detailed discussion of how utilities, which have access to little money for lobbying efforts, can best conduct a campaign to influence their legislators. Anyone intending to conduct a low-cost campaign to influence legislation should review this blog post before they seriously start to plan their own campaign.

Chemical Security Situation 

Of more interest to readers of this blog is the background information on chemical security issues at water treatment facilities. There is a very good summary of the current requirements for security and the types of chemical security responses that many facilities have taken. Careful reading of this section of the blog shows why there really does need to be some sort of water treatment facility language in any comprehensive chemical security language that would extend current CFATS rules.

First the posting explains that current rules require these facilities to conduct a security vulnerability assessment (SVA) and emergency response planning (ERP). There is no standard for the SVA or ERP and there is no real requirement to establish and execute a security plan to protect the vulnerabilities identified. Finally there is no authority for state or federal regulators to review the SVA or ERP or allow them to require correction of deficiencies.

Like various chemical industry organizations the AWWA also likes to brag on the efforts that have been made at member facilities to protect chemicals like chlorine gas. The main difference here is that there is no mandatory security program that member facilities have to adhere to. All of the programs that the AWWA describes here are completely voluntary and this is reflected in the following statement made in the blog post: “Most [emphasis added] have restricted access and enacted other measures to secure critical assets, including chemical supplies.”

One last time I would like to point out that no outside agency has the authority to determine if the ‘restricted access’ and ‘other measures’ provides an adequate degree of security for the facilities that have taken such measures. And no government agency can require the facilities that have not take such measures to do so.

Remove CFATS Exemption 

If the AWWA really wants to preempt efforts in the Senate to require consideration, and a possible mandate to implement, disinfectant substitution they may want to consider finding some Senator to submit a proactive floor amendment to HR 2868 that would remove the current CFATS exemption for water treatment and waste water treatment facilities. If the AWWA were to craft such language to include provisions for denying the Secretary the authority to shut down water facilities and included a grant program for security measures at public treatment facilities, there would be little reason for water facilities to object to CFATS inclusion.

Facilities that have actually addressed security issues would get credit for those efforts in implementing CFATS SSP requirements. Only facilities that have done little to protect the public from the consequences of a potential terrorist attack would expect to have to take extensive actions to get up to CFATS standards.

The benefit to the AWWA and its members would be that they could then point at the CFATS coverage as a reasonable alternative to S 3598 and its included IST measures. While the activists that would like to see chlorine gas removed from all of these facilities would not accept that argument, it is likely that many Senators would accept the increased security provided by CFATS requirements as an adequate first step to protecting the adjacent communities.

AWWA and CFATS

The last two days have had blog posts about pro-IST lobbying campaigns that are using mass email and internet petitions to influence Congress to pass the legislation that they favor. Monday, on one of AWWA regional web sites, they urged their members to contact their Representatives and express their anti-IST views on the proposed legislation. AWWA Position The American Water Works Association is opposed to mandatory IST provisions in CFATS reauthorization and is not too happy to see that their exemption from the CFATS rules may be removed. The AWWA takes the position that ‘water treatment’ decisions (the methods and chemicals used) should be made locally. They are also concerned that there could be public health consequences “if homeland security concerns are made paramount to protecting public health”. New Information on Water Facility Provisions The web site provides some new (to me) information about how the bill being drafted in the House Energy and Commerce Committee (there is no discussion about the Homeland Security Committee’s work on the CFATS reauthorization bill) will address water treatment facilities. According to the web site: “A utility can apply for an exemption from this [IST] requirement under certain criteria to be spelled out in the bill, but the final decision will rest with EPA in Washington DC. There is no plan in this program for states to assume primacy and administer the program.” This would be a major change from the way that the CFATS reauthorization was written in HR 5577 last year. That proposed bill would have left all decisions about chemical security at water treatment facilities under the control of the Secretary of DHS. This change, if it actually appears in the committee draft of the bill, may be part of the coordination of efforts between the two committees. This would allow for more oversight by the Energy and Commerce Committee due to their coverage of the EPA. AWWA Method of Influence We looked at the two pro-IST efforts and how they intended to influence Congress. The AWWA site recommends that their members “call, fax or e-mail their members of Congress immediately” to express their views. They also note that most members of Congress will be in their district for at least a part of the Easter Recess. The site does provide a form letter that AWWA members can use. In the earlier discussion of mass mailings I made the comment that, to be effective, mass mailings needed very large number of form letters (or emails) to be delivered to be effective. The AWWA effort does intend to produce any where that large a number of mailings, but this is not really a mass mailing campaign. What they are attempting to do is to make Congress aware of how the proposed bill will affect facilities within their district. If most water treatment facilities in the district convince their Representative that the bill would have adverse effects on the efficiency or operation costs for their facility it might influence on of our fence sitters to come down on the side of the facility. The AWWA letter does not address the cost issue at all. In my opinion this is a major mistake. In the current economic environment adding costs to any publicly owned facility. AWWA had promised to provide to their members a program to evaluate the cost of changing from chlorine gas disinfection to bleach or ozone disinfection processes. Since costs of switching are not addressed I assume that the AWWA has not made this tool available. The one thing that the AWWA recommends that their members do is to invite their Representative to come and see their local facility. This would provide the water works personnel a chance to make a one-on-one appeal for avoiding the mandatory IST provisions of the CFATS reauthorization bill. This is a smart move. Providing a first hand opportunity to see the complexity of the water treatment process would make their point easier to make. The AWWA web site is not as fancy as, nor does it provide the story telling aspects of the two anti-IST programs I have already looked at. It doesn’t need to. The people that they are talking to already understand the problem and don’t need to be convinced. The one thing that is lacking is that it does not provide much in the way of tools for their members to use.

CFATS Reauthorization and Water Facilities

One aspect of potential CFATS reauthorization legislation that we haven’t looked at much since the 111th Congress came into session has been the issue of the exemption for water treatment and waste water treatment facilities. This last week 130 members of the American Water Works Association (AWWA) flew into Washington, DC to talk about this issue, among others, with their elected representatives. According to a press release from the AWWA, these local leaders made the following points about their facilities and any new chemical security legislation:

“Allow decisions about disinfectant choices to be made locally. “Prohibit the federal government from ordering the shut-down of water facilities. “Apply only to drinking water systems if they have chemicals of concern above certain threshold quantities.”

HR 5577 and Water Facilities Last year’s attempt at reauthorizing CFATS, HR 5577, clearly addressed the water treatment facility exemption. It would have removed that exemption and brought water treatment and waste water treatment facilities under the CFATS regulations. They could only have become ‘covered facilities’ if they had one or more DHS chemicals of interest (COI; typically chlorine, anhydrous ammonia and/or sulfur dioxide for this type facility) on-site above the screening threshold quantity (STQ) listed in Appendix A to 6 CFR part 27. Chairman Thompson’s proposal specifically addressed water facilities when it authorized the Secretary to shut down non-complying facilities. In §2105(b)(4) the legislation established a higher standard for that sanction for water facilities:

“Notwithstanding the preceding sentence, the Secretary may not issue an order to cease operations under this paragraph to the owner or operator of a drinking water or wastewater facility unless the Secretary determines that continued operation of the facility represents a clear and present danger to homeland security.”

There were no provisions in HR 5577 that would have exempted water facilities from the IST provisions of the legislation. In fact, there are many that feel that a large number of the facilities that the IST provisions were designed to affect would have been water treatment facilities using chlorine gas to disinfect the water. This is reflected in the large number of water treatment facilities that are listed in the Center for American Progress publication, Chemical Security 101. AWWA and CFATS Reauthorization It is almost a certainty that the legislation being developed by the House Homeland Security Committee will remove the water facility exemption from the §550 authorization. Where the AWWA will fall on the issue of CFATS authorization legislation then will be determined by the wording of the IST provisions. The AWWA had promised to come up with a model method for analyzing for technical and financial feasibility of replacing chlorine gas at water and waste water treatment facilities. If they can convince the Committee that the method provides a ‘legitimate analysis’ (technically and politically) they have a chance of getting IST wording that they can live with. Having the analysis peer reviewed would go a long way to convincing the Committee of the legitimacy of the analytical method.