Florida Water System Hack - 2-8-21

I have been watching the discussion on TWITTER® about yesterday’s ‘hack’ of a community water treatment facility in the City of Oldsmar, Florida, a relatively small town near Tampa. There are not a lot of details yet about the attack and we certainly are not at the point in the story yet where anyone is able to point a finger at the ‘attacker’. But I did get an interesting email from a reader that made the point:

“In case you haven’t seen it, there was a hack involving the water treatment facility for the City of Oldsmar, Florida.  Since you seem to be able to dig in and get to the bottom of things, I was hoping that you would report on it in your blog.”

No blogger could resist a request like that, so here goes.

The Hack

There is a nice summary (with links) about what is known and what is being discussed about the incident. Many thanks to Cynthia Brumfield for compiling this information.

The short story is that an operator at the plant was watching the display screen for the computer system that controls the operations of the drinking water treatment facility. He was surprised to see the mouse cursor move across the screen when he was not moving the mouse. The cursor showed someone increase the set point for the control that added sodium hydroxide (NaOH) to the water (used to adjust pH, a critical parameter of drinking water) from 100 ppm to 1,100 ppm. The operator took immediate action to correct the problematic NaOH controls.

Anyone that has had a corporate IT department fix something on their computer remotely, has seen the cursor move on their screen due to remote control. Apparently, this water system was set up to allow at least the facility supervisor remote access to the system to view and make changes to the treatment process. According to the news reports, the facility uses Team Viewer software to accomplish this remote access and presumably the attacker used the same software for their attack. There is no clear information publicly available about the security controls that the utility had in place for the use of Team Viewer.

Security Regulations

Security (physical security and cybersecurity) for drinking water systems in the United States is loosely regulated by the EPA. Depending on the size of the facility (determined by the number of people served) the facility is required to conduct a security vulnerability assessment and certify to the EPA that the assessment has been conducted. A facility the size of the Oldsmar treatment works is required to provide the certification that this assessment has been done to the EPA by June 31^st of this year.

The EPA has provided small treatment works like Oldsmar with a fill in the blank form for conducting their risk assessment. It asks facility management to identify which of ten listed malevolent acts pose ‘a significant risk’ to ten asset categories found at the water treatment facility. One of the listed potential malevolent acts is ‘cyberattack on process control system’ and applicable asset in this particular case would be ‘Pretreatment and Treatment’. If the ‘cyberattack’ act is selected, the utility is asked to briefly explain “how the malevolent act could impact this asset category at the CWS [community water system]”.

Finally, the form allows the facility the option to list countermeasures they could employ to reduce the risk to, or increase the resilience of, the CWS for a particular malevolent act.

Inadequate Cybersecurity

Control system security experts have long maintained that control systems for critical processes should not be exposed to remote access as it significantly increases the risk of system compromise. That did not stop Stuxnet from infecting Iranian centrifuges, but it is certainly the gold standard for industrial control system security. Unfortunately, the ‘gold standard’ seldom makes it into the marketplace.

Few would argue that drinking water systems are not ‘critical systems’. The health and prosperity of the community are directly impacted by safe and efficient drinking water systems. The reality, however, is that there is a relatively small amount of money available in small communities to create, maintain, and operate drinking water systems. There is not a large staff involved, seldom a control system engineer or technician available and certainly no cyber security experts in-house. Outside vendors design, install and periodically maintain these control systems. With no legal requirement to meet any cybersecurity standards, there is little incentive to put in the extra money that gold standard security would require. And even lesser security standards cost more than many communities think they can afford for these basic services.

Mitigation Measures

In reality, these facilities do have mitigation measures in place to protect against consequences of this type of attack. The increase in NaOH in this attack (if undetected by the operator) would have caused a significant increase of the pH of the water (possibly to as much as 12.3 by my calculations). There are, however, pH measuring devices through out the water treatment process that would have detected that increase and sounded alarms. Operators would have isolated the affected water, taken corrective action to correct the addition problem, and adjusted the pH of the water so that it could be released into the drinking water system. This is what operators get paid to do on a daily basis.

Fixing the Problem

The EPA does have cybersecurity assessment tools available for local treatment works. There should be a requirement for water treatment facilities to use these tools in their vulnerability assessment and to submit a plan to EPA about how they plan to correct any deficiencies noted. The EPA was prevented from requiring these types of assessments and mitigation plans by Congress. Our elected representatives decided that such mandates would cost too much money and municipal drinking water systems would not be able to afford them. In many cases, especially for smaller utilities, they were almost certainly correct. It is time for there to be a national conversation again on this topic, and perhaps this incident will provide the impetus to begin that conversation.