Industrial Chemical Incompatibility – Bleach

Hardly a week goes by when we don’t see a news story about a hazmat incident caused by the mixing of incompatible chemicals. Yesterday there was a story from a water treatment plant in Weslaco, TX. In this case there was no one hurt nor any long term damage to the facility. I haven’t written about one of these incidents lately, so this will be a good chance to review the problems associated with bulk chemical storage.

The Incident

As is usual the news reports (here, here and here) provide incomplete and conflicting data about what happened. Here is what I think we know.

Apparently just before 2 pm yesterday a tank truck showed up at the Weslaco water treatment facility for a routine chemical delivery. It is not clear (from news reports) what chemical was in that truck; one report claims is was ‘sodium chloride’. A sodium chloride solution is not a chemical normally used by water treatment facilities. Maybe they meant ferric chloride.

The truck contents unloaded into a sodium hypochlorite (industrial strength bleach) storage tank. It was never mentioned who unloaded the truck, but this is very often the truck driver. In this case there is at least one report that the truck driver did not know what he was carrying (I hope this was not true).

It is also not clear if a chemical reaction actually took place; one news report claims that one did occur, producing ‘hydrogen chloride’. Again, this is not typically a reaction product of sodium hypochlorite, we generally see chlorine gas as a reaction product.

Local residents (within 820 or 1,000 feet depending on the news story) downwind of the plant were told to shelter in place. Apparently a nearby (within 250 ft according to one report) residence and daycare center were evacuated. At about 5:00 pm (CDT) the all clear was apparently given.

Public Communications

It is easy to blame the news media for the conflicting information be provided to the public about this incident. It is apparent, however, that the local emergency response community was the source of most of the information provided and that community did not speak with one voice. If this had been a more serious incident this could have caused significant problems as the community reacted to the conflicting messages.

Sodium Hypochlorite (NaClO)

I have talked about this chemical on a number of different occasions. In water treatment and waste water treatment it is used as a source of chlorine for water disinfection. It is not quite as effective as chlorine gas, but it is not nearly as dangerous if released into the environment.

It is however a very reactive chemical. It reacts explosively with ammonia and it reacts violently with acids and reducers. It also reacts with heavy metals. In every case chlorine gas is given off as a byproduct of these reactions and the reactions are generally exothermic; frequently producing enough heat to produce steam. This will increase internal tank pressures significantly.

A number of chemicals routinely used in water and waste water treating facilities react violently with hypochlorite, even in very dilute solutions. They include: ammonia, ferric chloride, muriatic acid and sulfuric acid.

Sodium hypochlorite degrades over time and it degrades faster at higher temperatures. The byproducts produced during that degradation process are chlorine gas, water and sodium chloride (salt). For this reason there are typically frequent deliveries of sodium hypochlorite, typically by tank wagon or rail car for most treatment facilities.

Storage Tank Management

When dangerous chemicals (and due to reactivity at least sodium hypochlorite is a dangerous chemical) are stored in bulk storage tanks special care must be taken to ensure that the design and management of those tanks take into account both the stand alone hazards of the chemicals, and the reactions of those chemicals with those that could ‘reasonably’ be accidentally introduced into the storage tank. The EPA has a simple chemical compatibility chart for water treatment plants; it is, however, short on information about sodium hypochlorite.

Sodium hypochlorite it typically stored in a vented storage tank that allows vapors to escape the tank during filling operations and when decomposition vapors are produced. A common industry handbook for bleach recommends sizing the venting device to take into account the decomposition outgassing rather than the fill rate outgassing, meaning that the outgassing rate for decomposition may be significantly higher than for filling operations. Typically, decomposition rates are determined by maximum ambient temperature, not the much higher rates of decomposition associated with filling the tank with incompatible materials. That means that there may be a significant hazard of catastrophic failure of the storage tank in an incompatible filling situation.

Proactive strategies to prevent misfiling the tank are thus very important. The first step is to identify all of the undesirable chemical reactions that can take place at a facility based on the bulk chemicals that are received. The facility management and employees must be fully aware of the potential hazards associated with the improper mixing of chemicals in storage tanks.

While most medium to large chemical manufacturing facilities have a dedicated bulk unloader to handle transferring liquids from tank trucks and rail cars to storage tanks, most water treatment facilities and smaller chemical companies will not have that luxury. This means that it is very common for truck drivers to unload their own trucks. Unloading procedures must take that into account.

The facility needs to make it as difficult as possible to unload the wrong chemical into the wrong tank. The easiest technique to use is to have all storage tanks in widely different areas. This is frequently not possible due to physical layout constraints. Additionally, the requirement to have storage tanks in diked areas makes it impractical to have completely separate storage tank areas.

Another technique is the use of access control measures. The most effective that I have seen in use is a double lock system on each unloading line. There are two locks on each line and every lock is uniquely keyed. Two unload a tank wagon or rail car the unloader first gets a key from the operations supervisor; this is done to confirm that a load is expected and to stage the delivery truck at the correct location. The second key is obtained from the quality control lab; there the quality of the material being delivered is confirmed (either by testing or COA checking) before the second key is provided. In 20 years at one medium sized chemical manufacturing facility where I know this was employed there was never a successful unload of material into the wrong tank.

For very dangerous or highly reactive chemicals additional measures should be taken. There should be signs in the unloading area clearly stating what chemicals should never be unloaded. The sign should also provide immediate emergency response information for actions to be taken if an unloading accident should occur. Finally, and most importantly, specific advance coordination needs to be made with emergency response personnel, on and off site, so that they are well familiar with the actions they need to take if an unloading accident of this type occurs.

Emergency Response

Once an unloading accident occurs there is not much remedial action that can be taken. Particularly with sodium hypochlorite reactions, the reactions are practically instantaneous. By the time emergency response personnel arrive on scene there will be a chlorine gas cloud present if one is going to be produced so emergency responders need to approach the site from upwind.

For most metal tanks the danger for catastrophic failure of the tank will have passed by the time emergency responders arrive on scene. For plastic and fiberglass tanks the hot fluids in the tank may continue to weaken the tank shell over time so those tanks should be cooled if they are not already compromised.

Chlorine gas procedures should be implemented immediately and downwind testing for chlorine gas should be started. Particular attention should be paid to low lying areas where chlorine tends to collect since it is heavier than air. Evacuation and shelter in place criteria should be set in place prior to the incident based upon the worst case chlorine release from the tank. Local hospitals should be notified to prepare for chlorine gas casualties. All personnel with even the slightest exposure should be identified for follow-up medical checks due to the long term hazards of chlorine exposure.

As always in incidents involving industrial chemicals, advance planning is the key to an effective emergency response.

Cyber Security and Storage Tanks

I know that for the last week or so it seems that this blog has become the Cyber Security News, but that’s because of a slew of news reports that address vulnerabilities in control systems that might be used at high-risk chemical facilities. So I thought that this might be a good time to look at how the ability to remotely execute commands on a control system might be used for a physical attack on a chemical facility. The specific target was suggested by the key words used by an unidentified reader yesterday in a search that brought him/her to the site; ‘cyber security storage tanks’.

Storage Tanks

Most storage tanks are really nothing more than large metal drums holding some sort of chemical. Since everything is chemical, the only sure thing that we can say about a storage tank is that it is used to store chemicals. Those chemicals can be something as innocuous as water or air or as dangerous as chlorine gas or methyl isocyanate.

Probably the majority of storage tanks in existence in the United States have no connection to industrial control systems (ICS), but some large number of tanks are connected in one way or another to some sort of control system. An ICS can be used to monitor conditions (level, temperature, pressure, etc) within the tank. It can also be used to manipulate the contents of the tank through controls of mixing devices, loading or unloading valves, heating and cooling, or pressure manipulation.

Cyber Attack Vectors – Measurement Devices

The ability to manipulate the input or output signal of just about any monitoring or actuation device on a storage tank can be utilized to execute an attack on a high-risk chemical facility where a vulnerable control system is connected to storage tanks. Let’s start with the manipulation of signals from measurement devices:

Level Measurement: There are a wide variety of level measurement devices, but one of the most common is the delta-pressure (DP) device. Pressure measurement sensors are placed at the bottom of the tank and the top of the tank. Using the difference (delta) between those measurements and the programmed density of the material, the DP device calculates the height of the liquid column. Reducing the programmed density would lead to an under-reporting of the liquid level, allowing the tank to be overfilled.

Temperature Measurement: Most temperature measurement devices in storage tanks are used to make sure that the material is in the proper temperature range for the material to be moved from the tank. In some chemicals, however, the temperature is monitored for safety reasons; too high a temperature and chemical reactions start that can lead to uncontrollable chemical reactions that could produce toxic gasses or overpressure situations that could explosively destroy the storage tank. Manipulation of the output of temperature measurement devices could allow the temperature to rise above the critical value for that material while it looks like the temperature remains in safe ranges.

Just about any measurement device could be used as a means of physical attack on the facility. If the device monitors a physical parameter that could lead to an unsafe condition, the manipulation of that output of that device could lead to that condition.

Cyber Attack Vectors – Physical Control Devices

Almost certainly the most common control device at a chemical facility is the valve. Opening and closing valves can be used to physically move material into or out of storage tanks, manipulate the temperature by allowing for the flow of heating or cooling fluid through heat transfer devices, or manipulate storage tank pressures by opening vents or adding gasses to the ‘empty’ headspace in the tank.

The simplest sort of attack is to open the bottom valve on a storage tank when there is no hose or other transfer device attached to that valve. If the level reporting devices are manipulated at the same time, many storage tanks could be completely emptied before anyone becomes aware of the problem.

Catastrophic results could be obtained even if there are transfer lines in place to move the material to locations other than to the local environment. Moving material at the wrong time or in the wrong amount could lead to dangerous chemical reactions that could result in toxic gas releases, fires or explosions; all undesirable results from all but the terrorist’s or criminal’s perspective.

Again, just about any type of physical control can, under the proper circumstances, be used to affect a catastrophic result in a high-risk chemical facility. While manipulation of devices in the production process would require a detailed understanding of that process, much less sophisticated knowledge of storage tanks could allow for a catastrophic attack.

Storage Tank Cyber Vulnerabilities

These are just a few examples of the avenues that the manipulation of industrial control systems could be used to effect an attack on high-risk chemical facilities. Storage tanks may actually be the simplest targets for cyber attack. Since much less process knowledge, and thus intelligence collection or insider knowledge is needed to affect an attack on storage tanks, they are easier to attack. Furthermore, operators spend less time looking at storage tanks than they do process vessels so they would be less likely to detect an attack in progress.