Thursday, October 18, 2012

A New Cybersecurity Technique

There was an interesting blog post by Eric Byres over at describing a new way to use their Tofino Industrial Security Solution to address the problem of updating multiple control system devices as new security vulnerabilities are discovered and corrected. Now I am not qualified to evaluate how well this actually works, but the basic concept sounds so good that I just have to talk about it. WARNING: Technical errors in this post are mine alone, don’t blame Eric.

The Problem

In an earlier blog post Eric wrote about a real scary problem with Microsoft updates/patches that pose a particular problem for control systems. But that isn’t the limit of the problems with patches. The most basic problem is that most organizations don’t want to interrupt their process to take their control system down to execute the appropriate patches, complete with the appropriate testing and validation required to ensure that there are now unintended interferences with their operations.

Next there is the problem with the apparently ever increasing number of patches that would have to be applied to the system. The increase in attention that is being directed at the vulnerabilities to be found in control systems has resulted in an ever larger number vulnerabilities being published. If you look at the large number and variety of smart devices connected to modern control systems it is easy to see that there could be any number of patches having to be made to the control system on a fairly routine basis.

The more patches that are required to be made to keep the system protected against attacks, the more likely it will be that the owner/operator will make the very reasonable decision not to do any patches. Rather than trying to guess which vulnerability might really pose a threat to their system, the practical decision is that there is no reasonable method of making that determination so why waste the time making the wrong patches.

The Solution

There is one very common thing associated with a large percentage of the vulnerabilities reported by ICS-CERT; their remote exploit involves a very specific type of communication being made to the vulnerable device. These can be default passwords, specially crafted messages, or commands to specific ports on the devices.

What Tofino has is the ability to program these vulnerability signatures into their device which monitors control system communications. When it detects one of the signature events it blocks that communications, logs the event, and sends out an alarm so that humans can intercede. As new vulnerabilities are identified the signature library can be upgraded without any direct effect on the control system.

Okay, things are a bit more complicated than that, but this isn’t a Tofino sales presentation, contact Eric or one of his sales people if you want a more complete explanation of how their system works.


Now Eric doesn’t claim that this is a one-hundred percent answer to ICS security problems. Zero day vulnerabilities are obviously not covered; by definition there is no signature available for a 0-day. Spear phishing attacks that attempt to gain user credentials or insider attacks would not be directly affected by this system.

Tofino Security develops their signature list based on information provided by the vendors. Not all vendors are willing to work with them so not every device is covered. If you have one of these Tofino systems and one of your device vendors does not support it through Eric’s people you might want to ask pointed questions. Eric has mentioned that they have the capability to develop some signatures on their own based upon knowledge of known vulnerabilities, but I suspect that sort of service starts to get pricey.

This system monitors communications to and from the control system. If there is some sort of direct access to peripheral devices through physical connections, wireless access or whatever, this system cannot intercept/block that communications.

No one else has mentioned it, but one of my pet peeves doesn’t seem to be addressed by this system. If there are embedded systems, programs or subroutines from vendors that you are not aware of and those systems have vulnerabilities, it seems unlikely that signatures for those will be addressed by the system. You see, when this is installed you tell Tofino Security what systems you have and they program your device with the signatures for those systems. If you don’t tell them because you don’t know about the device, program, or subroutine, then they can’t help you.


The first suggestion is directed towards the vendors that aren’t currently working with Tofino; start. As this system is adopted by more owners the failure to be a supporting vendor is going to start to impact sales.

Second, it seems to me that for vendors that have been made aware of vulnerabilities, but have not yet had the chance to complete work on a distributable patch, this is a good way to provide some level of protection to your customers while you’re hard at work on the real fix. This would be especially true for those uncoordinated disclosures that put your customers at the most risk. Providing Eric’s team with vulnerability signatures as early in the process as possible would be an excellent selling point.

Finally, it seems to me that Tofino Security would benefit from having a team actively watching the uncoordinated disclosures being made and working with those researchers to develop signatures for those vulnerabilities. For vendors that they are working with, this would provide a valuable service. For their customers that have equipment from non-cooperative vendors it would provide them some level of protection while they wait out the replacement life-cycle of the equipment.


Anonymous said...

The bottleneck is network latency. Average DCS / SCADA has issues when latency in the realtime segments gets above 80 ms. Tofino firewalls cause a high latency, which will have an impact if all data acquisition is slowed down.

HP TippingPoint IPS has been used for this function for some time, it is all ASIC and has a latency of 80 micro second. Combined with Zero Day Initiative this offers also protection prior to patching and protects legacy systems / unpatched systems. For expample used in combination with Honeywell EPKS for several years between level 3 and level 2.

Joel "the SCADAhacker" said...

When reading the blog and considering the response made by "Anonymous" above, I would like to add a couple observations that may help clarify the interpretation of what is trying to be done.

It would appear that the reference architecture used by "Anonymous" has the HP appliance sitting on the conduit between levels 2 and 3 (not sure if "Anonymous" is referring to OSI "layers" or ISA95 "levels"), but this is not typically the place that latency is critical. I stress that I say "typically", as not all ICS architectures and systems are the same, and some may have different requirements than others. Latency is commonly the most critical at ISA95 Levels 0-2 which is typically managed by OSI Layer 2 equipment. I have implemented a variety of packet-level devices including TSA and vendor specific (Honeywell CF9, Siemens Scalance 61x), and these have not netted any negative latency issues with the devices they are supervising.

The response provided can in fact be very industry specific. If "Anonymous" is speaking about the issues around communication latency and jitter necessary to maintain reliable control, then I would typically recommended a different type of product that is more appropriate for securing packets at OSI L2 rather than L4-7 (which is where firewall or similar appliances function).

I believe that we need more visibility WITHIN the ICS networks; not just the conduits between the ICS nets and the other upper-level "less trusted" networks like a Control DMZ and Office/Enterprise nets. The Tofino solution is designed for filling in this particular gap, and my understanding is that it was not intended to replace, but rather augment, security controls utilized elsewhere in the architecture. The HP solution mentioned would do nothing if the vector originated "below" the conduit unless it made external calls. For this reason, the Security Policy approach is really what we are looking for.

As always ... stay secure!

Anonymous said...

I realize that latency is less critical between L3 and L2 (ISA levels), but you see today systems with central control rooms and multiple L2 clusters that require routing between two or more L2 clusters. Which means the IPS is actually filtering control traffic. So in those cases the IPS is active between two or more L2 zones / segments (including filtering peer to peer control schemes). Latency is very important here. I responded to the blog because there seemed to be a suggestion to place Tofino firewalls for filtering purposes.

I think Tofino firewalls are excellent devices but because they are in essence multi-purpose filters they also cause significant additional latency, much more than the dedicated vendor filters mentioned by you. This latency (when too high) causes issues for control cascades passing these filters (there are cases when mode shedding occurred) and when the network latency would be structurally raised process schematics with a high number of update values would get update issues. Particularly DCS is more sensitive to this, where SCADA allows more freedom. Similar issues occur for some systems that support a kind of tuning mode, that raises the scan frequency temporarily to support the tuning of a control loop.

Therefore you need to be careful with adding latency at L2 and L1. Protection where needed is fine ( for example Modbus protection) but we must be careful adding this communication delay in everywhere. The Tofino filter is adding significantly more latency than an IPS like the HP Tippingpoint build completely from ASICs (and of course is a totally different price) that also offers more protection than the typical policy filters found in firewalls.

Every security zone needs its access control but this is not necessarily a stateful filter. For several protocols (like Modbus) the stateful filter is not adding anything, so faster security filters should be applied. And sometimes we need more advanced filters when we want to protect against network worms that exploit vulnerabilities in the service itself.

Eric Byres said...

We may not be comparing “apples to apples” here. The Tofino Security Appliances are not trying to be a general purpose IDS/IPS sitting on a backbone or major L2/L3 connector. They are designed as Deep Packet Inspection (DPI) firewalls, with the ability to have very detailed and focused rule sets for specific SCADA and ICS products.

As a result of this focused design, we are not aware of a single case where Tofino firewalls have caused latency that is significant to the process. Certainly the Tofino would add more latency than the HP TippingPoint platform, IF it was used like a TippingPoint (i.e. as a general purpose IDS), but that isn’t how we suggest customers use them. Instead, the rules are applied based on the devices you have behind the Tofino. A short example might help clarify this (you can also see this in video format at

Imagine you have a control system that is a mix of Schneider PLCs, Rockwell PLCs and Iconics HMIs. Also imagine that they all have vulnerabilities. To address these vulnerabilities, you would load three Security Profiles into your Tofino Central Management Platform (CMP- the configuration tool) and draw a schematic in the CMP of your network using the device templates included in the profile. The Tofino CMP would then propose the special rules needed for each product on your network and assign them to specific Tofino firewalls as needed. For example, if Tofino A is protecting the PLC network that contains the Schneider and Rockwell PLCs, then only the appropriate Schneider and Rockwell special rules are loaded in Tofino A. They are also directed to only be used against the traffic going to the appropriate PLC. Specifically, Schneider rules should not be filtering traffic directed to the Rockwell PLC.

This is the advantage of the Tofino solution that would be difficult to implement with a general purpose solution like TippingPoint, where you typically only have a single appliance responsible for analyzing all network traffic passing through it. The Tofino solution targets specific platforms, and therefore, and processes this information very fast.

In other words, the idea behind the Security Profile is a combination of distributed appliances, custom rules and product templates, so that the right rules can be assigned to the right traffic going to the right device.

Finally, I disagree that stateful filtering does not add additional security for Modbus. A simple example is a Modbus reply flood, where an attacker takes advantage of an open 502 port to send a million spoofed replies in response to a single Modbus request. I won’t mention any brands, but we have seen Modbus masters fail badly when subjected to this sort of attack. Stateful inspection at both L3/L4 and L7 is an ideal way to prevent this. There are several other attacks our team discovered against Modbus that are even nastier and definitely need any Firewall or IDS to track state if they are to be blocked. But I don’t publicly release exploits – that’s Dale’s job (grin).

Joel "the SCADAhacker" Langill said...

One item that Anonymous mentions in his October 21 reply to my original comment is in the first paragraph. I have worked with many systems, and am very familiar with the CCR concept of having multiple units consolidate into a single "supervisory" zone. What stands out here is the comment that "IPS is actually filtering control traffic".

I will admit right off that I am primary a DCS type of ICS guy, and I am very interested in understand the TippingPoint solution that is able to perform content inspection on control-level traffic that is based on typically vendor-specific protocols. I want to be clear that I am not talking about Modbus traffic, because let's be honest, Modbus is not a protocol that cares much about latency. I am talking about the control protocols that are typically used today with modern BPCS controllers - for example, Honeywell's Control Data Access (CDA) protocol that is used between C300's, FIM's, etc.

Am I to understand that Anonymous has implemented a TippingPoint Platform with these type of protocols? If so, please share details, as this would be very exciting details for the community to understand!

I appreciate this constructive dialogue and look forward to some additional input.

Stay secure ...

/* Use this with templates/template-twocol.html */