There was an interesting
blog post by Eric Byres over at TofinoSecurity.com describing a new way to
use their Tofino Industrial Security Solution to address the problem of
updating multiple control system devices as new security vulnerabilities are
discovered and corrected. Now I am not qualified to evaluate how well this
actually works, but the basic concept sounds so good that I just have to talk
about it. WARNING: Technical errors in this post are mine alone, don’t blame
Eric.
The Problem
In an earlier
blog post Eric wrote about a real scary problem with Microsoft
updates/patches that pose a particular problem for control systems. But that
isn’t the limit of the problems with patches. The most basic problem is that
most organizations don’t want to interrupt their process to take their control
system down to execute the appropriate patches, complete with the appropriate
testing and validation required to ensure that there are now unintended interferences
with their operations.
Next there is the problem with the apparently ever
increasing number of patches that would have to be applied to the system. The
increase in attention that is being directed at the vulnerabilities to be found
in control systems has resulted in an ever larger number vulnerabilities being
published. If you look at the large number and variety of smart devices
connected to modern control systems it is easy to see that there could be any
number of patches having to be made to the control system on a fairly routine
basis.
The more patches that are required to be made to keep the
system protected against attacks, the more likely it will be that the
owner/operator will make the very reasonable decision not to do any patches.
Rather than trying to guess which vulnerability might really pose a threat to
their system, the practical decision is that there is no reasonable method of
making that determination so why waste the time making the wrong patches.
The Solution
There is one very common thing associated with a large percentage
of the vulnerabilities reported by ICS-CERT; their remote exploit involves a
very specific type of communication being made to the vulnerable device. These
can be default passwords, specially crafted messages, or commands to specific
ports on the devices.
What Tofino has is the ability to program these vulnerability
signatures into their device which monitors control system communications. When
it detects one of the signature events it blocks that communications, logs the
event, and sends out an alarm so that humans can intercede. As new
vulnerabilities are identified the signature library can be upgraded without
any direct effect on the control system.
Okay, things are a bit more complicated than that, but this
isn’t a Tofino sales presentation, contact Eric or one of his sales people if
you want a more complete explanation of how their system works.
Shortcomings
Now Eric doesn’t claim that this is a one-hundred percent
answer to ICS security problems. Zero day vulnerabilities are obviously not
covered; by definition there is no signature available for a 0-day. Spear
phishing attacks that attempt to gain user credentials or insider attacks would
not be directly affected by this system.
Tofino Security develops their signature list based on
information provided by the vendors. Not all vendors are willing to work with
them so not every device is covered. If you have one of these Tofino systems
and one of your device vendors does not support it through Eric’s people you
might want to ask pointed questions. Eric has mentioned that they have the
capability to develop some signatures on their own based upon knowledge of
known vulnerabilities, but I suspect that sort of service starts to get pricey.
This system monitors communications to and from the control
system. If there is some sort of direct access to peripheral devices through
physical connections, wireless access or whatever, this system cannot intercept/block
that communications.
No one else has mentioned it, but one of my pet peeves
doesn’t seem to be addressed by this system. If there are embedded systems,
programs or subroutines from vendors that you are not aware of and those
systems have vulnerabilities, it seems unlikely that signatures for those will
be addressed by the system. You see, when this is installed you tell Tofino
Security what systems you have and they program your device with the signatures
for those systems. If you don’t tell them because you don’t know about the
device, program, or subroutine, then they can’t help you.
Suggestions
The first suggestion is directed towards the vendors that
aren’t currently working with Tofino; start. As this system is adopted by more
owners the failure to be a supporting vendor is going to start to impact sales.
Second, it seems to me that for vendors that have been made
aware of vulnerabilities, but have not yet had the chance to complete work on a
distributable patch, this is a good way to provide some level of protection to
your customers while you’re hard at work on the real fix. This would be
especially true for those uncoordinated disclosures that put your customers at
the most risk. Providing Eric’s team with vulnerability signatures as early in
the process as possible would be an excellent selling point.
Finally, it seems to me that Tofino Security would benefit
from having a team actively watching the uncoordinated disclosures being made
and working with those researchers to develop signatures for those
vulnerabilities. For vendors that they are working with, this would provide a
valuable service. For their customers that have equipment from non-cooperative
vendors it would provide them some level of protection while they wait out the
replacement life-cycle of the equipment.
Posts
5 comments:
The bottleneck is network latency. Average DCS / SCADA has issues when latency in the realtime segments gets above 80 ms. Tofino firewalls cause a high latency, which will have an impact if all data acquisition is slowed down.
HP TippingPoint IPS has been used for this function for some time, it is all ASIC and has a latency of 80 micro second. Combined with Zero Day Initiative this offers also protection prior to patching and protects legacy systems / unpatched systems. For expample used in combination with Honeywell EPKS for several years between level 3 and level 2.
When reading the blog and considering the response made by "Anonymous" above, I would like to add a couple observations that may help clarify the interpretation of what is trying to be done.
It would appear that the reference architecture used by "Anonymous" has the HP appliance sitting on the conduit between levels 2 and 3 (not sure if "Anonymous" is referring to OSI "layers" or ISA95 "levels"), but this is not typically the place that latency is critical. I stress that I say "typically", as not all ICS architectures and systems are the same, and some may have different requirements than others. Latency is commonly the most critical at ISA95 Levels 0-2 which is typically managed by OSI Layer 2 equipment. I have implemented a variety of packet-level devices including TSA and vendor specific (Honeywell CF9, Siemens Scalance 61x), and these have not netted any negative latency issues with the devices they are supervising.
The response provided can in fact be very industry specific. If "Anonymous" is speaking about the issues around communication latency and jitter necessary to maintain reliable control, then I would typically recommended a different type of product that is more appropriate for securing packets at OSI L2 rather than L4-7 (which is where firewall or similar appliances function).
I believe that we need more visibility WITHIN the ICS networks; not just the conduits between the ICS nets and the other upper-level "less trusted" networks like a Control DMZ and Office/Enterprise nets. The Tofino solution is designed for filling in this particular gap, and my understanding is that it was not intended to replace, but rather augment, security controls utilized elsewhere in the architecture. The HP solution mentioned would do nothing if the vector originated "below" the conduit unless it made external calls. For this reason, the Security Policy approach is really what we are looking for.
As always ... stay secure!
I realize that latency is less critical between L3 and L2 (ISA levels), but you see today systems with central control rooms and multiple L2 clusters that require routing between two or more L2 clusters. Which means the IPS is actually filtering control traffic. So in those cases the IPS is active between two or more L2 zones / segments (including filtering peer to peer control schemes). Latency is very important here. I responded to the blog because there seemed to be a suggestion to place Tofino firewalls for filtering purposes.
I think Tofino firewalls are excellent devices but because they are in essence multi-purpose filters they also cause significant additional latency, much more than the dedicated vendor filters mentioned by you. This latency (when too high) causes issues for control cascades passing these filters (there are cases when mode shedding occurred) and when the network latency would be structurally raised process schematics with a high number of update values would get update issues. Particularly DCS is more sensitive to this, where SCADA allows more freedom. Similar issues occur for some systems that support a kind of tuning mode, that raises the scan frequency temporarily to support the tuning of a control loop.
Therefore you need to be careful with adding latency at L2 and L1. Protection where needed is fine ( for example Modbus protection) but we must be careful adding this communication delay in everywhere. The Tofino filter is adding significantly more latency than an IPS like the HP Tippingpoint build completely from ASICs (and of course is a totally different price) that also offers more protection than the typical policy filters found in firewalls.
Every security zone needs its access control but this is not necessarily a stateful filter. For several protocols (like Modbus) the stateful filter is not adding anything, so faster security filters should be applied. And sometimes we need more advanced filters when we want to protect against network worms that exploit vulnerabilities in the service itself.
We may not be comparing “apples to apples” here. The Tofino Security Appliances are not trying to be a general purpose IDS/IPS sitting on a backbone or major L2/L3 connector. They are designed as Deep Packet Inspection (DPI) firewalls, with the ability to have very detailed and focused rule sets for specific SCADA and ICS products.
As a result of this focused design, we are not aware of a single case where Tofino firewalls have caused latency that is significant to the process. Certainly the Tofino would add more latency than the HP TippingPoint platform, IF it was used like a TippingPoint (i.e. as a general purpose IDS), but that isn’t how we suggest customers use them. Instead, the rules are applied based on the devices you have behind the Tofino. A short example might help clarify this (you can also see this in video format at http://www.tofinosecurity.com/sites/default/files/flash/Tofino_Predeployment.swf).
Imagine you have a control system that is a mix of Schneider PLCs, Rockwell PLCs and Iconics HMIs. Also imagine that they all have vulnerabilities. To address these vulnerabilities, you would load three Security Profiles into your Tofino Central Management Platform (CMP- the configuration tool) and draw a schematic in the CMP of your network using the device templates included in the profile. The Tofino CMP would then propose the special rules needed for each product on your network and assign them to specific Tofino firewalls as needed. For example, if Tofino A is protecting the PLC network that contains the Schneider and Rockwell PLCs, then only the appropriate Schneider and Rockwell special rules are loaded in Tofino A. They are also directed to only be used against the traffic going to the appropriate PLC. Specifically, Schneider rules should not be filtering traffic directed to the Rockwell PLC.
This is the advantage of the Tofino solution that would be difficult to implement with a general purpose solution like TippingPoint, where you typically only have a single appliance responsible for analyzing all network traffic passing through it. The Tofino solution targets specific platforms, and therefore, and processes this information very fast.
In other words, the idea behind the Security Profile is a combination of distributed appliances, custom rules and product templates, so that the right rules can be assigned to the right traffic going to the right device.
Finally, I disagree that stateful filtering does not add additional security for Modbus. A simple example is a Modbus reply flood, where an attacker takes advantage of an open 502 port to send a million spoofed replies in response to a single Modbus request. I won’t mention any brands, but we have seen Modbus masters fail badly when subjected to this sort of attack. Stateful inspection at both L3/L4 and L7 is an ideal way to prevent this. There are several other attacks our team discovered against Modbus that are even nastier and definitely need any Firewall or IDS to track state if they are to be blocked. But I don’t publicly release exploits – that’s Dale’s job (grin).
One item that Anonymous mentions in his October 21 reply to my original comment is in the first paragraph. I have worked with many systems, and am very familiar with the CCR concept of having multiple units consolidate into a single "supervisory" zone. What stands out here is the comment that "IPS is actually filtering control traffic".
I will admit right off that I am primary a DCS type of ICS guy, and I am very interested in understand the TippingPoint solution that is able to perform content inspection on control-level traffic that is based on typically vendor-specific protocols. I want to be clear that I am not talking about Modbus traffic, because let's be honest, Modbus is not a protocol that cares much about latency. I am talking about the control protocols that are typically used today with modern BPCS controllers - for example, Honeywell's Control Data Access (CDA) protocol that is used between C300's, FIM's, etc.
Am I to understand that Anonymous has implemented a TippingPoint Platform with these type of protocols? If so, please share details, as this would be very exciting details for the community to understand!
I appreciate this constructive dialogue and look forward to some additional input.
Stay secure ...
Post a Comment