Reader Question – Which advisory?

 Yesterday, I had an interesting question asked me over on LinkedIn about my post last Tuesday on CISA’s control system security advisories. Taisia Berg asked: “Which of these advisories do you think will have the biggest impact on operators this week?” Quite aside from that fact that each facility is going to have its own unique mix of operational technology (and thus there will certainly be many facilities that are not affected by any of these eight advisories), manufacturing facilities generally are not able to respond to vulnerabilities on a daily or even weekly basis. Unless they can patch while operating (really not a good idea), they typically must wait for the next scheduled shutdown to patch any vulnerable equipment, and that could be months away. 

Having said all of that, this particular set of advisories presents a unique set of circumstances. The three ABB advisories relate to vulnerabilities that were all disclosed by the vendor back in January. The Hitachi Energy vulnerabilities were disclosed last week. One would like to think that owners of the affected devices/software would already have started their internal risk assessment and mitigation process for those vulnerabilities. 

So, what is the whole point about the CISA advisories, or gadflies like me writing about them? It is all about expanding the communications network that allows facilities to become aware of vulnerabilities in their equipment. In a perfect world (well almost perfect, vulnerabilities still exist) vendors would directly notify owners of their devices/software of each vulnerability as it was identified. But that is not practical because vendors frequently (usually?) sell through intermediaries and equipment frequently changes hands on resale markets. So, push notifications are not a total solution (probably not even a reasonably useful solution). 

Since many (certainly not all) vendors publish advisories for vulnerabilities in their products, one would expect owner/operators to watch the vendor's web sites for new advisories and updates. Since facilities may often have dozens (maybe hundreds) of different OT vendors to deal with, this could be a very time-consuming process (as I am acutely aware). So, CISA Advisories and blog posts like this are a shortcut to identifying new vulnerabilities (and updated information). 

Of course, CISA advisories also provide another important function, Security researchers who have not been able to successfully contact a vendor with a vulnerability notification can contact CISA to act as an intermediary. Even if CISA is similarly unable to coordinate with the vendor, they can issue an advisory based upon the researcher’s information.  

So, the unfortunate (and decidedly unhelpful) answer to the reader’s question is: “It depends.” 

Reader Comment – Affected Versions

Got some interesting feedback from an anonymous reader on Tuesday’s advisory post. Its about the conversion of affected version data from the Rockwell advisory into the CISA Advisory. Please read the comment. To better illustrate the reader’s comment, the table below is abstracted from the Rockwell advisory.

From a straightforward reading of the language used in the table the affected versions are ≥ v34.011 and <v34.014. And it looks like that is the way that CISA translated their data: “Versions v34.011 and later” and “Update to v34.014 and later”. The problem is that the more common way of defining affected versions is “≤ vXX.XXX”. This is because the assumption is made that when a vulnerability is found that it must have existed in versions preceding the one tested. It looks like Rockwell is taking a more forward looking attitude; ‘we know it exists at this point, and forward until we fixed it’. Owners can make the assumption about earlier versions, but since Rockwell (or the researcher) did not go back and test earlier versions, no claims about earlier versions are being offered in the Rockwell method of announcing affected versions.

Now, having said that, the Rockwell data is not always consistent with what I just said. I think this is an editorial consistency issue rather than a consistency of intent. The table below shows the last ten single CVE advisories, and the data provided in the affected version table in each advisory.

The data for CVE-2024-6078 reflects the more common method of listing vulnerability status. But that is the only data point that does not completely support my interpretation of the Rockwell data. Of course, the fact that we have having this discussion is a sure sign that there is not really a standardized reporting process for cybersecurity vulnerabilities.

Reader Comment – Continuing Broadcom Conversation

This morning an anonymous reader left a comment on my BIS Cyber-Enabled Activities post that was actually a reply to my last Reader Comment post about 3^rd party KEV problems. The commenter makes some good points about advisories and 3rd party vulnerabilities; well worth the effort to read.

Vendors need to realize that people using their software/devices look to their advisories for both information about the scope of the vulnerabilities to use in their risk management processes, as well as information about how to mitigate (short term risk minimization) or fix the vulnerability. In my opinion, Broadcom does not do a good job in either department, but they are hardly the worst of those that I look at in my reporting on ICS vulnerabilities.

Interestingly, some of the most informative ‘advisories’ I have seen are open source software discussions on github sites that frequently include discussions about how the vulnerabilities could be exploited. That may be a tad bit overbroad in the information sharing department, but it is probably helpful to developers using the software to mitigate the problem in their own usages.

Vulnerability Descriptions

An interesting article yesterday over at DarkReading.com that looked at default passwords was well worth reading for that reason alone, but an even more important aspect was buried towards the bottom of the article. Kelly Jackson Higgins looked at a recently released control system vulnerability and brought out more details from the researcher than were published in the related ICS-CERT advisory on the vulnerability.

The Vulnerability

Last month the DHS ICS-CERT published an advisory for a buffer overflow vulnerability in the Schneider Modicon M340 PLC. The Advisory described the vulnerability in this way:

“David Atch of CyberX has identified a buffer overflow vulnerability in Schneider Electric’s Modicon M340 PLC product line. Schneider Electric has produced a new firmware patch to mitigate this vulnerability.”

In yesterday’s article Higgin’s described the vulnerability in more detail”

“CyberX found a buffer overflow flaw in the products that can be exploited when a random password of between 90 and 100 characters is typed into the PLC's web interface. It basically crashes the device, and allows an attacker to execute code remotely.”

The publicly available Schneider Security Notice (.PDF Download) for the vulnerability provides even more details about the vulnerability which is actually in the Ethernet communication module for the PLCs:

• The attacker enters a password greater than 65 characters into the Password field (Generally between 90-100 characters is enough to cause the device to crash) and presses OK (no username is required).

• The web server handles the request and attempts to copy the oversized password into a 65-character buffer using strcpy(), which is void of buffer length protection.

Advisory Deficient?

Obviously, the Advisory from ICS-CERT does not provide the same level of detail as found in the Schneider Notice. The question is whether or not the Advisory is in some way deficient because of that lack of detail. To be sure, ICS-CERT does provide a link to the Schneider site where the Notice can be downloaded, so, in a round-about way the information is available to the reader. The question is, is that good enough?

ICS-CERT maintains that, as part of their disclosure policy that:

“After coordinating with vendors and gathering technical and threat information, ICS-CERT will take appropriate steps to notify end users about the vulnerability. ICS-CERT strives to disclose accurate, neutral, objective information focused on technical remediation and mitigation [emphasis added] for asset owners and operators.  ICS-CERT will reference other available information and correct misinformation when possible.”

If we look at the information solely for its focus on ‘technical remediation and mitigation’ ICS-CERT can be forgiven for not openly providing the same sort of detailed information on the mechanism of the vulnerability that is provided in the linked Schneider Notice. That information does not directly affect the ‘remediation and mitigation’ of the vulnerability.

On the other hand, only two major vendors routinely provide public access to vulnerability notices and a similarly limited number provide notices to registered owners. The bulk of the vendors associated with ICS-CERT advisories do not provide any information on vulnerabilities, even in the version notes when those are provided for updates. If this vulnerability were in a device from one of these other manufacturers, the owner would not have access to this information.

How are the Advisories Used?

There is little or no information available on how these ICS-CERT advisories are actually used. There is usually little written about the advisories and most of what is written (other than this blog) is a very brief; one-sentence description and a link to the advisory. The underlying vulnerabilities are sometime discussed in the literature, but one has to follow a number of different publications or writers to keep up with what vulnerabilities have been uncovered. Additionally, there is seldom information readily available about mitigation measures or links for patches and updates from any but a few of the major vendors.

For anyone that is interested in one-stop shopping for information about control system vulnerabilities and their remediation, ICS-CERT is just about the only source. ICS-CERT is to be commended for conducting this outreach effort to try to make this information readily available to the disparate control system community.

With that in mind, I would like to call attention to a couple of sentences that are found in the impact section of every advisory published by ICS-CERT:

“Impact to individual organizations depends on many factors that are unique to each organization. NCCIC/ICS-CERT recommends that organizations evaluate the impact of this vulnerability based on their operational environment, architecture, and product implementation.”

Why is this important? Every facility manager that becomes aware of a vulnerability in their automation control system has to make a very important risk-based analysis of what will be done about that vulnerability in their facility. The cost in time and effort (including acquiring the services of a control system engineer for many facilities) to test and verify that the mitigation measures suggested will work in the unique environment at that facility and then to shut down production to upgrade or update the affected devices can be quite high. Against that must be weighed the potential consequences of a successful attack (or even an accidental ‘attack’ in some cases).

Determining the cost side of the equation will be very simple for most managers. They will be painfully aware of the costs of shutting down operations, something that they are generally striving to avoid at nearly any cost. The consequences of a potential attack are much harder to calculate, particularly since figuring the likelihood of an attack is an integral part of that calculation.

If the manager is using the only one-stop source of control system vulnerability information as the source of information for determining if a vulnerability is of sufficiently high potential cost to justify the cost of shutting down production to fix the problem, then the information provided in this ICS-CERT Advisory probably falls short. A quick read of the Advisory identifies the vulnerability and potential consequence, but could easily leave the manager figuring that at least one mitigation measure is in place; password protected access to the device. That would probably be enough to postpone the fix until the next scheduled turnaround or the next time maintenance has to take one of the PLC’s out of service for another unrelated problem.

What Should We Expect from ICS-CERT?

Given all of this, what should we be expecting from ICS-CERT in the vulnerability advisories? First we have to acknowledge the legal disclaimer at the top of every alert and advisory:

“All information products included in http://ics-cert.us-cert.gov are provided "as is" for informational purposes only. The Department of Homeland Security (DHS) does not provide any warranties of any kind regarding any information contained within.”

Not only is this the type disclaimer that we have become accustomed to in the software world (and that is room for a whole other diatribe), but in a very real sense it specifically applies to ICS-CERT disclosures. ICS-CERT is not actually doing most (if any) of the research that is forming the basis for these information products. ICS-CERT is typically informed of the vulnerabilities by researchers and there are frequently disagreements about the extent (or sometimes even the existence) of the vulnerabilities between the researchers and the vendor. So there is no way that ICS-CERT can provide any guarantees about the information that it does share.

There is also another complicating factor; the whole public disclosure debate. The more information that is made publicly available about vulnerabilities, the argument goes, the easier it is for less skilled attackers to make use of that information to effect exploits against uncorrected systems. So, ICS-CERT has to weigh the amount of information it makes available to owners so that they can make a reasonable decision to fix their systems in a timely manner against providing early information to attackers. This can, for critical infrastructure however, be mitigated by early disclosure via the US-CERT Secure Portal.

The bottom line is that ICS-CERT has to decide, or maybe congress has to decide for them, what their mission is. If they are going to be the link between researchers and vendors and users so that vulnerabilities in control systems are going to be identified and fixed and control systems are going to be upgraded/updated with those fixes, then they are going to have to provide the information necessary to all three parties. For control system users that information is going to have to be as complete a definition of the vulnerability as possible so that a legitimate risk-cost-benefit decision can be made about how and when mitigation measures (including measures other than vendor supplied updates) are applied to the system.

If they are not, or for some reason cannot, be that intermediary between vendors/researchers and the user community, then we are going to have to find/establish that agency that will provide user/owners with the necessary information to make intelligent decisions about when and how patches and updates will be applied. Without that information, the vast majority of control system owners are going to assume that their systems that are working for the intended purpose are fine and updates and upgrades will be considered if and when new tools will make them worthwhile changing. And those systems will be vulnerable to attack.