ICS-CERT Publishes Vulnerability Coordination Report

Yesterday the DHS ICS-CERT published a new Vulnerability Coordination Report with data specific for FY 2015. This is being billed as an annual report, but this is the first time the report has been published. If it is an annual report, I would expect that the next version (for FY 2016) will be published early next year.

Coordination Policy

While there is a great deal of statistical data for 2015 and previous years, the most valuable information in this report is the policy explanation for its vulnerability coordination process. The document describes a five step process for handling vulnerability coordination (pg 1):

1. Detection and Collection: ICS-CERT obtains vulnerability information in three ways: ICS-CERT vulnerability analysis, monitoring public sources of vulnerability information, and direct notification of vulnerabilities to ICS-CERT by vendors and independent security researchers.

2. Analysis: Once ICS-CERT has catalogued the vulnerabilities, vendor and ICS-CERT analysts work to understand the vulnerabilities by examining and identifying the issues, as well as the potential threat.

3. Mitigation Coordination: After validating a reported vulnerability, ICS-CERT will continue to work with the vendor on mitigation, including possible patch issuance. Researchers then have the opportunity to validate solutions prior to publication.

4. Application of Mitigation: ICS-CERT will work with the vendor to allow sufficient time for end users to obtain, test, and apply mitigation strategies prior to disclosure. This time window is variable depending on the circumstances of the vulnerability and the impact to critical infrastructure.

5. Disclosure: After gathering the technical and threat information related to the vulnerability, ICS-CERT will notify asset owners about the vulnerability through the publication of an ICS-CERT advisory.

The document describes an ‘advisory’ this way (pg 2):

Advisories provide timely information about current security issues, vulnerabilities, and exploits. An ICS-CERT advisory is intended to provide awareness to or solicit feedback from critical infrastructure owners and operators concerning ongoing cyber events or activities with the potential to impact critical infrastructure computing networks. An advisory contains information from the researcher’s initial report, validation of the vulnerability, a description of the vulnerability including exploitation impact, and mitigations steps that asset owners can apply. ICS-CERT issues an advisory after the vulnerability coordination process has occurred. This means the researcher has contacted ICS-CERT before issuing a public notification of their findings.

This is followed by their description of an ‘alert’:

“ICS-CERT intends for its alerts to provide timely notification to critical infrastructure owners and operators concerning threats or activity with the potential to impact critical infrastructure computing networks. ICS-CERT produces alerts based on a vulnerability discovery and the vendor’s validation and uses them to rapidly disseminate information about a vulnerability that someone has publicly released without coordination.”

Interestingly there is no discussion of the publicly stated 45-day disclosure policy explicated on the ICS-CERT Vulnerability Disclosure Policy web page. That policy states:

“In cases where a vendor is unresponsive, or will not establish a reasonable timeframe for remediation, ICS-CERT may disclose vulnerabilities 45 days after the initial contact is made, regardless of the existence or availability of patches or workarounds from affected vendors.”

That 45-day time limit, according to the data provided in this new report shows that a ‘reasonable timeframe’ is based upon a very generous definition of ‘reasonable’. The report notes (pg 5) that in 2015 only 27% of the tickets opened for control system vulnerabilities were resolved in 50 days and 29% were not resolved by 200 days into the coordination process.

Coordination with Governments

The Executive Summary of this new document states that: “The goal of ICS-CERT is to reduce industrial control systems (ICS) risks within and across all critical infrastructure sectors by coordinating efforts among Federal, state, local, and tribal governments, as well as industrial control systems owners, operators, and vendors.” There are a number of instances where the coordination with owners, operators and vendors are discussed within this report.

There are no indications within the report that ICS-CERT is doing anything about coordinating efforts with ‘Federal, state, local, and tribal governments’ unless those governmental organizations are control system owners or operators. To be fair, I am not aware of any State, local or tribal government organizations that specifically deal with industrial control system security issues. The only organizations at that level that might need vulnerability information would be criminal investigators dealing with cyberattacks on industrial control systems. At this point I do not believe that there are any criminal investigators currently targeting ICS attacks.

On the Federal level there is a completely different issue. A number of Federal regulatory agencies are facing issues with control system security problems. The Food and Drug Administration (FDA), the National Highway Transportation Safety Administration (NHTSA) and the Federal Aviation Administration (FAA) immediately come to mind. It would have been nice to see some level of discussion of what efforts (if any, I suppose) that ICS-CERT is taking to support those organizations in their regulatory efforts in dealing with control system security issues.

Statistical Analysis

ICS-CERT has included a large amount of statistical information about vulnerability reporting, coordination and composition within this report. The information provided is valuable for people looking at changes in the industrial control system security environment over time. Unfortunately, that data presentation frequently makes that data analysis problematic.

One of my pet peeves in popular statistical analysis, for example, is the reporting of ‘average’ numbers and tracking changes in those ‘averages’ over time. Discussing changes in average without looking at the standard deviation (variation) associated with those averages destroys any confidence in the analysis. In this case the large variation in the number of vulnerabilities reported in Figure 7 (for instance) ensures that there is very likely a wide variation in standard deviations. That could easily mean that the ‘8.55’ average reported for 2010 is essentially the same number (the famous ‘within the margin of error’) as the ‘6.85’ average reported for 2015.

This is not a problem unique to this ICS-CERT report. It is very common in any popular reporting of changes in average values, particularly in reports from government agencies. Still, these types of statistical reporting errors greatly decrease the utility of the information being presented in this report.