Yesterday the DHS ICS-CERT published a new Vulnerability
Coordination Report with data specific for FY 2015. This is being billed as an
annual report, but this is the first time the report has been published. If it
is an annual report, I would expect that the next version (for FY 2016) will be
published early next year.
Coordination Policy
While there is a great deal of statistical data for 2015 and
previous years, the most valuable information in this report is the policy
explanation for its vulnerability coordination process. The document describes
a five step process for handling vulnerability coordination (pg 1):
1. Detection and Collection:
ICS-CERT obtains vulnerability information in three ways: ICS-CERT
vulnerability analysis, monitoring public sources of vulnerability information,
and direct notification of vulnerabilities to ICS-CERT by vendors and
independent security researchers.
2. Analysis: Once ICS-CERT has
catalogued the vulnerabilities, vendor and ICS-CERT analysts work to understand
the vulnerabilities by examining and identifying the issues, as well as the
potential threat.
3. Mitigation Coordination: After
validating a reported vulnerability, ICS-CERT will continue to work with the
vendor on mitigation, including possible patch issuance. Researchers then have
the opportunity to validate solutions prior to publication.
4. Application of Mitigation:
ICS-CERT will work with the vendor to allow sufficient time for end users to
obtain, test, and apply mitigation strategies prior to disclosure. This time
window is variable depending on the circumstances of the vulnerability and the
impact to critical infrastructure.
5. Disclosure: After gathering the
technical and threat information related to the vulnerability, ICS-CERT will
notify asset owners about the vulnerability through the publication of an
ICS-CERT advisory.
The document describes an ‘advisory’ this way (pg 2):
Advisories provide timely
information about current security issues, vulnerabilities, and exploits. An
ICS-CERT
advisory
is intended to provide awareness to or solicit feedback from critical
infrastructure owners and operators concerning ongoing cyber events or
activities with the potential to impact critical infrastructure computing
networks. An advisory contains information from the researcher’s initial
report, validation of the vulnerability, a description of the vulnerability
including exploitation impact, and mitigations steps that asset owners can
apply. ICS-CERT issues an advisory after the vulnerability coordination process
has occurred. This means the researcher has contacted ICS-CERT before issuing a
public notification of their findings.
This is followed by their description of an ‘alert’:
“ICS-CERT intends for its
alerts to provide timely
notification to critical infrastructure owners and operators concerning threats
or activity with the potential to impact critical infrastructure computing
networks. ICS-CERT produces alerts based on a vulnerability discovery and the
vendor’s validation and uses them to rapidly disseminate information about a
vulnerability that someone has publicly released without coordination.”
Interestingly there is no discussion of the publicly stated
45-day disclosure policy explicated on the ICS-CERT
Vulnerability
Disclosure Policy web page. That policy states:
“In cases where a vendor is
unresponsive, or will not establish a reasonable timeframe for remediation,
ICS-CERT may disclose vulnerabilities 45 days after the initial contact is
made, regardless of the existence or availability of patches or workarounds from
affected vendors.”
That 45-day time limit, according to the data provided in
this new report shows that a ‘reasonable timeframe’ is based upon a very generous
definition of ‘reasonable’. The report notes (pg 5) that in 2015 only 27% of
the tickets opened for control system vulnerabilities were resolved in 50 days
and 29% were not resolved by 200 days into the coordination process.
Coordination with Governments
The Executive Summary of this new document states that: “The
goal of ICS-CERT is to reduce industrial control systems (ICS) risks within and
across all critical infrastructure sectors by coordinating efforts among
Federal, state, local, and tribal governments, as well as industrial control
systems owners, operators, and vendors.” There are a number of instances where
the coordination with owners, operators and vendors are discussed within this
report.
There are no indications within the report that ICS-CERT is
doing anything about coordinating efforts with ‘Federal, state, local, and
tribal governments’ unless those governmental organizations are control system
owners or operators. To be fair, I am not aware of any State, local or tribal
government organizations that specifically deal with industrial control system
security issues. The only organizations at that level that might need
vulnerability information would be criminal investigators dealing with
cyberattacks on industrial control systems. At this point I do not believe that
there are any criminal investigators currently targeting ICS attacks.
On the Federal level there is a completely different issue.
A number of Federal regulatory agencies are facing issues with control system
security problems. The Food and Drug Administration (FDA), the National Highway
Transportation Safety Administration (NHTSA) and the Federal Aviation
Administration (FAA) immediately come to mind. It would have been nice to see
some level of discussion of what efforts (if any, I suppose) that ICS-CERT is
taking to support those organizations in their regulatory efforts in dealing
with control system security issues.
Statistical Analysis
ICS-CERT has included a large amount of statistical
information about vulnerability reporting, coordination and composition within
this report. The information provided is valuable for people looking at changes
in the industrial control system security environment over time. Unfortunately,
that data presentation frequently makes that data analysis problematic.
One of my pet peeves in popular statistical analysis, for
example, is the reporting of ‘average’ numbers and tracking changes in those ‘averages’
over time. Discussing changes in average without looking at the standard deviation
(variation) associated with those averages destroys any confidence in the
analysis. In this case the large variation in the number of vulnerabilities
reported in Figure 7 (for instance) ensures that there is very likely a wide
variation in standard deviations. That could easily mean that the ‘8.55’
average reported for 2010 is essentially the same number (the famous ‘within
the margin of error’) as the ‘6.85’ average reported for 2015.
This is not a problem unique to this ICS-CERT report. It is
very common in any popular reporting of changes in average values, particularly
in reports from government agencies. Still, these types of statistical reporting
errors greatly decrease the utility of the information being presented in this
report.