NIST Publishes Initial Analysis of Framework Responses

In just over a month since the last comment was posted on the Cybersecurity Framework RFI web site NIST has published an initial review of the over 200 comments received. As one would expect from an technology oriented organization like NIST, this review was based upon an automated identification, correlation and review of specific search terms. NIST notes that:

“This initial analysis will serve as the basis for additional discussion and study at the Cybersecurity Framework Workshop #2 to be hosted at Carnegie Mellon University in Pittsburgh on May 29-31, 2013. In preparation for this workshop, we ask that all participants review the RFI submissions and this initial analysis.”

Categorizing Common Themes

The automated review/search techniques used by NIST allowed the identification of common themes within the submitted comments and the abstraction of comments related to those themes so that those comments could be grouped together for future review and analysis. (NOTE: The methodology used here should be adapted into a standard package that could be used by any federal regulatory agency for the initial analysis of large volumes of comments received in regulatory actions; 30-days to conduct this level of analysis is remarkable.)

The NIST review document breaks these comment components into three categories with a number of themes identified within each category. Those categories and themes are (note ‘X%’ refers to the percentage of comments that addressed the specific theme):

Framework Principles - Characteristics and considerations the Framework must encompass:

• Flexibility (35.8%)

• Impact on Global Operations (64.6%)

• Risk Management Approaches (81.1%)

• Leverage Existing Approaches, Standards, and Best Practices (33.3%)

Common Points - Practices identified as having wide utility and adoption:

• Senior Management Engagement (67.0%)

• Baseline Security (20.9%)

• Understanding Threat Environment (75.3%)

• Business Risk/ Risk Assessment (68.7%)

• Separation of Business and Operational Systems (60.0%)

• Models / Levels of Maturity (19.7%)

• Incident Response (27.9%)

• Cybersecurity Workforce (61.7%)

Initial Gaps - initial gaps are those areas where RFI responses were not sufficient to meet the goal of the Executive Order:

• Metrics (59.2%)

• Privacy / Civil Liberties (52.2%)

• Tools (55.9%)

• Dependencies (57.2%)

• Industry Best Practices (65.4%)

• Resiliency (46.5%)

• Critical Infrastructure Cybersecurity Nomenclature (27.1%)

Discussion of Themes

Each of the themes identified above has its own associated high-level discussion provided in this analysis document. The discussion includes:

• A brief description;

• Associated key terms and phrases;

• A brief statistical analysis;

• Examples of specific supporting comments in RFI’s; and

• A list of the associated RFI questions.

I’m not exactly sure why NIST did this, but each of the examples of supporting comments has been sanitized so that it is not possible to identify the commentor. This information is available if one were to read each of the 200+ comments, so it is not done to protect the reputation of the commentor. I suppose that if the comments were not sanitized that some people might not be able to generalize the comments to the larger universe of potentially affected organizations.

Further Discussion

If NIST really wants these comments to be a basis for the discussion at the next cybersecurity framework workshop (and the draft agenda certainly seems to indicate that) then it would be helpful if they were to make their database of extracted comments available on-line. That way all of the specific comments on a particular theme could be accessed without having to read the totality of each of the submitted comments.

Control System Themes

Of the total of 19 themes identified in this analysis, only one specifically refers to industrial control system security issues; Separation of Business and Operational Systems. The fact that this was actually identified in 60% of the comments submitted is absolutely amazing because of the small number of manufacturing organizations submitting comments. To be fair, NIST actively solicited comments on this topic with three separate questions addressing the issue.

The four abstracted comments that are included in this theme discussion are motherhood and apple pie comments supporting the separation of ICS and IT systems. I have not seen anything that addresses the very real problem of de-linking enterprise and control systems. This is certainly an area that I would like to see the complete listing of the specific comments (in 60% of the responses????) posted on this theme.

I am severely disappointed that the topics raised by Chris Blask, Chair of the Industrial Control System Information Sharing and Analysis Center (ICS-ISAC), do not fall into any of the neat categories or themes identified in this analysis. His comments on vulnerability reduction should be an important part of any framework discussion about control systems.

Not All Comments Considered

There is an interesting footnote on page one of this report; it states:

“Responses identified as spam or marketing and sales materials were not posted or reviewed by NIST.”

I certainly sympathize with the folks at NIST. Over the years writing this blog I have read a number of comments submitted to various rules that were, objectively, a complete waste of time. The reasons varied from being completely off topic, to being so poorly written as to be incomprehensible or their being political diatribes. But, these were all included in the political record of the rulemaking process.

I am particularly concerned about the exclusion of ‘marketing and sales material’. There are a number of organizations, particularly in the control system security community, that have cutting edge ideas and approaches to securing cyber-systems. While I certainly do not expect (nor would I condone) NIST to specify a specific security system or device, I think that this discussion needs to take into account the state of the art in security systems and devices. It seems to me that the exclusion of these from the public record is short sighted at best and probably legally indefensible.

Moving Forward

I think that this document produced by NIST is a valuable initial analysis of the lengthy and varied comments submitted to the agency is a very short period of time. The speed with which NIST accomplished this high-level review should be a bench mark for other regulatory actions. Actually, the mere publication of this review at this stage of the development of the framework should serve as a model for developers of regulations.

It will be interesting to see how the discussions based upon these comments at the Pittsburg workshop will turn out.