Tuesday, November 10, 2015

Reader Comments – Online CSI Applications

Over the weekend I had two comments from apparently different anonymous readers (here and here) about an error in my Friday post about Chemical Security Inspector (CSI) jobs listed on USAJobs.gov. In my post I complained about having to fax the job application into the site rather than being able to apply on-line. Both readers responded that you could apply for the job on-line. After actually trying the on-line application process I found that they were correct.

I misunderstood the portion of the instructions relating to the submission of “A complete Assessment Questionnaire” in the Required Documents portion of the job listing (the other job listing has already closed). The only other reference to that questionnaire in the listing is found above that requirement with a link to a manually completed document that is required to be faxed to the number provided. After working through the on-line application it is clear that the ‘Assessment Questionnaire’ is included in that application a format that is much easier to deal with.

As a recent expert in the field of on-line job applications I found the on-line application for this job to be a tad bit more detailed than most application that I have completed, but not onerously so. I am still disappointed in the job listing itself as it is way to bureaucratic for my taste, but I suppose that is to be expected for what is after all a bureaucratic job.

BTW: The opening that I explained above that had been closed? Well it has actually been replaced with an updated version that does not close until Thursday. So there are still two current listings for Chemical Security Inspector with a total of 21 open positions between them.

Monday, November 9, 2015

More Info on Advantech Advisory

I got an interesting email this weekend from a reader (I don’t yet have permission to use reader’s name, so we’ll just leave it at reader for the moment), Neil Smith (the researcher who is credited with reporting the vulnerability) about the recent ICS-CERT Advisory on the Advantech EKI-122X series products that I discussed last week. The reader Neil had some comments on the mitigation measures outlined in the Advisory.

The Advisory explained the mitigation measures this way:

“Advantech released new firmware in October 2015 to mitigate this vulnerability. For the EKI122* BE (v1.65) and EKI-136* (v1.27) product lines, HTTPS and SSH is disabled. For the EKI132* (v1.98) product line, additional configurations were added to allow customization for the HTTPS and SSH keys.”

The reader Neil notes that “HTTPS and SSH is disabled” means that the Advantech is “reverting back to plaintext device configuration by default, and leaving it up to the end user to configure SSL/SSH with their own keys”.

Since there is no publicly available documentation for these firmware updates, I have no way of knowing if Advantech has made this clear to the users.

It seems to me that this is actually a step backwards (if Advantech has not made this clear) in that, without additional owner/integrator actions, it will be even easier to make unauthorized changes to these Modbus gateways than it was before the update was installed. At least with the original firmware, you had to know the hardcoded password.


In closing his email the reader Smith wanted me to remind readers that “if a user updates to the latest firmware, they need to double check these services are turned back on and make sure their own certs/keys are being used”. [Updated with 'reader' name - 11-9-15, 21:05 CST]

Sunday, November 8, 2015

Plan for Failure

Yesterday Kevin Dunn from NCC Group gave an interesting talk at BSides DFW; “Plan to Fail: Failure Planning and Worst Case Thinking”. He made the point that, even after a company has properly employed the standard security best practices, there will still be a number of ways that the company can be breached. He claimed that his company (and most penetration testers) can gain enterprise domain administrator level access within six to seven hours of attempting penetration in most cases.

Breaches of Control Systems

Kevin’s presentation was mainly focused on IT systems, but a recent report from Billy Rios on his penetration testing at the Snohomish Public Utility District seems to indicate that the same is probably true for control systems. The time frame may be different, but the systems are hackable. So what is a control system owner going to have to do to protect the production system from being owned.

Kevin pointed out in his talk, that while it would be nice to keep attackers completely out of the corporate system, what is really necessary is to protect the company’s ‘secret sauce’; that information asset that, if compromised, will do severe damage to the company’s bottom line. The same is also true for control systems. While we would like to keep adversaries out of the control system completely, we must keep them out of that portion of the system that can have catastrophic results.

Problems and Controls

What is going to be considered a ‘catastrophic result’ is going to vary between companies and even locations. It is going to be something that could be a business ending result. For control system vulnerabilities it will be something that falls into one of three categories:

• Safety;
• Quality; or
• Inventory.

Safety events are going to be the easiest to identify. Fires, explosions or chemical releases are the most obvious, but death and damage can also happen at the lower end of the event size spectrum. Analog or stand-alone electronic safety systems are common mitigating measures that can be put into place to deal with these types of issues.

Quality issues are usually not considered catastrophic events, but in the pharmaceutical industry, for example, failure to control certain process variables can lead to the formation of chemical byproducts, or under formation of active ingredients, that cannot be identified by the quality tests used in a production environment. The presence of these non-standard chemicals in a drug can lead to death due to unexpected side effects or underactivity of the drug. Where these process variables have been properly identified in advance of production a facility can employ the same sort of systems used to identify and mitigate process safety incidents.

Most people completely overlook inventory events when they consider catastrophic issues. For companies working on slim margins using a just-in-time manufacturing philosophy, running out of key raw materials substantially before the planned re-supply is due to arrive will lead to unplanned facility shut downs. These shut downs and subsequent start-ups are not only very expensive (both for the facility and down-stream customers), but they are the most common times for encountering production problems that can cause additional production delays. Redundant inventory controls are the most readily available tools to prevent these sorts of problems.

Additional Security Protections

In addition to these mitigation measures, facility management should also look at putting additional security measures into place to slow the ability of an attacker to gain access to the critical systems that control the potentially catastrophic consequences. Those controls need to include monitoring tools that allow for an attack to be discovered in process rather than after it has been successful.

For the most critical systems, owners need to consider isolated standby systems to which production can be manually switched when the primary control system is breached. This is not likely to be of much use when end-point devices like PLCs have been compromised, but it the attack is identified early enough these stand-alone control systems may allow continued production or even just orderly process shutdown.

Security is More than Preventing Attacks


A well implemented security system will be able to stop most attacks on a manufacturing facility. But, since an advanced attacker will be able to bypass even the most secure system, a facility needs to take additional steps to prevent a catastrophic attack on the facility. What constitutes a catastrophic attack needs to be identified and additional security and operational controls need to be put into place to stop a successful attack.

Friday, November 6, 2015

Bills Introduced – 11-5-15

Yesterday there were 95 bills introduced in the House and the Senate as the House prepared to depart for their Veterans Day Recess. Of the bills introduced only three may be of specific interest to readers of this blog:

HR 3994 To direct the Administrator of the National Highway Traffic Safety Administration to conduct a study to determine appropriate cybersecurity standards for motor vehicles, and for other purposes. Rep. Wilson, Joe [R-SC-2]

H Con Res 92 Providing for a conditional adjournment of the House of Representatives and a conditional recess or adjournment of the Senate. Rep. Woodall, Rob [R-GA-7]

S 2249 A bill to amend title 18, United States Code, to impose criminal penalties for the unsafe operation of unmanned aircraft. Sen. Whitehouse, Sheldon [D-RI] 

HR 3994 will almost certainly include a report to Congress that might actually get used to craft appropriate legislation.

S 2249 the focus in the title on ‘unsafe operation’ sounds interesting, but is certainly prone to causing unintended consequences.


The adjournment resolution sends the House home yesterday for their Veterans Day Recess (excuse me ‘district work session’). The Senate will return to Washington on Monday and probably Tuesday before they start their recess. Both houses of Congress will be back in Washington on Monday, November 16th.

21 CSI Job Openings

There are currently two USAJobs.gov listings for Chemical Security Inspectors with DHS National Protection and Programs Division (NPPD). Between the two listings there are 21 vacancies that NPPD is trying to fill in the CFATS inspection program at various locations around the country. These listings stay open for a very short time period and close next Tuesday and Thursday respectively.

Qualifications

For the entry level (GS-09) position you must either have a Masters degree in Safety Engineering, Industrial Hygiene Inspection, Chemical Engineering, or Process Safety Engineering or 1 year of experience in:

• Conducting or assisting with on-site inspections and audits of regulated facilities to determine if violations have occurred;
• Conducting investigations using accepted inspection, enforcement, and investigative procedures; and
• Preparing reports to both private and public sector personnel on security compliance and enforcement matters.

Competencies

The job notices report that your qualification will be evaluated against the following competencies:

• Knowledge of and ability in conducting on-site physical inspections, documentary reviews, personnel interviews, and site vulnerability analysis of chemical facilities;
• Knowledge of and ability to review and evaluate the physical, personnel, information and cyber security plans and countermeasures of chemical facilities against risk based performance standards;
• Ability to communicate effectively orally; and
• Ability to communicate effectively in writing.

Salary

The salary ranges for the two positions are listed as:

$48,403.00 - $90,129.00
$48,403.00 - $84,800.00

Commentary

I have had extensive experience filling out employment applications over the last nine months and I must say that the application process through USAJobs.gov is one of the least professional, most complex and least professional (I know, I said that twice, with good justification) that I have ever seen. The first thing that you have to know is that it cannot be completed on-line. You have to print out the forms, fill them in by hand, and then fax them in to the application office.

I have not actually completed filling in the forms (since I do not meet the qualifications for entry level for either of these positions), but it looks like this could take a couple of hours. And the site clearly states that if the application is not complete in every detail, it will not be considered. In my opinion, if you can complete the application process in one go, you are fully qualified for the job on bureaucratic grounds alone.

If you are qualified for the positions, and have a desire to help chemical facilities complete the CFATS process to reduce their risk of terrorist attacks, please apply. It may sound corny, but the country really does need you.


BTW: Vacancies will be filled as funding permits – This is always a caveat for federal jobs.

Thursday, November 5, 2015

HR 22 Amended and Passed in the House

Today the House completed three days of consideration of HR 22. Instead of the originally intended 29 amendments, the House considered 117; passing 73, rejecting 35 and had 9 withdrawn from consideration after debate. The bill passed by a largely bipartisan vote of 363 to 64 (58 Republicans voted nay). After passage of the bill the House insisted on their amendments to HR 22 and requested a conference with the Senate.

Of the nine amendments that I originally identified as being of potential interest to readers of this blog, three were not considered (Lipinski, Esty, and Jackson-Lee), one (Lynch) was rejected on a near party-line recorded vote and the remaining five were adopted by voice vote.

Neither the House nor Senate will be in Washington next week (Veterans Day Recess), so it would normally be almost two weeks before the first conference committee meeting. The current surface transportation authorization runs out on November 20th so we may see conference committee meetings (out of the public eye) next week to work out the differences.

ICS-CERT Updates VxWorks Advisory and Publishes New Advantech Advisory

Today the DHS ICS-CERT updated a control system advisory for Wind River VxWorks that was originally published in June. It also published a new advisory for Advantech’s EKI-122X series products.

Wind River Update

This update provides updated information on the systems affected by the vulnerability and the mitigation measures available for Wind River devices. There is no mention of any changes in mitigation measures for Schneider products and there are no new vendors added to the list using the vulnerable VxWorks embedded software.

Three versions of VxWorks Cert have been added to the list of affected products. The Schneider Electric Sage 2300 RTU and SAGE LANDAC2 Upgrade Kit have also been added. The Schneider advisory on this vulnerability is not currently available on-line.

Patches are now available for more of the affected products, but Wind River is recommending that owners upgrade to newer versions that are not affected by the vulnerability.

VxWorks Commentary

It seems a bit odd to me that ICS-CERT has not yet identified any other vendors that are using the vulnerable VxWorks firmware. I suppose that they may know of some, but are waiting for word that a patch is available.

It sure would be nice if there were some simple test that could be performed by an owner to see if their RTU’s were subject to the TCP predictability vulnerability. Of course, since a facility may have a large number of RTU’s, the test would have to be very quick for anyone to use it in practice.

BTW: I learned of this update via a twitfication from @ICSCERT.

Advantech Advisory

This advisory describes a hard-coded SSH key vulnerability in the Advantech EKI-122X series products. The vulnerability was first reported by Neil Smith. Advantech has produced a new firmware version that mitigates the vulnerability, but there is no indication that Smith has been provided an opportunity to verify the efficacy of the fix.

ICS-CERT reports that a relatively low-skilled attacker could remotely exploit this vulnerability to intercept communications to and from the device.
 
/* Use this with templates/template-twocol.html */