Reader Comment – 03-07-09

Saturday we had an anonymous reader comment left on a previous response to another reader comment about an earlier blog on the DHS IG report on the effectiveness of Transportation Security Inspectors – Surface (TSIs). The current reader comment, in its entirety is shown below. “PJ, did DHS make any report data available for scrutiny? Do we know what methods the IG used to assess risk? We need a transparent government, especially if private corporate entities seem determined to keep large populations at high risk (e.g. railroads with TIH routing, food companies with salmonella sales).” IG Data and Methodology There is very little hard data in the IG report. In fact, there is no indication that the IG has done more than report the data and conclusions that the TSA has already reported. There are no claims that the IG has independently verified any of the data provided by TSA. From the IG reports that I have seen (admittedly a limited data universe for making claims) this is not unusual. Concrete data and data verification are much more common in GSA reports to Congress or reports by the Congressional Research Service. The IG report does briefly explain the data collection methodology that was used by TSA to collect their data (page 8).

“To evaluate freight rail risks, TSA created a five-phase SAI [Security Action Items] assessment program. In Phases I-III, TSIs collect data to assess freight railroads’ implementation of 24 recommended security practices (see appendix C). TSIs evaluate a freight company’s security systems and plans, employee security and identification procedures, and other practices. In Phases IV-V, TSIs visit freight railroad locations to assess employee TIH awareness, verify the location of loaded TIH cars, and determine whether loaded cars are attended. TSA uses the data to calculate the risk posed by TIH in a given High Threat Urban Area.”

The report then goes on to explain (page 9) that, as of May 2008: “TSIs have performed more than 3,000 SAI reviews of 500 freight rail facilities in 60 High Threat Urban Areas. The results of the reviews showed that TIH cars were left unattended 13% fewer hours and overall risk was reduced by 54%.” Now, a careful reading of that last sentence does not say that 13% fewer unattended hours directly led to the 54% risk reduction, but it certainly leaves that as an easy conclusion to draw. Admittedly, risk reduction is a difficult thing to accurately measure. Fred Millar in the earlier reader comment made reference to the general formula of “risk = probability x consequence”, but that is more of a philosophical statement than a real mathematical formula. Typically politicians and government agencies use surrogate equivalencies instead of actual calculations to define risk reductions; 13% fewer hours unattended = 13% risk reduction. This is a scientifically questionable evaluation, but it does have the advantage of being readily understood by politicians and the general public. In this case, unless there were additional, unreported, measures that contributed to the risk reduction TSA would have had to use a fudge factor of 4.15 to get from 13% to 54%. Politicians seldom use a fudge factor that large, it would be too noticeable. A major factor that could have contributed to that larger risk reduction number is the reduction in hours TIH shipments spent transiting the HTUA. That claim has been reported in other venues, but it is not discussed here in this report; a disturbing omission. Transparency vs Security Anonymous states that we “need a transparent government”, but transparency and security frequently run into conflict. Too much transparency and security is fatally compromised. Too little transparency and security lapses are missed. It is frequently difficult to find a safe middle ground. In the Federal government we have a system that is supposed to take care of this problem. The executive branch executes laws and regulations and the legislative branch is supposed to oversee that execution. This was part of the reason that this DHS IG report was produced; the 9/11 Implementation Act required the report to Congress. Now almost all of the data that TSA used to compile their risk reduction data would have been labeled Sensitive Security Information (SSI) which is treated as classified under Federal rules. Much of the data compiled from that SSI information would still be classified, but at some level the data could no longer be traced back to individual facilities and would no longer deserve the SSI classification. This is why the 13% figure mentioned above was releasable. Since the Homeland Security Committee members and staffers would be expected to have the requisite security clearances, the way around producing a publicly viewable document while providing the necessary supporting information to justify the conclusions would be to include a classified annex containing that data. That annex would be made available to Congress. It does not appear that such an annex was made available with this report. Congressional Response Since the production of this report was a requirement laid upon DHS by Congress, it is only Congress, or the House Committee on Homeland Security in this case, that can judge whether or not this report met their requirements. While neither Anonymous nor I are comfortable with the lack of supporting documentation, it is not up to us. Only Chairman Thompson and his staffers may judge the adequacy of this effort. One final interesting point; this report was prepared by the Bush DHS. It was written before November 26th, 2008 even though it was dated February 2009. You can discern this by the references to the ‘yet to be finalized’ TSA freight security rule. The delay between the writing and publishing of the report is no doubt due to the staff reviews that were required before it was released. In this case we must assume that it was reviewed by Secretary Napolitano’s office because of the release date.