Railhead, TIDE, and Government Terrorist Databases

The National Counterterrorism Center is running a program called 'Railhead' to upgrade the government's mater list of terrorists. Yes, there is a centralized master list, which is then used to create more targeted watchlists like the TSA's infamous no-fly list.

There are problems with the Railhead program, however. Rep. Brad Miller, D-NC, has authored a letter (here) pointing out some serious issues with the NTC program.

Miller, who chairs the Investigations and Oversight Subcommittee of the House Science and Technology Committee, cited "severe technical troubles, poor contractor management, and weak government oversight," which he said had brought the Railhead program to the "verge of collapse."

Railhead is supposed to replace the aging Terrorist Identities Datamart Environment (TIDE), originally built by Lockheed Martin. TIDE may have been functional and effective when it was delivered, but ad hoc developemnt practices and shoddy program management have seriously degraded its effectiveness.

Data tables have been added to the original TIDE database, without concern for duplication of data fields or proper indexing. As a result, over half of the 463 tables are not documented or indexed. What does that mean? Without a data dictionary (something that lets users know what is stored where, why, and how, in a database) it is nearly impossible for new users and admins to grasp just exactly what is stored in those ad hoc tables. This will result in data not be returned during database searches and a lack of database integrity.

Without a detailed index of the data stored in each table in TIDE, the SQL search engine is blindfolded, unable to locate or identify undocumented data.

For a mission as important as TIDE, seeing reports that the technology side has been so poorly managed just makes me shake my head.

So Railhead was supposed to come behind and fix everything, right? Well, in theory. But there are problems, of course.

First and foremost, there are accusations of financial misconduct and a lack of oversight within the Railhead project. But more concerning to me is that decisions are being made at a technical or program level that may render Railhead even less effective than TIDE.

Many of the undocumented tables in TIDE have been added over the years as 'enhancements' to the system, with the overall goal of improving effectiveness. As mentioned before, that has resulted in data being stored in multiple places, what I call 'data fracturing'.

The Railhead program cannot make sense of all this fractured data, and it appears that project managers have decided to just disregard it out of hand.

"Pocket liter [sic]," for instance, the scraps of information obtained when law enforcement, military or other officials empty a suspect's pockets, including phone numbers, addresses or credit card information, is contained in 23 seperate tables in TIDE, rather than one single uniform table. But as problematic as the current TIDE system is, counterterrorism analysts may lose access to key data if the new Railhead system comes on line as planned at the end of 2008. "Specifically, users will no longer have access to data that will not be migrated [to the new system], such as pocket litter and border summaries," another recent Railhead document warns.

So, Railhead will not migrate everything from TIDE? Why not? Is that a technical decision? A policy decision? Whatever it is, it seems to be a step back from where we are now.

TIDE is built in Oracle, one of several industry standard relational database management systems (RDBMS). It has (well, it had, prior to the creation of the ad hoc tables) a defined relationship structure, which lays out which tables are related to which other tables, and how. That is what relational databases are designed to do. Railhead is abandoning an RDBMS approach in favor of storing the data in an XML format. XML (eXtensible Markup Language) is a method of marking up data to describe what the data is and what it does. It can be like a database... but it isn't a database. There are concerns that the new Railhead will have inflated storage requirements and poor performance due to the decision to use XML. I tend to agree with those fears.

XML is a fantastic tool to transfer data in a neutral format from system to system, since it carries with it an internal description of what the data is. However, it does not contain, necessarily, a descriptor that explains how that data relates to other data.

So we have a deployed IT system that is a technical mess (TIDE), being replaced by a new system that is having serious development problems and is suffering from some questionable policy decisions (Railhead). Where does that leave us?

References:
Ars Technica Article
Rep. Miller Letter
Subcommittee on Investigations and Oversight Staff Memo
CNET Article