Biz & IT —

How to blow $6 billion on a tech project

Military's 15-year quest for the perfect radio is a blueprint for failing big.

The Army's cancelled Ground Mobile Radio cost $6 billion to fail.
The Army's cancelled Ground Mobile Radio cost $6 billion to fail.

In 1997, the Defense Department began its quest for the perfect family of radios: software-defined radios that, like computers, could be reprogrammed for different missions and could communicate with everything the US military used. Digital signal processing could adaptively use available radio spectrum based on the needs of the moment, turning soldiers, tanks, planes, and ships into nodes of a broadband radio-based network.

The goal was to solve radio problems like this one in Afghanistan, detailed by the Center for Public Integrity in January 2012. Soldiers who watched an ambush forming on a ridge nearby found themselves limited by the hugely variable needs of their many radio systems:

They had short-range models for talking with the reconstruction team; longer-range versions for reaching headquarters 25 miles away; and a backup satellite radio in case the mountains blocked the transmission. An Air Force controller carried his own radio for talking to jet fighters overhead and a separate radio for downloading streaming video from the aircraft. Some of these radios worked only while the troopers were stationary; others were simply too cumbersome to operate on the move.

But the program meant to fix the mess, called the Joint Tactical Radio System (JTRS), instead became a massive 15-year software and hardware development mess of its own, involving five sub-programs and multiple multi-billion dollar contracts. It has been a financial disaster for the DOD. Billions were thrown away on technology that will never see the light of day, despite multiple heroic efforts to pull the project back from the brink of disaster.

JTRS provides a textbook case of what not to do in a technology development program, proving that even a few great ideas can’t save a project that has been over-specified and under-tested, and that remains blinkered to what's going on in the world around it.

Pyrrhic victory

In May 2012, the Joint Program Executive Office JTRS, the DOD office that manages JTRS, announced what should have been a crowning achievement of the 15-year program: hardware certification. The Ground Mobile Radio (GMR), one of the main JTRS sub-programs and the first radio to use the military’s new Wideband Networking Waveform (WNW) standard, had been certified for use by the National Security Agency.

“This achievement enables the Department of Defense to harness years of investment and technological progress associated with the GMR development,” said a JTRS Joint Program Executive Office spokesperson in a statement at the time.

GMR formed the cornerstone of the Army's plan to build mobile ad hoc networks that connected troops in the field to each other, as well as to Army and Air Force aircraft flying in support, and to commanders back in operations centers. With WNW as the network's backbone, the Soldier Radio Waveform for communicating to handheld and other mobile radios, and waveforms for satellite and ground-to-air communications as well, the GMR would be the router on the battlefield Internet. And now it was ready to go.

Well, sort of. After burning through over $6 billion just to develop GMR, the Army had actually cancelled the project back in October 2011 after it failed the Army’s Network Integrated Environment (NIE) testing. Big, bloated, and complex, the GMR couldn’t handle the heat of White Sands, New Mexico, and it proved to be less "iPhone" than "mainframe" in complexity.

In a July 2012 article in the National Defense Review, Air Force Lt. Colonel Dan Ward, an acquisition officer deployed in Afghanistan, bluntly spelled out the failings of the GMR:

The NIE exercise highlighted the uncomfortable fact that soldiers might sometimes have to send critical messages during a fight. An impatient lot, they may not be terribly excited about waiting 10 minutes for the radio to run through a slow series of boot-up processes… If someone were to make a television commercial for the GMR, it would undoubtedly include a line about sophistication being worth the wait. Except in combat, sometimes you really can’t wait. Too bad the GMR was supposed to go to war, because it would apparently be a pretty sweet system in an environment where the temperature and operational tempo are both low.

So although GMR is "certified for use," the Army doesn't actually want to use it. And the system isn’t done costing the government money. While the Army waited years for GMR, it spent $11 billion more on “legacy” radios based on technology from the Cold War. As Ward pointed out, the Army may yet pay billions more to get the radios it actually needs—just as the war in Afghanistan is winding down.

The GMR, broken down in schematic form. Who wouldn't want to lug this around the battlefield?
The GMR, broken down in schematic form. Who wouldn't want to lug this around the battlefield?
Department of Defense

Open source radio

At its core, JTRS was built on what sounded like a good idea: separating radio standards from transmission hardware by creating an open “operating system” common to all military radios. If radios’ waveforms were defined in software rather than hardware, they could be moved like applications from one set of hardware to another, or upgraded with a simple software update.

Taking a page from open source licensing schemes like GNU, the core software architecture and the “waveforms”—application-specific formats for radio communication—were made available to anyone building radios for the DOD, with the provision that any implementation of the code be contributed back to the JTRS code repository. It was a success, if success can be measured by number of contributions: that repository now holds four million lines of code.

The core of JTRS is the Software Communications Architecture, an application framework for radios built on the Common Object Request Broker Architecture (CORBA) and a POSIX-based real-time operating system. (SCA has managed to find a life outside of DOD in an open-source implementation that runs on Windows and Linux, called OSSIE.)

The idea behind the open architecture was to ensure that all the communications gear that each of DOD’s services bought would work together seamlessly. And since the core software and the waveforms would be freely available to anyone—even companies that didn’t get in on the main JTRS programs—the overall cost of getting new radios should fall dramatically by allowing competition between manufacturers based on a well-defined standard. In other words, JTRS was supposed to commoditize the software-defined radio market.

When I spoke with then-JTRS program executive Dennis Bauman back in 2008, he said that the open code base allowed JTRS to “enroll” new vendors on a rolling basis. While there was still an open competition for the design phase of each component radio program, manufacturing would always have more than one option. “We require that we qualify at least two sources,” Bauman said; those qualified vendors would compete annually to manufacture radios in lots.

The Harris Falcon III handled "JTRS-compliant" radio, developed independently by Harris with access to the JTRS code repository.
The Harris Falcon III handled "JTRS-compliant" radio, developed independently by Harris with access to the JTRS code repository.
Harris Corp.

That approach paid off, at least with some of the first transitional handheld radios that could interoperate with the broader JTRS program. For example, in 2007, Thales won the contract for the Consolidated Interim Single Channel Handheld Radio (CISCHR)—a project merged into JTRS from the US Special Operations Command. But Harris developed a version of its Falcon III radios on its own that was "JTRS Compliant" and could use the Soldier Radio Waveform, as well as other legacy radio formats that were ported to the architecture, by using the common JTRS repository. Harris sold over 100,000 “JTRS-compliant” radios to the Army and Marine Corps in open competition with the "official" Thales radio; the competition between the two for JTRS business saved the Army $104 million on the first batch of 39,000 radios alone.

If JTRS had just been a software standards effort focused on waveforms, this might be an article about the “genius of JTRS.” Unfortunately, it wasn’t. And this isn’t. The various services, especially the Army, weren’t content just to have companies compete on the software standard. They increasingly wanted more hardware features as well.

Some of the most fundamental problems with JTRS (and especially with the GMR sub-program) are common to any large, failing technology project—only on a more massive scale. The scope of JTRS was so huge from day one that no manipulations of the “iron triangle” of project management could keep cost and schedule under control.

Inadequate understanding

Military technology often pushes at the edge of the doable—that’s why agencies like DARPA exist.  But when JTRS was originally conceived in 1997, the Army's most significant previous work on software-defined radio was an Army project called SpeakEasy—the first version of which took up the entire back of a military truck. Clearly, an ambitious project like JTRS had plenty of basic research to chew through before it could produce anything useful. In hindsight, the military badly underestimated the challenges before it.

"In the end, building a radio that worked with all the different waveforms envisioned by the project required bending some fundamental rules of physics."

In a letter sent to the chairman of the House Armed Service Committee in October 2011 explaining the cancellation of the GMR program, acting Undersecretary of Defense for Acquisition, Technology, and Logistics Frank Kendall wrote that “the technical challenges of mobile ad hoc networks and scalability were not well understood due to the immaturity of the technology” when the program began.

First and foremost was the software development problem. When JTRS started, software-defined radio (SDR) was still in its infancy. The project's SCA architecture allowed software to manipulate field-programmable gate arrays (FPGAs) in the radio hardware to reconfigure how its electronics functioned, exposing those FPGAs as CORBA objects. But when development began, hardware implementations of CORBA for FPGAs didn’t really exist in any standard form.

Moving code for a waveform from one set of radio hardware to another didn’t just mean a recompile—it often meant significant rewrites to make it compatible with whatever FPGAs were used in the target radio, then further tweaking to produce an acceptable level of performance. The result: the challenge of core development tasks for each of the initial designs was often grossly underestimated. Some of those issues have been addressed by specialized CORBA middleware, such as PrismTech’s OpenFusion, but the software tools have been long in coming.

Then there was the range of requirements demanded by the waveforms themselves. Each waveform called for different frequencies; while the radio’s electronics could be software-configured for those, significantly different frequencies required divergent antenna properties and modifications to the radio hardware itself. In the end, building a radio that worked with all the different waveforms envisioned by the project required bending some fundamental rules of physics.

Having an undefined technical problem is bad enough, but it gets even worse when serious "scope creep" sets in during a 15-year project.

Channel Ars Technica