The soldiers at the blast crater sensed something was wrong.
It was August 2008 near Taji, Iraq. They had just exploded a stack of old Iraqi artillery shells buried beside a murky lake. The blast, part of an effort to destroy munitions that could be used in makeshift bombs, uncovered more shells.
Two technicians assigned to dispose of munitions stepped into the hole. Lake water seeped in. One of them, Spc. Andrew T. Goldman, noticed a pungent odor, something, he said, he had never smelled before.
He lifted a shell. Oily paste oozed from a crack. "That doesn't look like pond water," said his team leader, Staff Sgt. Eric J. Duling.
The specialist swabbed the shell with chemical detection paper. It turned red - indicating sulfur mustard, the chemical warfare agent designed to burn a victim's airway, skin and eyes.
All three men recall an awkward pause. Then Duling gave an order: "Get the hell out."
Five years after President George W. Bush sent troops into Iraq, these soldiers had entered an expansive but largely secret chapter of America's long and bitter involvement in Iraq.
Weapons of mass destruction
From 2004 to 2011, U.S. and U.S.-trained Iraqi troops repeatedly encountered, and on at least six occasions were wounded by, chemical weapons remaining from years earlier in Saddam Hussein's rule.
In all, U.S. troops secretly reported finding roughly 5,000 chemical warheads, shells or aviation bombs, according to interviews with dozens of participants, Iraqi and U.S. officials, and heavily redacted intelligence documents obtained under the Freedom of Information Act.
The United States had gone to war declaring it must destroy an active weapons of mass destruction program. Instead, U.S. troops gradually found and ultimately suffered from the remnants of long-abandoned programs, built in close collaboration with the West.
The New York Times found 17 U.S. service members and seven Iraqi police officers who were exposed to nerve or mustard agents after 2003. U.S. officials said that the actual tally of exposed troops was slightly higher, but that the government's official count was classified.
The secrecy fit a pattern. Since the outset of the war, the scale of the United States' encounters with chemical weapons in Iraq was neither publicly shared nor widely circulated within the military. These encounters carry worrisome implications now that the Islamic State, a Qaida splinter group, controls much of the territory where the weapons were found.
The U.S. government withheld word about its discoveries even from troops it sent into harm's way and from military doctors. The government's secrecy, victims and participants said, prevented troops in some of the war's most dangerous jobs from receiving proper medical care and official recognition of their wounds.
"I felt more like a guinea pig than a wounded soldier," said a former Army sergeant who suffered mustard burns in 2007 and was denied hospital treatment and medical evacuation to the United States despite requests from his commander.
Congress, too, was only partly informed, while troops and officers were instructed to be silent or give deceptive accounts of what they had found. "'Nothing of significance' is what I was ordered to say," said Jarrod Lampier, a recently retired Army major who was present for the largest chemical weapons discovery of the war: more than 2,400 nerve-agent rockets unearthed in 2006 at a former Republican Guard compound.
Jarrod L. Taylor, a former Army sergeant on hand for the destruction of mustard shells that burned two soldiers in his infantry company, joked of "wounds that never happened" from "that stuff that didn't exist." The public, he said, was misled for a decade. "I love it when I hear, 'Oh there weren't any chemical weapons in Iraq,' " he said. "There were plenty."
Rear Adm. John Kirby, spokesman for Defense Secretary Chuck Hagel, declined to address specific incidents detailed in the Times investigation, or to discuss the medical care and denial of medals for troops who were exposed. But he said that the military's health care system and awards practices were under review, and that Hagel expected the services to address any shortcomings.
"The secretary believes all service members deserve the best medical and administrative support possible," he said. "He is, of course, concerned by any indication or allegation they have not received such support. His expectation is that leaders at all levels will strive to correct errors made, when and where they are made."
The discoveries of these chemical weapons did not support the government's invasion rationale.
Warheads and shells
After the terrorist attacks of Sept. 11, 2001, Bush insisted that Saddam was hiding an active weapons of mass destruction program, in defiance of international will and at the world's risk. U.N. inspectors said they could not find evidence for these claims.
Then, during the long occupation, U.S. troops began encountering old chemical munitions in hidden caches and roadside bombs. Typically 155 mm artillery shells or 122 mm rockets, they were remnants of an arms program Iraq had rushed into production in the 1980s during the Iran-Iraq war.
All had been manufactured before 1991, participants said. Filthy, rusty or corroded, a large fraction of them could not be readily identified as chemical weapons at all. Some were empty, although many of them still contained potent mustard agent or residual sarin. Most could not have been used as designed, and when they ruptured, they dispersed the chemical agents over a limited area, according to those who collected the majority of them.
In case after case, participants said, analysis of these warheads and shells reaffirmed intelligence failures. First, the U.S. government did not find what it had been looking for at the war's outset. Then it failed to prepare its troops and medical corps for the aged weapons it did find.
As Iraq has been shaken anew by violence, and as past security gains have collapsed amid Sunni-Shiite bloodletting and the rise of the Islamic State, this long-hidden chronicle illuminates the persistent risks of the country's abandoned chemical weapons.
Many chemical weapons incidents clustered around the ruins of the Muthanna State Establishment, the center of Iraqi chemical agent production in the 1980s.
Since June, the compound has been held by the Islamic State, the world's most radical and violent jihadist group. In a letter sent to the United Nations this summer, the Iraqi government said that about 2,500 corroded chemical rockets remained on the grounds, and that Iraqi officials had witnessed intruders looting equipment before militants shut down the surveillance cameras.
The U.S. government says the abandoned weapons no longer pose a threat. But nearly a decade of wartime experience showed that old Iraqi chemical munitions often remained dangerous when repurposed for local attacks in makeshift bombs, as insurgents did starting by 2004.
Participants in the chemical weapons discoveries said the United States had suppressed knowledge of finds for multiple reasons, including that the government bristled at further acknowledgment it had been wrong. "They needed something to say that after Sept. 11 Saddam used chemical rounds," Lampier said. "And all of this was from the pre-1991 era."
Others pointed to another embarrassment. In 5 of 6 incidents in which troops were wounded by chemical agents, the munitions appeared to have been designed in the United States, manufactured in Europe and filled in chemical agent production lines built in Iraq by Western companies.
Convention of Chemical Weapons
Nonproliferation officials said the Pentagon's handling of many of the recovered warheads and shells appeared to violate the Convention on Chemical Weapons. According to this convention, chemical weapons must be secured, reported and destroyed in an exacting and time-consuming fashion.
The Pentagon did not follow the steps but says it adhered to the convention's spirit.
"These suspect weapons were recovered under circumstances in which prompt destruction was dictated by the need to ensure that the chemical weapons could not threaten the Iraqi people, neighboring states, coalition forces, or the environment," said Jennifer Elzea, a Pentagon spokeswoman.
The convention, she added, "did not envisage the conditions found in Iraq."
Nonetheless, several participants said the United States lost track of chemical weapons that its troops found, left large caches unsecured, and did not warn people - Iraqis and foreign troops alike - as it hastily exploded chemical ordnance in the open air.
In early 2009, at U.S. prodding, Iraq entered the Convention on Chemical Weapons. From that moment, its fledgling government assumed primary responsibility for securing and destroying any chemical munitions remaining from Saddam's time.
Iraq took initial steps to fulfill its obligations. It drafted a plan to entomb the contaminated bunkers on Al Muthanna, which still held remnant chemical stocks, in concrete.
When three journalists from The Times visited Al Muthanna in 2013, a knot of Iraqi police officers and soldiers guarded the entrance. Two contaminated bunkers - one containing cyanide precursors and old sarin rockets - loomed behind. The area where Marines had found mustard shells in 2008 was out of sight, shielded by scrub and shimmering heat.
The Iraqi troops who stood at that entrance are no longer there. The compound, never entombed, is now controlled by the Islamic State.
Reporting was contributed by John Ismay, Duraid Ahmed, Omar al-Jawoshy, Mac William Bishop and Eric Schmitt. Alain Delaquérière contributed research.