New York Times
Sat Nov 23, 2013
Eric Lipton and Sharon LaFraniere
Excerpts
CGI Federal, the prime contractor for building the healthcare.gov web site, had concluded long ago that the goal of creating a cutting-edge website before Oct 1, 2013, that would use the latest technologies to dazzle consumers with its many features was. Knowing how long it would take to complete and test the software, the company’s officials and other vendors believed that it was impossible to open a fully functioning exchange on Oct. 1. The Obama Administration insisted that Oct. 1 was not negotiable and they were fed up with what they saw as CGI’s pattern of excuses for missed deadlines.
Vital components were never secured, including sufficient access to a data center to prevent the website from crashing. A backup system that could go live if it did crash was not created, a weakness the administration has never disclosed. And the architecture of the system that interacts with the data center where information is stored is so poorly configured that it must be re-designed, a process that experts said typically takes months. An initial assessment identified more than 600 hardware and software defects — “the longest list anybody had ever seen,” one person involved with the project said.
When the realization of impending disaster finally hit government officials at the Aug. 27 meeting — just 34 days before the site went live — they threw out nearly 30 requirements, including the Spanish-language version of the site and a payment system for insurers to receive government subsidies for the policies they sold.
Even then, the system failed a test of only 500 simulated users in late September. Panicked, agency officials sent out an urgent order to almost double the system’s data capacity, technicians involved in the project have now confirmed. But the site was still down more than half the time in mid-October.
CGI and other contractors complained of endlessly shifting requirements and a government decision-making process so cumbersome that it took weeks to resolve elementary questions, such as determining whether users should be required to provide Social Security numbers. Some CGI software engineers ultimately walked out, saying it was impossible to produce good work under such conditions.
Another sore point was the Medicare agency’s decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.
The Medicare agency was not everyone’s first choice to run the $630 million project. White House officials at first debated whether to name an outsider, such as Jon Kingsdale, who set up the landmark Massachusetts health insurance program, or even to create a new agency.
Both those ideas fell through, and over the past three years five different lower-level managers held posts overseeing the development of HealthCare.gov, none of whom had the kind of authority to reach across the administration to ensure the project stayed on schedule.
As a result, the president’s signature initiative was effectively left under the day-to-day management of Henry Chao, a 19-year veteran of the Medicare agency with little clout and little formal background in computer science.
Mr. Chao had to consult with senior department officials and the White House, and was unable to make many decisions on his own. “Nothing was decided without a conversation there,” said one agency official involved in the project, referring to the constant White House demands for oversight. On behalf of Mr. Chao, the Medicare agency declined to comment.
Sixteen companies were prequalified to bid on the project, according to administration officials. CGI was picked as the prime contractor over three other bidders: IBM, QSSI and Computer Sciences Corporation. But the Medicare agency reserved the role of general contractor, or system integrator, for itself, even though it lacked the necessary in-house software engineering resources to handle such a task.
A pattern of ever-shifting requirements persisted throughout the project, including the administration’s decision late last year to try to redesign the site’s appearance and content to make it more informative to consumers, according to many specialists involved. The administration also decided to reconfigure it as a national site, instead of one where each state had its own front page, after many states decided not to open their own exchanges.
Medicare agency officials began to suspect that staff members at CGI were intentionally trying to hide flaws in the system, to cover up for their inability to meet production deadlines. They ordered CGI technicians to drive from their offices near Dulles International Airport in Virginia to the agency headquarters near Baltimore to review their code with government supervisors.
The Medicare agency was also growing frustrated with tension among contractors, noting that initial tests of parts of the system were being delayed because of “coordination issues” between CGI and QSSI, which won another part of the job after losing the lead contractor role.
Mr. Chao seemed to colleagues to be at his wit’s end. One evening last summer, he called Wallace Fung, who retired in 2008 as the Medicare agency’s chief technology officer. Mr. Fung said in an interview that he told Mr. Chao to greatly simplify the site’s functions. “Henry, this is not going to work. You cannot build this kind of system overnight,” Mr. Fung said he told him.
“I know,” Mr. Chao answered, according to Mr. Fung. “But I cannot talk them out of it.”
In the last week of September, the disastrous results of the project’s inept management and execution were becoming fully apparent. The agency pressed CGI to explain why a performance test showed that the site could not handle more than 500 simultaneous users. The response once again exhibited the blame-shifting that had plagued the project for months.
“We have not identified any inefficient and defective code,” a CGI executive responded in an email to federal project managers, pointing again to database technology that the Medicare agency had ordered it to use as the culprit, at least in part.
Despite the behind-the-scenes crisis, the president expressed confidence about the exchange just days before its debut.
“This is real simple,” Mr. Obama said, during a speech in Maryland on Sept. 26. “It’s a website where you can compare and purchase affordable health insurance plans side by side the same way you shop for a plane ticket on Kayak, same way you shop for a TV on Amazon. You just go on, and you start looking, and here are all the options.”
Vanna Le
Kate Pierce
Lauren Gensler