The president had denied rumors about his health for months.
“Does my continued living pose a threat to some people?” he told reporters at a January event, according to the state-run Ghana News Agency.
On June 25, he returned from a medical checkup in the United States, but said he just went for a visit not medication.
A statement from his office said the 68-year-old died a few hours after being taken ill, but did not give details.
“It is with a heavy heart…that we announce the sudden and untimely death of the president of the Republic of Ghana,” the statement said.
Mr Atta Mills has ruled the west African country since 2009.
A presidential aide said Mr Atta Mills had complained of suffering pains on Monday evening and he died on Tuesday afternoon, Reuters reports.
He had returned to Ghana after visting the US for medical checks, the news agency says.
Vice-President President John Mahama’s swift inauguration underscored Ghana’s stability in a part of the world where the deaths of other leaders have sparked coups.
“We are deeply distraught, devastated as a country,” Mahama said after his swearing-in ceremony, where he raised the golden staff of office above his head.
Ghanaian state-run television stations GTV and TV3 broke into their regular programming to announce the president’s death Tuesday afternoon. Government officials did not release the cause of his death, which came three days after his 68th birthday.
Rumours had swirled about Atta Mills’ health in recent months after he made several trips to the United States, and opposition newspapers had reported he was not well enough to run for a second term.
Some radio stations even announced that he was dead during one of his recent trips to the States. When Atta Mills returned to Ghana, he jogged at the airport and blasted those who had falsely reported his death.