President Barack Obama challenged Latin American leaders on Saturday to improve human rights and democracy even as he works to end decades of hostility between the United States and communist-run Cuba.

Obama has won praise across much of Latin America for seeking to restore diplomatic relations with Cuba and he shook President Raul Castro's hand in a show of the detente on Friday night, but he took a tougher line on Saturday in a speech at the Summit of the Americas in Panama.

"I believe our governments together have an obligation to uphold the universal freedoms and rights of all our citizens," Obama told other leaders from across the Americas. "The voices of our citizens must be heard."

Responding to other leaders' criticism of past U.S. policy in Latin America, including its support of military coups and dictatorships during the Cold War, Obama said Washington's record was far from perfect but that it has changed and he would continue to push for greater democracy.

"I just want to make very clear that when we speak out on something like human rights, it's not because we think we are perfect but it is because we think the idea of not jailing people if they disagree with you is the right idea," Obama said.

Waving his hands for emphasis, Castro condemned the United States for its attempts to topple communist rule on the island but he praised Obama as "an honest man" and said he was not to blame for U.S. policies during the Cold War.

"I apologize to President Obama, because he is not responsible for any of this," the 83-year-old Castro said.

Obama and Castro were expected to meet later on Saturday to discuss progress in their goal, announced by both men in December, of restoring full diplomatic relations and freeing up trade and travel between the two countries.