By Andrew Cawthorne and Andreina Aponte

CARACAS (Reuters) - Venezuela's unpopular socialist leader Nicolas Maduro shook hands with opposition leaders at Vatican-convened talks on Sunday, but his wary foes threatened to boycott further meetings if some demands were not quickly met.

The irate opposition Democratic Unity coalition has escalated protests against Maduro after authorities scuttled a recall referendum that polls show he would have lost, triggering a presidential election.

Their top demand is to revive the plebiscite.

Opponents say 17 years of socialism have wrecked the OPEC nation's economy and crushed democracy, while the government says a U.S.-backed elite is seeking a violent coup.

Maduro, 53, who won election to succeed Hugo Chavez in 2013, arrived mid-evening for the meeting at a Caracas museum.

He cordially greeted the five opposition leaders, including coalition secretary-general Jesus Torrealba and opposition governor Henri Falcon.

Also attending were a Vatican envoy, representatives of the Unasur regional bloc, and three former heads of state or government from Spain, Panama and the Dominican Republic.

"The Pope is following the situation of this country very closely and hopes this process can continue peacefully," said papal envoy Archbishop Claudio Maria Celli.

Venezuela's opposition is also demanding freedom for political prisoners, humanitarian aid amid an unprecedented economic downturn, and respect for the opposition-led National Assembly.

The opposition delegation would "demand an end to the repression and persecution of democrats and the people, and will walk away from the dialogue if the demands are not resolved in the short term," the coalition said in a statement.

It termed the talks an "exploratory meeting" only.

'NO ALTERNATIVE TO DIALOGUE'

Maduro, who aides said had skipped Saturday's Ibero-American summit in Colombia to prepare for Sunday's meeting, only stayed for a few minutes.

"There is no alternative to dialogue and meeting in search of the nation's common interests," he said, before walking to the opposition delegation for formal handshakes in a ceremony televised live across Venezuela.

He has distanced himself from the quashing of the referendum, saying it was a decision of independent judicial and electoral authorities based on allegations of fraud in an initial opposition signature drive.

Those institutions have for years leant towards the government, however, and many Venezuelans believe it was a coordinated ploy to outwit the opposition.

In advance, some opposition leaders had been skeptical about talks, saying Maduro had become a dictator by denying a vote and was only promoting dialogue to buy time.

The hardline opposition Popular Will party of jailed protest leader Leopoldo Lopez abstained from the talks.

"We have always believed in dialogue ... but there has been no sign of rectification by the regime," it said.

Previous sit-downs between the two sides in recent years eased some tensions in the streets, but did little for real rapprochement or progress on core issues.

Senior Socialist Party official Jorge Rodriguez, who was at Sunday's talks, said the government hoped to persuade its foes to renounce street violence and reject neo-liberal economics like those being applied in Argentina and Brazil.

A swing to the right in South America has left Maduro more isolated, though he still counts on friends in leftist-led countries from Cuba to Bolivia.

Sunday's meeting followed massive opposition marches last week, which led to scores of injuries and arrests plus the death of a policeman, and a partially-successful national strike.

This week, the opposition is planning a march on the Miraflores presidential palace, drawing government accusations it wants to reprise a short-lived 2002 coup against Chavez.

Maduro's predecessor, who died of cancer after a 14-year rule, allowed and won a recall referendum in 2004.

(Additional reporting by Daniel Kai, Efrain Otero; Editing by Peter Cooney and Mary Milliken)