By Eduardo Simões and Caroline Stauffer

RIO DE JANEIRO (Reuters) - Armed robbers posing as police officers held a gun to the forehead of U.S. gold medalist swimmer Ryan Lochte in a taxi hold-up in Rio de Janeiro on Sunday, feeding concern over safety at the first Olympic Games in South America.

Lochte was returning to the Athletes' Village from a party hosted by the French Games delegation, with swimming team mates Gunnar Bentz, Jack Conger and Jimmy Feigen, when armed men carrying police badges pulled them over.

The men told the swimmers to drop to the ground and demanded their money and belongings, Lochte said. When he refused, one of the men cocked a pistol and placed it against the swimmer's head, and he then handed over his wallet.

"While it is true that my teammates and I were the victims of a robbery early Sunday morning, what is most important is that we are safe and unharmed," Lochte said in a statement.

He said he was looking forward to returning to the United States and starting training with an eye to competing at the Tokyo Games in 2020.

The incident stoked worries over the safety of competitors and visitors to Rio, which has a long history of violent crime, though Brazil's sports minister insisted that athletes who remained on the Olympic site had experienced no problems.

"The security operation for these Games has been totally effective," Leonardo Picciani told reporters. "No athlete has had problems in their place of residence, in training or in the Athletes' Village."

Police are investigating the incident and trying to locate the taxi driver involved, police sources said.

In addition to the swimmers, Swedish tourists were briefly abducted when they visited a slum, Portugal's visiting education minister was robbed at knife point, bullets flew into the equestrian center and a Games bus was attacked with stones.

Lochte thanked his friends, family and fans for their support after the unsettling incident.

"The guy pulled out his gun, he cocked it, put it to my forehead and he said, 'Get down,' and I put my hands up, I was like 'whatever'," he told NBC. "He took our money, he took my wallet — he left my cell phone, he left my credentials."

Lochte's mother said the swimmer was unharmed but shaken.

"I think they're all shaken up. There were a few of them," Ileana Lochte told USA Today. "They just took their wallets and basically that was it."

All four athletes are cooperating with authorities, U.S. Olympic Committee spokesman Patrick Sandusky said.

Fellow American swimmers Ryan Murphy and Nathan Adrian told reporters that they felt security for the Games was adequate, despite the incident.

"Rio is an amazing city. There are going to be problems anywhere you go and we've been briefed on how to mitigate those risks as best possible," Adrian told a news conference.

(Reporting by Eduardo Simoes and Caroline Stauffer; Editing by Ossian Shine, Ed Osmond and Meredith Mazzilli)