By Steve Holland and Emily Stephenson

ST. LOUIS (Reuters) - Donald Trump, defiant amid a tide of criticism of his sexually aggressive remarks about women, said Hillary Clinton would go to jail if he were president and attacked her husband for his treatment of women in a vicious presidential debate less than a month before the U.S. election.

The Sunday night debate, the second of three before the Nov. 8 vote, was remarkable for the brutal nature of the exchanges between Trump, the Republican presidential nominee, and Clinton, his Democratic rival.

The New York businessman called Clinton a "devil" who repeatedly lies, someone with tremendous hate in her heart. The former secretary of state called Trump an abuser of women who is unfit for the White House.

There was a palpable sense of mutual contempt as they stood on stage, refusing to shake hands at the start. Moderators Anderson Cooper of CNN and Martha Raddatz of ABC both seemed at points to be grimly watching two trains collide.

Through it all, Trump, 70, and Clinton, 68, both landed punches as they clashed over taxes, healthcare, U.S. policy in the Syria civil war and Clinton's comments that half of Trump's supporters belonged in a "basket of deplorables."

Trump took the stage in St. Louis, Missouri, at the most perilous time of his 16-month-old candidacy.

He gave a more disciplined performance than at the first debate two weeks ago, but left Republicans torn over whether to publicly abandon a badly wounded candidate who is endangering closely contested congressional races, or to stand behind him in the dimming hope he can still win them the White House.

A 2005 video, made public on Friday, of Trump making predatory remarks about women prompted a stampede of Republican politicians to abandon him during the weekend.

On Monday, Trump campaign manager Kellyanne Conway called Trump's comment over jailing Clinton "a quip." She told MSNBC what Trump critics have already noted, that, "whether she goes to jail is not up to Donald Trump."

Clinton campaign manager Robby Mook told NBC that Trump had not appeared to take the video seriously. "Donald Trump came to try to rescue his spiraling campaign," Mook said.

REPUBLICAN CONCERNS

Even after the debate, party leaders remained concerned that Trump had not shown enough contrition over the remarks to win over independents and women voters who could decide the election.

He again described the comments as "locker-room talk" and said he had never kissed or groped women without their permission, despite having bragged about doing so on the video that emerged on Friday.

President Bill Clinton had done worse to women, Trump said in one of several forceful attacks that may reinforce his popularity with his core supporters who detest the Democratic nominee.

Hillary Clinton responded that Trump's comments showed he is unfit for the White House.

"He has said the video doesn’t represent who he is but I think it’s clear to anyone who heard it that it represents exactly who he is," Clinton said.

She accused Trump of dodging a discussion of policy issues to avoid talking about his campaign because of “the way yours is exploding and Republicans are leaving you."

A CNN/ORC snap poll of debate watchers found that 57 percent thought Clinton won the encounter, versus 34 percent for Trump. U.S. stock futures and the Mexican peso jumped as markets saw less chance of a Trump victory.

Trump's plans to slap tariffs on imports and renegotiate the North American Free Trade Agreement (NAFTA) are seen as negative for Mexico and Canada, which is why their currencies swing when his odds of winning change.

TRUMP VOWS PROSECUTION

Early in the 90-minute debate, Trump said he would appoint a special prosecutor to look into Clinton's use of a private email server during her tenure as President Barack Obama's secretary of state from 2009-2013.

Clinton said, "You know it’s just awfully good that someone with the temperament of Donald Trump is not in charge of the law in this country.” Trump shot back: “Because you’d be in jail.”

A nearly year-long FBI investigation into the emails concluded earlier this year that no charges should be filed, although FBI Director James Comey said Clinton had been careless in her handling of sensitive material.

In a startling admission, Trump dismissed a statement from his vice presidential running mate, Mike Pence, last week in which Pence said the United States should be prepared to intervene militarily in Syria.

"He and I haven't spoken, and I disagree," Trump said.

Pence, already the subject of rumors that he might bolt the Trump ticket in disgust at the lewd video, tried to quell the rumors by praising Trump on Twitter after the debate.

On Monday, he stood by Trump and said his position on Syria had been mischaracterized in Sunday's debate.

"I'm proud to stand shoulder-to-shoulder with him," Indiana Governor Pence said on Fox News, adding that he looked forward to hitting the campaign trail later on Monday.

Responding to an Oct. 1 New York Times story, Trump acknowledged using investment losses to avoid paying taxes, saying "of course I do." The Times reported he took so substantial a tax deduction on a declared $916 million loss in 1995 that he could legally have avoided paying any federal income taxes for up to 18 years.

MOVING FREELY AROUND THE STAGE

The debate's town hall-style format allowed the candidates to move freely around the stage and address the questioners directly.

Clinton frequently stood directly in front of the questioners to try to connect with voters. At times Trump stood almost over her shoulder, or wandered the stage, while she spoke. He paced, frowned and physically loomed over Clinton, prompting bewilderment and mockery from some on social media.

Clinton defended her remark at a fundraiser in September in which she said half of Trump's supporters were part of a "basket of deplorables."

"Within hours I said I was sorry about the way I talked about that, because my problem is not with his supporters, it's with him," Clinton said.

Asked at the end to name one thing each admired about the other, Clinton said she respected his children for their ability and devotion to Trump. In response, Trump called her a fighter and said he admired her for her refusal to give up.

As the moderators announced the end of the debate, the two candidates turned toward each other and shook hands.

Their next and last debate is on Oct. 19.

(Writing by Steve Holland and John Whitesides; Additional reporting by Amanda Becker, Amy Tennery, Michelle Conlin, James Oliphant and Susan Heavey; Editing by Howard Goller and Frances Kerry)