By Arno Schuetze, Kathrin Jones and Mariana Ionova

FRANKFURT/LONDON (Reuters) - Deutsche Bank is looking to cut its loan securitization business further starting with repackaged U.S. mortgages, two people familiar with the matter said, as the lender braces for a large fine in the United States for alleged mis-selling of such debt.

A final decision about this core business is set to come early next year, the people said, and securitization cutbacks could become a central part of an expected strategic overhaul at the bank, once U.S. authorities have settled on a penalty.

As well as rolling back the repackaging and resale of U.S. mortgages, European car loan securitization and other areas may also be cut, the people said, adding that management were still debating the scale of the reductions.

"We have already shrunk the business over the last two to three years," a person with direct knowledge of the bank's plans said. "It could shrink a lot more. Not only sales and trading, but also in origination."

Such a move would mark a retreat from a core business that helped Deutsche become one of the most dominant investment banks in the world before the financial crash. Now, the International Monetary Fund considers it a big risk to the financial system.

Deutsche Bank declined to comment.

Germany's flagship lender is among the top six investment banks globally in securitization, according to research group Coalition. The asset-backed Securities (ABS) market in the United States alone was worth almost $2 trillion in 2015.

But tougher regulation following the financial crash has made it more expensive for banks to trade such complex securities as they tie up more capital.

Compounding Deutsche's difficulties, the U.S. Department of Justice has demanded it pay up to $14 billion to settle claims it misled investors when selling mortgage-backed securities in the United States before the crisis.

FALLING RETURNS

Despite its insistence the final fine will be less, Deutsche Bank's stock price fell by almost a quarter in the two weeks that followed the U.S. demand, dragging down many of its rivals.

Deutsche Bank is one of the most high profile cases in the United States. It had a 6.4 percent share of the U.S. residential mortgage-backed securities market before the crash, according to rating agency Moody's.

U.S. investment bank Goldman Sachs, which had a slightly larger share of the U.S. mortgage-backed securities market than Deutsche, reached a settlement of roughly $5 billion over similar allegations.

In paring back its presence, Deutsche is responding not only to tighter regulation but also tougher market conditions. Banks are now sometimes required to hold more than a fifth of the value of such securities as capital to guard against losses, making it more expensive than many other forms of banking.

That prompted banks to shift from trading parcels of loans to helping clients create such securities as a way of raising finance. But competition in this sector is stiff and Deutsche Bank has struggled to get a foothold, market players said.

Although it has largely recovered in the United States, the European market for asset-backed securities - securities based on pooled loans such as mortgages, car loans or consumer credit - is only half the size it was before the crash in 2008.

While trading such securities has been a money spinner in the past, one of the people said Deutsche's track record was mixed: "We don't always have the right level of returns."

Paring back securitization would be in line Deutsche Bank Chief Executive John Cryan's pledged to get out of unprofitable businesses.

(Writing by Arno Schuetze and John O'Donnell; editing by David Clarke)