VIENNA (Reuters) - Austria's government moved on Tuesday to seize the house where Adolf Hitler was born to prevent it becoming a site of pilgrimage for neo-Nazis, and the country's Interior Minister said he wanted to tear it down.

Hitler's family lived in the house in Braunau on the Inn for only three years around his birth on April 20, 1889; but the fate of the three-storey building coated in pale yellow paint has long been the subject of controversy.

A spokesman for the interior ministry said the government had agreed a law to take ownership after the building's landlord, a local woman, had refused to sell it to the state. The bill would now go before parliament.

"The decision is necessary because the Republic would like to prevent this house from becoming a 'cult site' for neo-Nazis in any way, which it has been repeatedly in the past, when people gathered there to shout slogans," Interior Minister Wolfgang Sobotka told reporters before the cabinet meeting.

"It is my vision to tear down the house," he added. A commission consisting of 12 members from the fields of politics, administration, academia and civic society will ultimately decide the fate of the building.

A retired local woman owns the property, which Austria's interior ministry has been renting since 1972 and has sublet to Braunau. The ministry pays around 4,800 euros ($5,332) a month in rent.

The building has housed workshops for disabled people, but has been empty since 2011, because the owner repeatedly rejected ideas for the future use of the house and purchase offers from the state, according to the interior ministry spokesman.

Once the law has passed parliament, the owner has no right to appeal the decision or negotiate her compensation, which will be in line with the sum paid to home owners evicted in the course of railway line construction, he said.

Nazi Germany annexed Austria in 1938. Debate still smolders over whether Austrians were willing accomplices, many having cheered his return to his country of birth at the time, or the first victims of a dictatorship that ultimately reduced much of Europe to ruins and cost tens of millions of lives.

(Reporting By Shadia Nasralla; editing by Ralph Boulton)