However, to pass any measures he will have to rely on the support of two Catalan pro-independence parties, the anti-establishment Podemos and a Basque nationalist group - all who have differing views.
The no-confidence vote was tabled by the opposition Socialist Party whose leader Pedro Sánchez needs a majority of 176 votes to become leader.
The Madrid stock exchange was up almost 1.6 percent after Sanchez won the vote, earning a standing ovation from his party's lawmakers.
Mr Sanchez, 46, is expected to take office by Monday after King Felipe VI swears him in, and appoint his cabinet next week.
The end of Rajoy's more than six-year reign as Spanish prime minister was the first ouster of a serving leader by parliament in four decades of democracy and brought a rare success for a centre-left party in Europe in recent times.
Rajoy became Spain's first sitting prime minister to give evidence at trial when he was called as a witness previous year.
Trump moves ahead with Chinese trade tariffs, threatening to reignite tensions
The final list of Chinese products will be announced by June 15 and the tariffs will go into effect shortly afterwards. Commerce Secretary Wilbur Ross is scheduled to meet with officials in Beijing on June 2-4 to continue negotiations.
The incoming prime minister has outlined that his priorities will be social issues - such as measures to help young people and the elderly - before calling elections, though he hasn't said when there might be a vote.
Leftist Podemos, which will offer parliamentary support to Mr Sanchez's government, is also unlikely to gain big influence over the new Prime Minister, who is keen to differentiate his Socialist party from its anti-austerity ally and win back centrist voters.
Aitor Esteban, a Basque party MP, warned Mr Sanchez: "Your government will be very complicated, weak and hard".
During Thursday's pre-vote debate, Rajoy said the corruption case "does not concern members of the government" and repeated the party's argument that only a tiny number of its politicians have been tainted by corruption.
He put Spain back onto the path of growth after a devastating economic crisis although unemployment remains sky-high, jobs precarious and many complain inequalities have risen.
Sánchez, whose nickname locally is "Mr".
His term in office was also marked by a series of corruption scandals involving former members of his Popular Party (PP). The National Court in Madrid last week handed down prison sentences to the tune of a combined 351 years to 29 people in connection with a slush fund set up in the 1990s and early 2000s to illegally finance PP campaigns.