He promised he would do all he could to work towards a new start for Germany.
As she left the chancellery in Berlin, ending a 31-year political career, Mrs Merkel told her former vice-chancellor to approach the task "with joy".
His centre-left Social Democrats will govern alongside the Greens and the business-friendly Free Democrats.
Mr Scholz, a soft-spoken 63-year-old, steered the Social Democrats to election victory in late September, positioning himself as the continuity candidate because he played a key role in the Merkel government as vice-chancellor.
The German parliament, the Bundestag, backed him as chancellor by 395 votes to 303, and he was then formally appointed as the ninth federal chancellor by President Frank-Walter Steinmeier.
After the vote in parliament, he was asked by Bundestag President Bärbel Bas whether he accepted the appointment and said "yes". He later took the oath of office but - unlike his predecessor - left out the religious reference "so help me God".
Since the election, Mr Scholz's party has worked with the Greens and the Free Democrats on a coalition deal, which was finally signed on Tuesday. All 16 ministers took the oath of office on Wednesday, becoming Germany's first cabinet to include as many women as men.
The new government has ambitious plans to fight climate change by phasing out coal early and focusing on renewable energy, but their initial priority will be tackling the coronavirus pandemic.
Health authorities have recorded another 69,601 cases in the past 24 hours and a further 527 deaths - the highest number since last winter.
Mr Scholz is already a known and trusted face in Berlin and Brussels. He marketed himself to voters as sort of Merkel Mark Two, despite hailing from a different political party.
But it's not all about continuity, and friends and trade partners of this rich and powerful nation will be watching closely.
Mr Scholz's coalition government is a never-before-seen marriage of convenience. What unites them, they claim, is a determination to modernise Germany, while preserving the country's treasured stability.
Immediate challenges for them are:
Covid - Germany is in the midst of a pernicious fourth wave and considering mandatory vaccination
The threat of a Russian invasion of Ukraine
Angela Merkel was accused of putting trade before politics. Team Scholz is expected to be somewhat tougher on Moscow and Beijing despite the potential economic hit to German business.
Relations with Washington may improve as a result, though, and that is a declared priority of Olaf Scholz's fledgling coalition.
Mr Scholz's first foreign trip as chancellor will be to Paris and Brussels on Friday. He and Greens joint leader Annalena Baerbock, who becomes foreign minister, will have to respond to fears surrounding Russia's military build-up near the border with Ukraine.
Although Russia has denied plans to invade its neighbour, Angela Merkel agreed with US President Joe Biden and the leaders of the UK, France and Italy late on Tuesday that they would adopt a joint strategy to respond by imposing "significant and severe harm on the Russian economy".
One obvious economic measure would be to threaten Russia's Nord Stream 2 gas pipeline to Germany, which has been completed but is still awaiting approval from the German energy regulator.
US officials say they have reached an understanding with Germany that the pipeline would be shut down, which would be a significant intervention.
In a message of congratulations to Mr Scholz, Russian President Vladimir Putin said he looked forward to constructive dialogue, and hoped that Germany continued to recognise "there's no alternative to dialogue".
Leaving the chancellery on Wednesday afternoon, Mrs Merkel was applauded by staff as she told her successor to take over and work for the good of Germany. In turn, he spoke of the crises that had brought them together and of the "deeply trusting collaboration" they had developed.
First elected in December 1990, Angela Merkel was immediately given a ministerial job by then Chancellor Helmut Kohl.
But she then helped oust him and became Christian Democrat leader in 2000, before becoming chancellor herself in November 2005.
She will still have an office close to the Bundestag, in a flat that was once used by Margot Honecker, once dubbed the most powerful woman in communist East Germany.