Starmer has now suffered the most ministerial resignations, outside government reshuffles, of any prime minister at the beginning of their tenure in almost 50 years - more even than Boris Johnson in his chaotic period in office.
"I deeply regret my decision to not seek additional specialist tax advice... I take full responsibility for this error," she said in her letter to Starmer.
"Given the findings, and the impact on my family, I have therefore decided to resign," said Rayner, who also stepped down as a minister and deputy leader of the Labour Party.
In a particularly emotional letter, Starmer said he believed she had made the right decision but understood it was one "which I know is very painful for you".
"On a personal note, I am very sad to be losing you from the government ... Even though you won't be part of the government, you will remain a major figure in our party."
The independent adviser on ministerial standards ruled Rayner had broken the code because she had failed to heed the warning within the legal advice she said she relied on to seek out expert advice on her complicated financial situation.
"It is with deep regret that I must advise you that in these circumstances, I consider the Code to have been breached," he said, referring to rules to make sure the conduct of politicians meet the standards of public service.
With Labour trailing Nigel Farage's populist Reform UK in the polls, Starmer faces difficult state spending and tax choices as he seeks to repair his party's image after accusations of hypocrisy by critics over accepting expensive items including clothing and concert tickets from donors.
Opening Reform's party conference in the central English city of Birmingham, Chairman David Bull announced Rayner's resignation to loud cheers and clapping at the party conference.
"The government is falling apart as we speak," he said. "They are on the run from us. One by one we are taking them down."
For Starmer, losing his deputy is particularly damaging, especially as Rayner - once a working-class teenage mother - had been able to mediate between Labour's left and centrist wings to keep the party united, and had a wider appeal than Starmer.
"Any resignation is a blow, especially Ange (Rayner), but she clearly had to go," said one Labour lawmaker, adding she would probably stay quiet for a while but could, at a later date, try to mount a challenge against Starmer.
Rayner had been forced to refer herself to the independent adviser on ministerial standards on Wednesday after admitting that she had made a mistake over the tax payment.
In an interview in which she appeared close to tears, she described setting up a trust for one of her sons, who has lifelong disabilities as a result of an injury.
It was to that trust that she sold her share of her family home in northern England to pay for an apartment in the southern English seaside resort of Hove, believing she would not have to pay the higher rate of tax charged when buying a second home.
After taking further legal advice, she then said she had made a mistake and was taking steps to pay the additional tax.
(Reporting by Andrew MacAskill and Elizabeth Piper; additional reporting by William James and Muvija M; Editing by Kate Holton, Alex Richardson, Philippa Fletcher)