A blunder by goalkeeper and captain Itumeleng Khune after five minutes cost South Africa a goal in Nouakchott and set the home team on the road to their greatest Cup of Nations triumph.
Khune allowed a long-range Ali Abeid free-kick to slip from his grasp into the net and Bafana Bafana (The Boys) had debutant Siyabonga Zulu red-carded after half-time for a last-defender foul.
South Africa equalised midway through the second half when Thamsanqa Gabuza soared above goalkeeper Mohamed Salahdine to nod in a cross.
But a Mauritanian team ranked 42 places below their opponents in the world finished strongly and superb individual goals by Boubacar Beyguili and Ahmed Khalil sealed victory.
Success lifted Mauritania from the bottom to the top of Group M on goals scored although Cameroon will regain first place if they do not lose in Gambia Sunday.
Four-time African champions Ghana left it late to succeed in Kigali with Russia-based Mubarak Wakaso the 87th-minute match-winner.
The Black Stars were awarded a free-kick just outside the box on a poor pitch and Wakaso curled the ball past the wall and wide of goalkeeper Olivier Kwizeri into the corner.
It was a cruel blow for the Wasps, who fought gallantly before a near-capacity crowd and did not deserve to lose.
The last time Ghana visited Kigali for a Cup of Nations qualifier they suffered a stunning 1-0 loss 12 years ago.
Ghana, runners-up to Ivory Coast in the 2015 Cup of Nations, lead Group H with six points, three more than Rwanda, while Mauritius and Mozambique are pointless ahead of their Sunday clash.
There was no dream debut for new Nigeria coach and former midfield star Sunday Oliseh as the three-time Cup of Nations title-holders found Tanzania a revitalised side.
– Ikeme impresses –
Super Eagles goalkeeper Carl Ikeme from English second-tier club Wolves made several excellent saves in Dar es Salaam on his debut at 29.
Ikeme started after first choice shot-stopper and captain Vincent Enyeama pulled out in midweek following the death of his mother.
The result was another feather in the cap for caretaker coach Charles Boniface Mkwasa, who ended a string of Taifa Stars losses with an African Nations Championship draw in Uganda.
A point took Nigeria to the top of Group G, but record seven-time African champions Egypt will overtake them if they dodge defeat in Chad Sunday.
English Premier League pair, Cheikhou Kouyate of West Ham and Sadio Mane of Southampton, netted as Senegal triumphed 2-0 in Namibia
Kouyate struck in the first half and Mane in the second in Windhoek as the Teranga Lions maintained a perfect Group K start with a second win.
Burundi are second, three points adrift, after a 2-0 victory over Niger in Bujumbura thanks to goals from South Africa-based duo Fiston Abdul Razak and Papy Faty.
Uganda President Yoweri Museveni funded the $210,000 (190,000 euros) travel costs of the Cranes to the Comoros islands off the south-east coast.
The cash-strapped national team triumphed 1-0 in Group D courtesy of a goal midway through the first half from recalled Iceland-based Tonny Mawejje.