Brexit chaos could change the political map of Britain

The Liberal Democrat victory in the Richmond Park byelection could be just the first of many electoral aftershocks that could reconfigure the electoral landscape

Good news had been in short supply for Lib Dems over the previous 18 months, so when some hugely encouraging data arrived at their byelection headquarters in Richmond last Wednesday morning, the reaction was one of excitement coupled with scepticism. Party strategists had just received the results of internal polling showing they had pulled ahead of Zac Goldsmith, the former Tory MP turned independent, with 24 hours to go until voting began.

Goldsmith, a hardline pro-Brexiter who had quit the Conservatives over the decision to expand Heathrow airport, had won a 23,000 majority at last year's general election and was still a popular figure locally. But if this data was correct his reputation would count for nothing and a party that came close to being obliterated at last year's general election was on course for a byelection sensation to rival any in British political history.

The small Lib Dem team led by campaign manager James Lillis studied the figures and wondered how to respond. One of the first rules of campaigning is never to appear too confident. Releasing them might give the impression that the party thought it had victory in the bag. But this was not proving to be an ordinary byelection in any sense and did not necessarily require stock responses. It had already broken all the rules. Over previous days Lillis's team had amassed solid evidence that many thousands of voters were shunning traditional party loyalties and deciding their votes not according to their views on issues like Heathrow or the NHS - but in line with what they thought about the biggest issue of the day - Brexit.

The Lib Dems had run their entire campaign in this wealthy part of west London suburbia, in which 72% of people had voted Remain on June 23, on an anti hard-Brexit message. "For the first time I can remember we were not the pot-hole party. We were promoting our views on the EU, on internationalism, tolerance, what sort of country we want to be," said one party insider.

Tory Remainers had told them on doorsteps that they would vote Lib Dem because they disapproved of Theresa May's hard Brexit approach. Labour supporters were doing the same wanting to send an anti-hard Brexit message. The Greens and Women's Equality Party had not run candidates and instead had backed the Lib Dems, forging a fledgling progressive pro-EU alliance. A few Labour supporters had even campaigned for the Lib Dem candidate Sarah Olney.

Despite all the positive feedback, the Lib Dems were far from sure. The mood was too unusual, too volatile. A decision was reached that they needed to instil confidence that the shift was real and they could actually win, so, departing from all precedent, they put out the data. "We had never leaked polling of this kind before," said an insider. "But because we had to overcome a 23,000 majority, probably the biggest barrier was simply that people didn't believe we could do it. We needed to show that progressive voters were responding to our message that Britain should remain open, tolerant and united. It worked. In the last day Zac Goldsmith's team just seemed to crumble. On polling day we were out from 5am delivering 'good morning' leaflets and were still out at 10pm knocking up our supporters. We got our vote out."

In the early hours of Friday morning the grin on Olney's face was so broad that at times she struggled to get the words out in her victory speech. She had overturned Goldsmith's huge majority and beaten him by 1,872 votes. Brexit had changed everything. Labour, which had argued internally over to whether to field a candidate at all but eventually did, saw its man Christian Wolmar lose his deposit and receive fewer votes (1,515) than it had local members (1,600).

Later voters began to explain why they had switched from Goldsmith to the unknown politically inexperienced accountant Sarah Olney. "It was a shame, really. Zac was always really diligent," said Susannah, a young mother who did not not give surname, on the school run in Mortlake. "But the Brexit stuff was the most important issue. Lots of people here still can't really believe Brexit is happening. It was much harder to vote for Zac knowing he'd supported the people who got us here. It's made a lot of people like me change our votes."

Down the road in North Kingston, Andrée Frieze, who was the Greens' candidate at the 2015 general election, said his party's decision not to stand, and instead to back Olney, was a sign of how narrow party interests and loyalties were giving way to new alliances and a fight for what mattered most. "Brexit was absolutely the main issue on the doorstep here in Richmond," Frieze said. "Zac had been really popular, and if the Greens had run, this election would have felt the same as any other. But because we didn't and explained why, people felt they could vote Lib Dem again. This wasn't about the person, it wasn't about the party, it was about the bigger picture. People told me that voting Lib Dem this time round - when people like me weren't running - made them feel part of a team."

This weekend all the main parties are grappling with the implications of the Richmond Park result. While it is an unusual seat, home to an uncommonly high number of well-heeled, well-educated Remain voters, and the election was triggered by the special circumstances of Goldsmith's resignation, it has single-handedly revived the Lib Dems' morale and sense of purpose as the anti-hard Brexit party. The question now is whether Brexit can have a similar transformative effect elsewhere. Tim Farron, the party leader, wasted no time in declaring that his party was "back in the big time". The Lib Dems are now eyeing up other seats in the south, and installing candidates where they have strong local roots and are second to the Tories, in case of an early general election or more byelections. They claim Richmond will not be a one-off and cite their success in David Cameron's former seat of Witney, in Oxfordshire, where they leapfrogged Labour and Ukip into second place in October, as evidence that they are now the party of the 48%.

For Theresa May and the Tories, despite their assertion that Richmond changes nothing, it is a warning shot that Brexit is shifting the landscape, in profound, if very different, ways across the country. May's already wafer-thin Commons majority has been cut to just 14, and if more byelection losses were to follow, her ability to govern effectively would be called ever more into doubt. Today, a group of former Tory ministers and senior MPs warn that if the prime minister panders too much to anti-EU hardliners, the party will lose middle ground voters en masse as they did in Richmond, and risk defeat at the next general election.

The tactical counter to that is that a soft Brexit will deliver ammunition to Ukip, under its new leader Paul Nuttall, particularly in the Midlands and the north where they pose a threat to both the Tories and Labour. For Labour the dilemmas are just as acute. They were never expected to do well in Richmond, but their miserable tally of votes there has sounded loud alarm bells nonetheless. The fear among Labour MPs is that the party led by Jeremy Corbyn, a lukewarm supporter of the EU but a keen advocate of free movement and defender of immigration, risks being trampled between a newly resurgent Lib Dem party in pro-Remain seats in urban areas and the south, and Ukip in Leave strongholds. As one senior Labour MP put it: "Our leader seems to be anti-markets and lukewarm about the EU on the one hand, yet gives an unqualified pro-immigration message on the other. It is the worst of both worlds electorally." Yesterday, in a speech in Prague, Corbyn said it was vital that parties on the left did not respond to the surge of rightwing populism by scapegoating refugees and migrant workers, again refusing to be allied to those in his party who argue it is a problem that needs addressing.

Attention is now turning to another byelection this Thursday in the Tory-held Lincolnshire seat of Sleaford and North Hykeham, caused by the resignation of Stephen Phillips, a Brexit supporter who stood down because he thought parliament should be consulted more on the terms of departure. In this seat 62% of people voted for Leave. The electoral dynamics are therefore in many way the reverse of those in Richmond.

The Tories have another huge majority (more than 24,000) with Labour second at the 2015 election and Ukip narrowly behind in third. The Tories should hold on. But the fear in Labour circles is that they could be overtaken by Ukip. After he was voted in as leader last Monday Nuttall declared that his ambition was "to replace the Labour party and make Ukip the patriotic voice of working people." Thursday is his first test.

Last week, at the outermost edge of the constituency, a group of Labour activists made a lonely picket outside Grantham Hospital's A&E unit, which is now closed overnight. They were joined for half an hour by Jon Ashworth, the shadow health secretary, and the party's local candidate, binman Jim Clarke. Clarke says Labour should do well in this area but even the party faithful doubt his prospects. "Who knows if we can beat Ukip the way things are going," said David Taylor, a Labour member. "If people are voting Ukip, then we really need to look at what we're doing as a party."

Down the road in Sleaford, Jack Croll is campaigning for Ukip. "We'll beat Labour," he says. "That's easy. I don't like giving money away, for a start. Do you? [He means to the EU.] The other issue, of course, is all this immigration. Where I come from it's absolutely spinning with immigrants. That needs to stop."

Victoria Ayling, a well-spoken former barrister and county councillor standing for Ukip, said local residents across the heavily rural constituency had had enough of the two main parties, and that even Clarke's impeccable working-class credentials couldn't save Labour here.

For Theresa May, as for Labour, there are difficult if not impossible balances to be struck. Whichever line the Tories take on Brexit, part of their support base will be angered. Last week there were signs of May moving away from a hard Brexit - something that will please moderate Tories but alarm the right of her party. David Davis, the Brexit secretary, suggested as much when he said that the government might be prepared to pay into the EU budget in order to stay in the single market. Eurosceptics reacted with horror, saying there should be no backtracking and that voters would be appalled if money still went to Brussels to pay for a Brexit lite. Ukip piled in, saying the great betrayal was under way.

This week 11 judges will begin hearing the government's appeal in the supreme court over whether Parliament must approve the triggering of the Brexit process. If it is defeated the battle over Brexit will be fought out in parliament over the coming months. Already Brexit has altered the landscape as uncertainty prevails over the direction it will take.

Back in Richmond a Lib Dem worker celebrated on Friday saying that his party, and perhaps Ukip, were the only ones with reasons to be hopeful amid the Brexit chaos. "We are the only ones who offer any clarity. The Tories and Labour are divided over Brexit and we are united. It is a big chance for both parties."