The Co-operative Group has given up on its ambition to become a major challenger in high street banking by pulling out of talks to buy 632 branches from Lloyds Banking Group, dealing a blow to the government's ambitions to bolster competition among the big banks.

After more than a year of discussions, the biggest mutual in the country admitted on Wednesday that it could not proceed with a deal that would have created a 974-strong branch network – three times its current size – and a large increase in its share of current accounts to 7%.

The government had worked hard behind the scenes to try to facilitate a transaction with the Co-op that could have fostered competition on a high street still dominated by the "big four" of Lloyds, Royal Bank of Scotland, HSBC and Barclays. Richard Lloyd, executive director of the consumer body Which?, said it was a "setback" for the government's attempt to "tackle the unhealthy dominance of our biggest banks".