Think you will find that recent by election results tells a very different story & we will see what happens with the next couple of Tory by elections, it is the sheer stupidity of this Tory government that is making them so unpopular with so many voters not just the farming & associated community, farmings influence stretches much further than we assume.No, and the wider the 'community' the less chance of coherence. Consider how many are in Scottish, Welsh and NI constituencies where there wouldn't be a hope - or possibility - of a Tory MP anyway; a quick calculation makes that about 30% of them. The remaining 70k, even if just across the approx 300 'rural' constituencies, work out at about 230 votes each.
Looking at the most marginal constituencies in the UK as a whole, only the top 18 would be affected by this number (assuming a straight switch to nearest rival), only six are 'rural', only two of those are in England, and only one is currently in Tory hands (High Peak).
So, a hung parliament aside, what you you suggest doesn't seem very plausible.
It doesn't always take a switch of vote to make a change, often a collapse of the Tory vote has a very dramatic effect.