The Tories can be their own worst enemy