Cold air always finds a way through the gap at the helmet, straight onto your neck and chest. That is where you cool down fastest, and with the cold your concentration goes.
A tube scarf is the cheapest fix and stays under the jacket where it belongs. If it gets colder, wear a thin balaclava under the helmet. Many helmets also have a chin curtain or wind deflector that you can simply fit before winter.
A few francs, and the ride feels like half as much cold.