I second the cashmere cardigan!
I have a black V-neck for formal days and a camel one for informal.
They go under the vest and are not seen once your coat is on. They provide just the right amount of extra warmth without adding too much bulk. This is a great way to get some use out of a sweater that a moth has had it's way with...
I have also been known to wrangle it off once the weather warms up and tie it on to my breastplate or throw it in one of the follow vehicles with a friend if I start to overheat.