I keep a spare jacket at the office too. The gear jacket is cumbersome when I go out for lunch and people stare at it and ask questions. Not that I mind that.
My gear usually sits on one of my conference table chairs during the day. This time of year I have all the liners in my jacket. Just the windbreaker liner in my pants. My pants are one size larger than my jacket because I wear them over my office clothes. I rarely feel it necessary to wear the heavy liner in my pants. My legs never seem to get that cold. If my commute time was longer I might reconsider.
I often wear an LL Bean rain jacket over my gear just because it is such an excellent windbreaker. Plus, the jacket is red, my bike is blue, hence I ride in Jayhawk colors.