Freezing cold air in winter - damper door issue?

[Jorgy]

I have a VenMar 3500 air exchanger. I may or may not have a problem but at least I need some education. Last winter I had freezing cold air coming into the house. I have radiant floor heat with slab on grade with a backup furnace and my air exchanger. I was only using in floor heat and the air exchanger and had incredibly cold air coming through my hvac ductwork, along with my cold air returns. I'm compiling a laundry list of why I am getting an influx of cold air. Suspecting it might be my air exchanger, I put my cell phone inside it to see if the damper door was opening and closing properly. Troubles is, I think the one I am looking at is for defrost mode only. It is the front left port. It would close only when power was removed and no matter what the temperature was set to, it was wide open. Is that damper open all the time except for when it is cold, then it closes more or less depending on the temperature? Granted it was very cold outside and I'm in Minnesota. This seemed new though. I'm concerned that something isn't working properly with the air exchanger and the only other thing I can think of is insulation around the ductwork degrading over time. (The house is 25 yrs old). What else should I be looking at?