SURFER'S EAR (exostosis) is found among salt water enthusiasts, usually males aged 20-39. Possibly due to cold water, the bony portion of the external canal becomes thickened and eventually creates a knob-like projection. Over time, several of these growths may close the ear canal causing significant hearing loss. The condition is painless and may go unnoticed for a number of years until hearing loss becomes obvious. Surgical correction by a specialist may be necessary to restore normal hearing.
There seems to be more study needed, because it maybe in our genes, and it maybe because of the rush of blood to the area due to the cold. If you wear a hood it traps water and your body warms it so that the ear does not flush with blood to warm its self. So wear a hood and you shouldn't have a problem!