Actually I don't believe you are supposed to swim away.

The Fish and Wildlife people produce a rules video called "Manatee Manners" that says touching them with one hand is ok, such as scratching them, which they like, bit no holding on, no impeding, no blocking, etc.