I almost always have at least one lift bag with me when diving, and a Jersey Reel, so of course that would get me to the surface. I always have ditchable weight as well if necessary.
About being able to swim a balanced rig to the surface, you hit the nail on the head. Really if your buoyancy is spot on, then you should be able to move about without much if any air in your BC or Drysuit.
I did notice a few of the old timers I have gone diving with did not use any BC. Well one guy had a wing, but it had no inflator and many tears in it. He said the only reason he had it on was to allow him to say he had a BC so he could get on the boat. LOL
Anyway, I bought a wing so it really does not matter any more.