have done this a few times, i do not recommend it. too many "What If's" involved.
and as was said here: why reinvent the wheel?
i would take the smallest bladder i could if it is drag i was worried about.
there have been many dives that i had not touched my inflator at all.
but i would rather have it there for when i need it.