Yep pretty much as you described. You'd have to experiment with compressed vs. uncompressed audio to see which is faster. Obviously compression adds latency, but also results in less packets which are quicker to send.