For devices behind NAT that don't use STUN or SIP ALGs, FS waits to receive 10 packets (switch_rtp.c line 3232) before sending RTP to the address used by the device instead of the one advertised in SDP.

The bad thing in case of PCMA@20ms, is that it means that the first 200ms (at least) of audio are not received by the device. In practice, this delay is between 300ms and 1 full second.

Is there an issue with lowering this "adjust" to just 1 packet instead of 10 in FS?

Out of curiosity, I checked how it's done in Asterisk (res_rtp_asterisk.c). Apparently, anytime Asterisk receives a single RTP packet from a different address (and not just once as in FS), it starts sending to that address. Which is quite cool because you could hijack audio of any session by just sending dummy RTP packets to that port a bit faster than the real source.

As for FS, it has an adjust window, and does the IP change only once, so it would be way harder to do the same, even with an adjust of a single packet.