Rich Rodecker added a comment - 01/Dec/11 7:44 PM rtmp profile, config, and FS multi-home details.

Rich Rodecker added a comment - 01/Dec/11 7:45 PM Flex application source.

Brian West added a comment - 02/Dec/11 1:48 PM Does this behave the same if you use our included example client? /b

Rich Rodecker added a comment - 02/Dec/11 3:03 PM Yes, same issue.

Anthony Minessale II added a comment - 02/Dec/11 4:02 PM what about when you connect to http://conference.freeswitch.org ?

Travis May added a comment - 02/Dec/11 4:19 PM The best destination to test to has continuous music: 4151595@services.sip.is (press 2 after connect) By testing this from our SWF and BLINK using speex only, you will see this is not a transcoding issue.        BLINK will have perfect audio        SWF will have choppy audio Both of the following will go through our FS server and be transcoded from Speex to g711uLaw FROM BLINK: 4151595@216.52.221.166        will forward to 4151595@services.sip.is FROM TFF SWF: 4151595@services.sip.is         http://216.52.221.154/freeswitchlatest/        replace ringTo with 4151595@services.sip.is Travis May

Rich Rodecker added a comment - 02/Dec/11 4:41 PM It seems I get similar choppiness just by going to http://conference.freeswitch.org in two different browser windows, calling in to the conference, and reloading one of the browser windows and calling back in again. It's not breaking up to the extent as we're experiencing, maybe because of the duration of the audio?

Travis May added a comment - 02/Dec/11 6:01 PM In our test to 4151595@services.sip.is , the choppiness varies. Sometimes it is so bad it is unusabe, other times the chop is still present and detectable, but less severe. This is why I think tests through our FS to 4151595@services.sip.is demonstrates the issue very well.

Travis May added a comment - 06/Dec/11 1:17 AM We have tried the following, but no better: 1. Increased the buffer length <param name="buffer-len" value="100" /> ( Current ) <param name="buffer-len" value="50" /> ( Old ) 2. Changes to rtmp.c if (rsession->tech_pvt->maxlen && switch_buffer_inuse(rsession->tech_pvt->readbuf) > rsession->tech_pvt->maxlen * 3) [ Old ] if (rsession->tech_pvt->maxlen && switch_buffer_inuse(rsession->tech_pvt->readbuf) > rsession->tech_pvt->maxlen * 20) [ Current ] fprintf(rsession->io_debug_in, "[chunk_stream=%d type=0x%x ts=%d stream_id=0x%x] FLUSH BUFFER [exceeded %u]\n", rsession->amfnumber, state->type, (int)state->ts, state->stream_id, rsession->tech_pvt->maxlen * 3); [Old] fprintf(rsession->io_debug_in, "[chunk_stream=%d type=0x%x ts=%d stream_id=0x%x] FLUSH BUFFER [exceeded %u]\n", rsession->amfnumber, state->type, (int)state->ts, state->stream_id, rsession->tech_pvt->maxlen * 20); [Current]

Travis May added a comment - 06/Dec/11 1:59 AM *** BOUNTY = $5,000 --> 2% PACKET LOSS MUST BE UNNOTICEABLE *** **** EASILY REPLICATE ON YOUR OWN SYSTEM *** **** DISREGARD ALL OF THE ABOVE ABOUT OUR CODE AND SERVERS **** This issue can be replicated on the original Flex client provided by FS and any FS server. Issue appears to be caused by very minor packet loss <1%. In my 10 years of VoIP experience up to and beyond 2% packet loss is normally handled well by PLC (Packet Loss Concealment) mechanisms within the codec. However, since this is running over TCP, the retransmissions and DUP ACKs seem to be overwhelming the conversation. Steps to replicate: 1. Use default Flex Client (we have placed one at http://216.52.221.154/freeswitchorig/) 2. Send calls to any FS with mod_rtmp enabled. a. rtmp_url: ' rtmp://216.52.221.166/default&#39; 3. Call sip: 4151595@services.sip.is 4. Enable Wireshark or other packet capture on client or server 5. Watch for NetStream.Buffer.Empty a. See wireshark for DUP ACK, TCP Retransmission, and Malformed Packet b. Cause some packet loss, in windows "ping -t -l 65000 216.52.221.166" Is sip: 4151595@services.sip.is causing the problem ? No, the audio is perfect when called from other client, using Speex or any other codec between client and FS. PROVE: Call sip: 4151595@services.sip.is directly using Blink or any other SIP client, you will not hear the choppy audio regardless of codec used. Is the outbound leg causing the problem? No, this problem is witnessed even when playing looped audio file directly from FS. PROVE: Put any audio into infinite loop and call it from the default flex client (Songbird8kHz64kbpsMono.mp3 attached) Call “sip:songbird” from http://216.52.221.154/freeswitchorig/ You will still see the Buffer.Empty and hear the choppy audio after some time. Is Speex codec the problem? No, the exact same behavior is witnessed when using ULAW. http://216.52.221.154/freeswitchULAW/ Call: sip:songbird Is packet loss causing the problem ? Probably, but it should be more tolerant. In other VoIP systems/phones/clients, packet loss in VoIP of up to 2% is not noticeable in g711uLaw, g729, g726, or other major codecs. Our tests have a fraction of 1 percent packet loss. It is possible that minor packet loss is causing significant retransmissions and/or processor overhead on the server side. Therefore exacerbating the problem. Also, when using mod_rtmp, we sometimes notice an overall slowdown of the FS server as witnessed by a slowdown in log/debuggging response with only 1-2 rtmp calls open. This is very difficult to reproduce but only happens when mod_rtmp calls are in progress. This same server can do hundreds of non-rtmp calls and never displays such behavior.                 

Travis May added a comment - 06/Dec/11 3:24 AM Note that we receive those NetStream.Buffer.Empty events because the incoming stream has a 0.2 second buffer applied to it in the original SWF app. If we set the buffer to 0, we get the same issues without any NETSTREAM.BUFFER.EMPTY events.

Matt Stockton added a comment - 07/Dec/11 4:01 PM I've been playing around with this update, and I'm trying to understand what the implications of the multiplier on rsession->tech_pvt->maxlen is. When I use the multipler in the git head commit ( * 5), I have a bunch of buffering issues from MacOSx flash player, as evidenced by a lot of these logs in fs_cli: 2011-12-07 15:35:05.246833 [DEBUG] rtmp.c:896 rtmp/default/55555 buffer > 265 for 10 consecutive packets... Flushing buffer When I change the multiplier from 5 to something like 40: if (rsession->tech_pvt->maxlen && switch_buffer_inuse(rsession->tech_pvt->readbuf) > rsession->tech_pvt->maxlen * 40) I don't seem to have any problems at all. Audio being sent from flash player to FS is fine. However, I'm failing to understand the full implications of changing this multiplier. What are the side-effects of this?

Git added a comment - 07/Dec/11 4:09 PM Repository: freeswitch Branch: refs/heads/master Commit: edcb132 http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=edcb132 Updated By: anthm@freeswitch.org Comment: FS-3735 that is a resonable figure, if that works we can set it to this. can we close out this bounty then? FreeSWITCH Support Contracts and Consulting Services available! Contact us: Email: consulting@freeswitch.org Web: http://www.freeswitch.org Phone: +1-918-420-9266 Tollfree: +1-877-742-2583 Fax: +1-918-420-9267 iNum: +883 5100 1420 9266 Come To ClueCon in August to learn more about FreeSWITCH and Internet Telephony! http://www.cluecon.com

Matt Stockton added a comment - 08/Dec/11 1:46 PM I still see these fairly frequently on my Mac flash player:  rtmp.c:896 rtmp/default/55555 buffer > 2120 for 10 consecutive packets... Flushing buffer Must be something weird about the Mac flash player that's causing issues. Doing some research online, haven't found much but came across this: http://code.google.com/p/red5phone/issues/detail?id=45 Setting the outgoing buffer time didn't seem to help me though....will continue to investigate

Anthony Minessale II added a comment - 08/Dec/11 2:17 PM Its latency from the fact that its a TCP connection and possibly poor timing in the flash. We could opt to not flush the buffers but who knows how far behind you may fall. You could comment out the line that does the switch_buffer_zero and replace it with a debug line to see the size of the buffer on each read and see how bad it gets. On a side note, I have a mac too and i am not getting that problem. I look forward to some day having an alternative to this using webrtc.

Travis May added a comment - 08/Dec/11 10:13 PM I am out of the office this week. The initial feedback from my developers is that "there is improvement, but it's still not up to expectations". The bounty requirement was "unnoticeable" and even my initial test had a number of audio drops. For $5,000 I expect a fundamental fix, not a patch that sort-of works. Our intention is to use this worldwide and as the latency increases this problem will be worse.

Travis May added a comment - 08/Dec/11 10:19 PM I hired a well-known network analyst to provide feedback on this issue, here is that feedback. #2 below is a great question, does RTMP require an ACK for every single packet? We all know VoIP should not go over TCP, so we should concentrate on streamlining the TCP/RTMP connection. Hi Travis, I took traces to both sites from both a Mac/Safari and a Windows7/IE host. You are right – super choppy audio. There is no way I could understand it. I can hear the pause between packets and my screen filled with Buffer.Empty then Buffer.Full then Buffer.Empty messages. They were populating with the same frequency as the incoming packets in Wireshark. I don’t think packet loss is your issue. I had minimal loss in the http conversations which I could not detect as the user. For the RTMP streams on port 1935, I had zero packet loss. When I focused on the RTMP streams, I noticed that there was never more than either 47 or 94 bytes of audio sent at a time, yet the Set Chunk Size is 512 in packet 26 of “IE to freeswitchorig – very choppy audio.pcap”. In the Safari traces, the chunk size is set to 1024. I don’t know enough about RTMP to know why the server is only sending a single packet and then waiting for a TCP ack, but it is. It is not streaming. When you look at the delta times between each of the sent RTMP packets from the server, they correlate directly to the TCP handshake time i.e. the round trip time. So your two issues are: 1. small packet size (this may be due to me completely misunderstanding what Chunk Size is, I think it is to negotiate how much data in each packet, but when I read http://p2p-sip.blogspot.com/2009/06/problems-in-rtmp.html it goes down a completely different path). There is no reason in TCP to be doing this, both sides negotiate an MSS of 1460. 2. Server not queuing multiple packets to send to the client. The pattern should be audio/audio/audio/audio….audio/ack not audio/ack/audio/ack. There is no reason in TCP to be doing this, both sides have Window Scaling enabled and have window sizes in excess of 64k each. I would recommend having the developers review the trace files to help them isolate the issue. Make sure they are using the latest version of Wireshark 1.6.4 to ensure they have dissectors for RTMP. (If they have a really old version, the packets might not decode)

Travis May added a comment - 08/Dec/11 10:24 PM I agree on WebRTC. Unfortunately, WebRTC will not have 90%+ penetration for 24-36 months at the earliest and FreeSwitch is on the upswing now. I believe mod_rtmp is CRUCIAL to the future success of FreeSwitch. Travis

Anthony Minessale II added a comment - 09/Dec/11 3:32 PM That comment is actually very insulting. I tried to shake it off but after nearly 16 hours it's still annoying me so I feel the need to comment: So basically you are saying that mod_rtmp is CRUCIAL to the future success of FreeSWITCH? So it is utterly doomed without RTMP? Since lacking a CRUCIAL component would just cause the whole project to fade into the abyss? It's success has nothing to do with the countless hours we pour into making sure its up to date and running and all the other work we do? None of the providers who use it care about RTMP nor anyone using a PBX, it's cool to have RTMP and maybe a niche market might even be out there to make a few web based things with it..... However, I disagree with your statement since I believe that once flash is dead on mobile devices it won't be long before its all the way dead. http://www.guardian.co.uk/technology/2011/nov/09/adobe-flash-mobile-dead Plus its FreeSWITCH not FreeSwitch...... Anyway, That does not mean your findings are not correct on the RTMP module so I will have a look at the points made by your analyst.

Anthony Minessale II added a comment - 09/Dec/11 7:15 PM Has this been tested by the analyst and the others before or after my most recent commit? I am currently sitting here with both my mac pro desktop pc and samsung i5 win7 laptop calling into our conference server running latest HEAD I am calling the milliwatt tone extension and both are emitting a solid tone. I was doing a ping -t to the conference server from the windows box at the same time. one time in like 10 minutes the windows pc had an audio blip and the pings also failed during that time. The mac on the same network never skipped a beat. This is with the default values in rtmp.conf.xml besides the auth disabled. Please try the box at http://conference.freeswitch.org/flex its already set by default to call 9997 9997 - milliwatt 9996 - echo 1235 - imperial march 888 - public conference

Marc Olivier Chouinard added a comment - 13/Dec/11 11:01 PM Flash does support other protocol more adapt for voip communication (udp), but it require someone to take the time to implement it. In the mean time, with tcp packet lost are retransmit and any more up to date data will be put on hold, so there is a limit on how good it possible to be made, also there might have limitation into the flash client code itself we can't fix. I do not beleive FS depend on flash client at all... But it does make a really neat addition.

Travis May added a comment - 13/Dec/11 11:30 PM Hi Marc Olivier, Can you expand on the UDP protocol that flash supports? I may be willing to hire someone to implement that. Do you know anyone willing to do this on a contract basis? Thanks, Travis

Anthony Minessale II added a comment - 14/Dec/11 10:27 AM consulting@freeswitch.org is the contact address for contract work. That and other contact info appears in several of the notes on this bug already. I was expecting a response about my last posting with the demo server and questions etc.

Marc Olivier Chouinard added a comment - 14/Dec/11 12:58 PM Protocol is : RTMFP It also allow P2P communication between 2 flash player

Ken Rice added a comment - 27/Mar/12 5:17 PM please update this bug or it will be closed due to inactivity

Jeff Lenk added a comment - 30/May/12 6:28 PM Closed due to inactivity

Auto Admin added a comment - 10/Sep/12 8:17 PM Due to a long period of inactivity (13 or more days), this issue is due to be automatically close within 24 hours. If this issue is not actually resolved, please reopen it and add appropriate comments to help developers fix the issue. Thanks,   Jira Admin

Auto Admin added a comment - 11/Sep/12 8:17 PM Since there has been no change to the status of this issue, it is being closed for inactivity Thanks,   Jira Admin FreeSWITCH Support Contracts and Consulting Services available! Contact us: Email: consulting@freeswitch.org Web: http://www.freeswitch.org Phone: +1-918-420-9266 Tollfree: +1-877-742-2583 Fax: +1-918-420-9267 iNum: +883 5100 1420 9266 Come To ClueCon in August to learn more about FreeSWITCH and Internet Telephony! http://www.cluecon.com