Anthony Minessale II added a comment - 30/Jun/11 11:41 AM Today is June 30th so how do you report an issue against the June 15th revision and answers yes to the reproduced with GIT HEAD? If you feel it is related to other issues already open, why open more to put more strain on our bug marshal who are volunteers? Also did you test calling to an extension that does something else like play a file to prove it was conference related. Have you tried to reproduce a slower version of your test under valgrind? valgrind --tool=memcheck --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/local/freeswitch/bifreeswitch -vg I have to say with your persistance and large volume of calls you might want to consider commercial support at consulting@freeswitch.org

Anthony Minessale II added a comment - 30/Jun/11 11:55 AM also have you ruled out other components of your test such as not using the ESL etc. There seem to be a lot of missing paramaters.

Min Xie added a comment - 30/Jun/11 2:24 PM Anthony, Thanks for the reply. 1. This is the current testbed version information: FreeSWITCH Version 1.0.head (git-84f8868 2011-06-30 11-59-58 -0500) 2. To simplify the factors might be involved (such as event socket module), We used the following dialplan in public.xml to do the same test. Conference announcement is turned off and in-conference control is turned off. So there is no playback. Just the essential call+conference handling modules is involved.     <extension name="loadtest-2">       <condition field="destination_number" expression="^(557[0-9])$">         <action application="answer"/>         <action application="conference" data="${uuid}@default"/>       </condition>     </extension> 3. To possibly avoid impact from other event socket application, we stopped the script to display vmstat, calls and conferences every 5 seconds. Only couple of stats dips were performed. 4. And here is the result again, we sample the data at 0, 23K, 50K, 100K and 200K calls. Please notice that the concurrent call peak is 320 calls at 104th second since the traffic start. The VM size keeps going up: 0.441G -> 1.035G -> 1.593G -> 2.189G -> 3.364G. 5. The test itself is really easy to rerun. Regarding valgrind, we did it too but it didn't report any information for us. But we don't think it means there is no leak. If the heap allocations are still legally referenced by FS internal data structure. Valgrind won't report it. 6. Please advise if there are other miss parameters we could avoid in the test. Yes, we are prototyping FS as the telephony service for large scale service platform. But we do need to make sure it can run cleanly. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Command for getting VMPEAK: cat /proc/$(pidof freeswitch)/status | grep VmPeak Thu Jun 30 13:13:52 CDT 2011 VmPeak: 441692 kB ~~~ 0 session(s) since startup 0 session(s) 0/1000 5000 session(s) max Conferences:       0 0 0 Thu Jun 30 13:20:33 CDT 2011 VmPeak: 1035488 kB ~~~ 23554 session(s) since startup 303 session(s) 2/1000 5000 session(s) max Conferences:     301 1806 21070 ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host   60.0(0 ms)/1.000s 5060 434.77 s 22970 192.168.41.219:5060(UDP)   60 new calls during 1.002 s period 0 ms scheduler resolution   300 calls (limit 900) Peak was 320 calls, after 104 s   0 Running, 2282 Paused, 188 Woken up   0 dead call msg (discarded) 0 out-of-call msg (discarded)   3 open sockets   0 Total echo RTP pckts 1st stream 0.000 last period RTP rate (kB/s)   0 Total echo RTP pckts 2nd stream 0.000 last period RTP rate (kB/s)                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 22970 0 0          100 <---------- 22970 0 0 0          180 <---------- 0 0 0 0          200 <---------- E-RTD1 22959 0 0 0          ACK ----------> 22959 0        Pause [ 5000ms] 22959 0          BYE ----------> 22670 0 0          200 <---------- 22670 0 0 0 ------ [+|-|*|/]: Adjust rate ---- [q]: Soft exit ---- [p]: Pause traffic ----- Thu Jun 30 13:28:09 CDT 2011 VmPeak: 1593504 kB ~~~ 50938 session(s) since startup 308 session(s) 3/1000 5000 session(s) max Conferences:     302 1812 21142 ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host   60.0(0 ms)/1.000s 5060 895.57 s 50619 192.168.41.219:5060(UDP)   60 new calls during 1.002 s period 0 ms scheduler resolution   301 calls (limit 900) Peak was 320 calls, after 104 s   0 Running, 2284 Paused, 183 Woken up   0 dead call msg (discarded) 0 out-of-call msg (discarded)   3 open sockets   0 Total echo RTP pckts 1st stream 0.000 last period RTP rate (kB/s)   0 Total echo RTP pckts 2nd stream 0.000 last period RTP rate (kB/s)                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 50619 0 0          100 <---------- 50619 0 0 0          180 <---------- 0 0 0 0          200 <---------- E-RTD1 50618 0 0 0          ACK ----------> 50618 0        Pause [ 5000ms] 50618 0          BYE ----------> 50318 0 0          200 <---------- 50318 0 0 0 ------ [+|-|*|/]: Adjust rate ---- [q]: Soft exit ---- [p]: Pause traffic ----- Thu Jun 30 13:42:03 CDT 2011 VmPeak: 2188668 kB ~~~ 100988 session(s) since startup 303 session(s) 4/1000 5000 session(s) max Conferences:     303 1818 21211 ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host   60.0(0 ms)/1.000s 5060 1729.03 s 100626 192.168.41.219:5060(UDP)   60 new calls during 1.001 s period 1 ms scheduler resolution   300 calls (limit 900) Peak was 320 calls, after 104 s   0 Running, 2283 Paused, 183 Woken up   0 dead call msg (discarded) 0 out-of-call msg (discarded)   3 open sockets   0 Total echo RTP pckts 1st stream 0.000 last period RTP rate (kB/s)   0 Total echo RTP pckts 2nd stream 0.000 last period RTP rate (kB/s)                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 100626 0 0          100 <---------- 100626 0 0 0          180 <---------- 0 0 0 0          200 <---------- E-RTD1 100626 0 0 0          ACK ----------> 100626 0        Pause [ 5000ms] 100626 0          BYE ----------> 100326 0 0          200 <---------- 100326 0 0 0 ------ [+|-|*|/]: Adjust rate ---- [q]: Soft exit ---- [p]: Pause traffic ----- Thu Jun 30 14:09:50 CDT 2011 VmPeak: 3364328 kB ~~~ 200980 session(s) since startup 301 session(s) 3/1000 5000 session(s) max Conferences:     302 1812 21141 ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host   60.0(0 ms)/1.000s 5060 3394.90 s 200579 192.168.41.219:5060(UDP)   61 new calls during 1.001 s period 0 ms scheduler resolution   301 calls (limit 900) Peak was 320 calls, after 104 s   0 Running, 2283 Paused, 184 Woken up   0 dead call msg (discarded) 0 out-of-call msg (discarded)   3 open sockets   0 Total echo RTP pckts 1st stream 0.000 last period RTP rate (kB/s)   0 Total echo RTP pckts 2nd stream 0.000 last period RTP rate (kB/s)                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 200579 0 0          100 <---------- 200579 0 0 0          180 <---------- 0 0 0 0          200 <---------- E-RTD1 200578 0 0 0          ACK ----------> 200578 0        Pause [ 5000ms] 200578 0          BYE ----------> 200278 0 0          200 <---------- 200278 0 0 0 ------ [+|-|*|/]: Adjust rate ---- [q]: Soft exit ---- [p]: Pause traffic -----

Min Xie added a comment - 30/Jun/11 2:25 PM Eehh, sorry for the duplicate comments. How to remove one?

Anthony Minessale II added a comment - 30/Jun/11 3:16 PM and you also changed that extension to just playback a wav instead of calling a conference and compared the results?

Min Xie added a comment - 30/Jun/11 3:27 PM We have performed the following tests for this memory issue: 1. answer call#1 -> playback to call#1 -> conference outdial call#2 -> join call#1 to conference - memory leak 2. answer call#1 -> conference outdial call#2 -> join call#1 to conference - memory leak 3. answer call#1 -> playback to call#1 -> BRIDGE call to SIPP UAS - NO memory leak 4. answer call#1 -> join call#1 to conference - memory leak. From the test, we think it's the conference (create/destroy, performed by FS internally) caused the memory leak.

Anthony Minessale II added a comment - 30/Jun/11 3:43 PM What parameters do you have set in your conf config. is it the default conference.conf.xml? The conference uses a single memory pool which is destroyed as soon as the thread exits. Does your console log on debug show the write lock on write lock off message that should indicate the pool was destroyed?

Min Xie added a comment - 30/Jun/11 3:56 PM We have minimized the features in the conference default profile. Here is what we have: [ root@freeswitch219 bin]# cat !$ cat ../conf/autoload_configs/conference.conf.xml <configuration name="conference.conf" description="Audio Conference">   <advertise>     <room name="3001@$${domain}" status="FreeSWITCH"/>   </advertise>   <caller-controls>     <group name="default">     </group>   </caller-controls>   <profiles>     <profile name="default">     </profile>   </profiles> </configuration> I just ran 2 calls and here are the DEBUG log lines related to "conference". 2011-06-30 15:52:15.530434 [NOTICE] switch_loadable_module.c:252 Adding Application 'conference' 2011-06-30 15:52:15.530468 [NOTICE] switch_loadable_module.c:252 Adding Application 'conference_set_auto_outcall' 2011-06-30 15:52:15.530506 [NOTICE] switch_loadable_module.c:274 Adding API Function 'conference' Dialplan: sofia/internal/sipp@192.168.41.206:5060 Action conference(${uuid}@default) EXECUTE sofia/internal/sipp@192.168.41.206:5060 conference( 95dc9658-ce65-492b-b269-469df5ab482f@default ) 2011-06-30 15:52:38.526108 [INFO] mod_conference.c:6640 using channel sound prefix: /usr/local/freeswitch/sounds/en/us/callie 2011-06-30 15:52:38.526108 [DEBUG] mod_conference.c:5611 Raw Codec Activation Success L16@8000hz 1 channel 20ms 2011-06-30 15:52:38.526108 [DEBUG] mod_conference.c:5656 Raw Codec Activation Success L16@8000hz 1 channel 20ms 2011-06-30 15:52:38.526108 [DEBUG] mod_conference.c:1083 Setup timer success interval: 20 samples: 160 2011-06-30 15:52:38.526108 [DEBUG] mod_conference.c:2538 Setup timer soft success interval: 20 samples: 160 Dialplan: sofia/internal/sipp@192.168.41.206:5060 Action conference(${uuid}@default) EXECUTE sofia/internal/sipp@192.168.41.206:5060 conference( e0580a91-f0cc-4928-80fb-178b35e8c1c1@default ) 2011-06-30 15:52:43.550134 [INFO] mod_conference.c:6640 using channel sound prefix: /usr/local/freeswitch/sounds/en/us/callie 2011-06-30 15:52:43.550134 [DEBUG] mod_conference.c:5611 Raw Codec Activation Success L16@8000hz 1 channel 20ms 2011-06-30 15:52:43.550134 [DEBUG] mod_conference.c:5656 Raw Codec Activation Success L16@8000hz 1 channel 20ms 2011-06-30 15:52:43.550134 [DEBUG] mod_conference.c:2538 Setup timer soft success interval: 20 samples: 160 2011-06-30 15:52:43.550134 [DEBUG] mod_conference.c:1083 Setup timer success interval: 20 samples: 160 2011-06-30 15:52:43.570098 [DEBUG] mod_conference.c:2817 Channel leaving conference, cause: NORMAL_CLEARING 2011-06-30 15:52:43.570098 [DEBUG] mod_conference.c:6133 sofia/internal/sipp@192.168.41.206:5060 skip receive message [UNBRIDGE] (channel is hungup already) 2011-06-30 15:52:43.591136 [DEBUG] mod_conference.c:1529 Write Lock ON 2011-06-30 15:52:43.591136 [DEBUG] mod_conference.c:1532 Write Lock OFF 2011-06-30 15:52:48.599158 [DEBUG] mod_conference.c:2817 Channel leaving conference, cause: NORMAL_CLEARING 2011-06-30 15:52:48.599158 [DEBUG] mod_conference.c:6133 sofia/internal/sipp@192.168.41.206:5060 skip receive message [UNBRIDGE] (channel is hungup already) 2011-06-30 15:52:48.620154 [DEBUG] mod_conference.c:1529 Write Lock ON 2011-06-30 15:52:48.620154 [DEBUG] mod_conference.c:1532 Write Lock OFF

Anthony Minessale II added a comment - 30/Jun/11 4:05 PM and what if you stop the test and leave FS running. Did you track the memory then?

Anthony Minessale II added a comment - 30/Jun/11 5:12 PM do you have manage-presence set to false in your sip-profile. I suspect you are hitting the box beyonds its limits to keep up with the eventing. Compare results with removing manage-presence from sofia and running FS with -nosql option

Min Xie added a comment - 30/Jun/11 5:24 PM @Anthony, 1. No, stopping the traffic and the VM size of the freeswitch won't go back. 2. Yes, the manage-presence in external.xml is set to "false". Could you help us understand what it does? I will change it to "true" and run again.

Anthony Minessale II added a comment - 30/Jun/11 5:52 PM managet_presence true is bad in this case. I think you are overloading the box. Have you tried this test at a slower speed?

Min Xie added a comment - 30/Jun/11 6:00 PM Anthony, we used to have "manage-presence" set to "false" and having the VM constantly increasing problem. Now I am running with "manage-presence" set to "TRUE". I want to get another 100K calls to see how it goes. Now it seems the VM increase stopped for past 40K calls. The CPU utilization ratio seems normal, 220% on the target 16-core box. 300 concurrent calls + 300 conferences.

Anthony Minessale II added a comment - 30/Jun/11 6:02 PM manage-presence true will produce large amounts of events and extra work for your sip stack.

Min Xie added a comment - 30/Jun/11 6:04 PM The VM is still going up. I will change manage-presence to "false" and run with -nosql now.

Min Xie added a comment - 30/Jun/11 6:14 PM @Anthony, I tested with -nosql option for 300 seconds, peak was 316 calls at 15th second. 20K call processed. VM stats: VmPeak: 375376 kB VmPeak: 375376 kB VmPeak: 375376 kB VmPeak: 375376 kB VmPeak: 375376 kB VmPeak: 1090240 kB VmPeak: 1146060 kB VmPeak: 1146060 kB VmPeak: 1147916 kB VmPeak: 1148596 kB VmPeak: 1148596 kB VmPeak: 1148624 kB VmPeak: 1149552 kB VmPeak: 1151844 kB VmPeak: 1151844 kB VmPeak: 1151844 kB VmPeak: 1151844 kB VmPeak: 1152080 kB VmPeak: 1154168 kB VmPeak: 1154424 kB VmPeak: 1154424 kB VmPeak: 1154424 kB VmPeak: 1154424 kB VmPeak: 1154424 kB VmPeak: 1155860 kB VmPeak: 1155860 kB VmPeak: 1155860 kB VmPeak: 1156500 kB VmPeak: 1159096 kB VmPeak: 1159096 kB VmPeak: 1159096 kB VmPeak: 1159956 kB VmPeak: 1161612 kB VmPeak: 1161868 kB VmPeak: 1161868 kB VmPeak: 1161868 kB VmPeak: 1161868 kB VmPeak: 1161868 kB VmPeak: 1161868 kB VmPeak: 1161868 kB VmPeak: 1163596 kB VmPeak: 1166160 kB VmPeak: 1166160 kB

Min Xie added a comment - 30/Jun/11 6:17 PM The -nosql option test was executed with "manage-presence" to "false". My understanding is once the peak calls(along with conferences) is reached. The VM should not increase any more?

Anthony Minessale II added a comment - 30/Jun/11 6:17 PM and did you stop the test and watch if it reduces? and did you repeat the test without restarting after waiting? FreeSWITCH uses memory pools which will store memory and retain some memory in the process for other operations. Have you waited for a possible ceiling?

Min Xie added a comment - 30/Jun/11 6:32 PM @Anthony, Yes, I stopped the traffic and let FS to clear out all the calls and conferences. ------------------------------------- Thu Jun 30 18:27:11 CDT 2011 VmPeak: 1918016 kB ~~~ 63672 session(s) since startup 0 session(s) 0/1000 5000 session(s) max Conferences:       0 0 0 ------------------------------------- Thu Jun 30 18:27:21 CDT 2011 VmPeak: 1918016 kB ~~~ 63672 session(s) since startup 0 session(s) 0/1000 5000 session(s) max Conferences:       0 0 0 ------------------------------------- Thu Jun 30 18:27:31 CDT 2011 VmPeak: 1918016 kB ~~~ 63672 session(s) since startup 0 session(s) 0/1000 5000 session(s) max Conferences:       0 0 0 After couple of minutes the VM didn't drop back. Every single time I repeat the test I will restart FS and SIPP. Traffic starts after FS is ready. No mistakes there. As for the VM ceiling. In my test (60cps/5s call length). The peak call/conference count is between 300~325. My test normally lasts 10+ minutes. For the cause of > 1mil calls, we waited for couple of hours and the VM keeps getting bigger and bigger.

Anthony Minessale II added a comment - 30/Jun/11 6:57 PM what happens when you call the same conference every time instead of a unique one per channel?

Git added a comment - 30/Jun/11 7:24 PM Repository: freeswitch Branch: refs/heads/master Commit: 95145a1 http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=95145a1 Closed By: anthm@freeswitch.org Comment: FS-3386 --resolve please try this

Min Xie added a comment - 30/Jun/11 8:16 PM Will "make current" pick up the changes?

Min Xie added a comment - 01/Jul/11 10:08 AM @Anthony, We picked up the fix in switch_ivr_async.c with "make current". Executed two traffic test same as yesterday with "manage_presence = false". Memory leak speed is much slower but still there and big. From the generated chart, the memory leak patterns are similar. The results from with/without "-nosql" are extremely close. We don't think DB access is the leak factor. I've reopened this issue and attached the Excel file we used in analysis. There are 3 work books in the file for 3 test scenarios. 1. WITHOUT "-nosql", 560K calls. a. There is still no "ceiling" of VM usage. After stopping the traffic, the VM usage stays and won't decrease. b. Good news is, the VM size is about 50% of what we saw before the fix. But it will keep going up. The unused memory pools are still not released correctly. c. The peak number of calls was way before the test ends. And the peak call number is very small. ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host   60.0(0 ms)/1.000s 5060 9395.06 s 561954 192.168.41.219:5060(UDP)   0 new calls during 1.002 s period 1 ms scheduler resolution   0 calls (limit 900) Peak was 332 calls, after 7275 s   0 Running, 1083 Paused, 64 Woken up   0 dead call msg (discarded) 0 out-of-call msg (discarded)   3 open sockets   0 Total echo RTP pckts 1st stream 0.000 last period RTP rate (kB/s)   0 Total echo RTP pckts 2nd stream 0.000 last period RTP rate (kB/s)                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 561954 0 0          100 <---------- 561954 0 0 0          180 <---------- 0 0 0 0          200 <---------- E-RTD1 561954 0 0 0          ACK ----------> 561954 0        Pause [ 5000ms] 561954 0          BYE ----------> 561954 0 0          200 <---------- 561954 0 0 0 ----------------- Traffic Paused - Press [p] again to resume ------------------ 2. WITH "-nosql", 1928K calls. a. There is still no "ceiling" of VM usage. After stopping the traffic, the VM usage stays and won't decrease. b. Good news is, the VM size is about 50% of what we saw before the fix. But it will keep going up. The unused memory pools are still not released correctly. c. The peak number of calls was way before the test ends. And the peak call number is very small. ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host   60.0(0 ms)/1.000s 5060 35812.54 s 1928143 192.168.41.219:5060(UDP)   0 new calls during 1.002 s period 1 ms scheduler resolution   0 calls (limit 900) Peak was 329 calls, after 385 s   0 Running, 2 Paused, 4 Woken up   0 dead call msg (discarded) 0 out-of-call msg (discarded)   3 open sockets   0 Total echo RTP pckts 1st stream 0.000 last period RTP rate (kB/s)   0 Total echo RTP pckts 2nd stream 0.000 last period RTP rate (kB/s)                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 1928143 0 0          100 <---------- 1928143 0 0 0          180 <---------- 0 0 0 0          200 <---------- E-RTD1 1928143 0 0 0          ACK ----------> 1928143 0        Pause [ 5000ms] 1928143 0          BYE ----------> 1928143 0 0          200 <---------- 1928143 0 0 0 ----------------- Traffic Paused - Press [p] again to resume ------------------

Min Xie added a comment - 01/Jul/11 10:09 AM 3 Tests: 1. Without Anthony's fix to memory_pool 2. With the fix but without "-nosql" option 3. With the fix and with "-nosql" option.

Min Xie added a comment - 01/Jul/11 10:12 AM @Anthony, Please close FS-3351 . I believe it's sharing the same root cause with FS-3386 . http://jira.freeswitch.org/browse/FS-3351

Git added a comment - 01/Jul/11 12:52 PM Repository: freeswitch Branch: refs/heads/master Commit: 6f62f39 http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=6f62f39 Updated By: anthm@freeswitch.org Comment: FS-3386 fix small mem leak in sofia

Min Xie added a comment - 01/Jul/11 12:58 PM We picked up the change and running test now.

Min Xie added a comment - 01/Jul/11 2:36 PM Retested with 500K calls. 1. Same VM growing issue. About similar VM size after 500K calls. 2. VM size did not go down after stopping the traffic.

Anthony Minessale II added a comment - 01/Jul/11 5:12 PM The problem is gone. I replicated your test and replicated your original memory problem and its now gone. I started and stopped a test at 75cps many times and the VM remained in the area of 850mb and retreated to 600mb when the test was stopped. If you are using the defualt sipp uac, you are probably causing a Ddos to the sip stack. Try this modified scenario file: http://deathstar.freeswitch.org/load_test/dft_cap.xml  

Min Xie added a comment - 01/Jul/11 6:43 PM @Anthony, We tried the scenario file you suggested but still observed the same VM growing problem (the test we just performed had 260K calls and FS grew up to 1.72GB). After we stopped the traffic, the VM did not retreat. Have all your changes committed to GIT? We pulled down the changes to switch_ivrj_async.c and sofia.c. Is that all? Is there any special parameters when running SIPP as UAC? Only thing I changed to your scenario files is set the "pause" to 5000ms. Here is ours: ./sipp 192.168.41.219 -sf uac_lt3.xml -s 5575 -i 192.168.41.206 -r 0 -rtp_echo

Min Xie added a comment - 01/Jul/11 11:47 PM @Anthony, We picked up the latest git again to make sure (make current, make update, make sure, etc). And deployed the fresh new build to the target server. The version is: FreeSWITCH Version 1.0.head (git-6f62f39 2011-07-01 12-27-40 -0500) Started freeswitch as: ulimit -s 240 -> ./freeswitch -nosql -nc (manage-presence set to false). Started SIPP traffic gen as: ./sipp 192.168.41.219 -sf dft_cap.xml -d 5000 -s 5575 -i 192.168.41.206 -r 0 -rtp_echo Traffic is set to 75cps, 5000ms call duration. The full trace is attached to this JIRA (75cps-call-vm.log) As we saw before, the memory increase is not linear. It always jump up like going up stair by stair from a certain time point. Here is the time line: 1. Traffic started around <Fri Jul 1 22:38:18 CDT 2011>, VM usage was roughly maintained around 971MB. 2. Traffic paused around <Fri Jul 1 22:40:18 CDT 2011>, VM usage started to retreat and get back to 719MB. Call count was 8481. 3. Traffic resumed around <Fri Jul 1 22:41:09 CDT 2011>, VM usage still could maintained around 971MB to 982MB until call count hit somewhere around 27500. 4. You could see at <Fri Jul 1 22:45:29 CDT 2011>, VM usage started to jump up. 5. There was another big jump from 1.576GB to 1.982GB at <Fri Jul 1 22:53:11 CDT 2011> when call count hit to around 63000. 6. At <Fri Jul 1 23:00:02 CDT 2011>, call cont 939000, the VM hit to 4.182GB. 7. Paused the traffic, the VM usage dropped somehow back to 3.93GB and stayed there without change (we waited 20min and it's still the same). All timestamps are from the log file attached. How many calls did you run in your previous test? From the excel file we attached you could see the pattern how the VM jumps (the one with 500K calls, should be the 2nd workbook in that excel file). ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host 60.0(5000 ms)/1.000s 5060 627.88 s 38755 192.168.41.219:5060(UDP)   0 new calls during 0.000 s period 0 ms scheduler resolution   0 calls (limit 900) Peak was 411 calls, after 142 s   0 Running, 614 Paused, 0 Woken up   0 dead call msg (discarded) 0 out-of-call msg (discarded)   1 open sockets                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 38755 0 0          100 <---------- 38755 0 0 0          101 <---------- 0 0 0 0          180 <---------- 0 0 0 0          183 <---------- 0 0 0 0          480 <---------- 0 0 0 0          487 <---------- 0 0 0 0          404 <---------- 0 0 0 0          488 <---------- 0 0 0 0          502 <---------- 0 0 0 0          503 <---------- 0 0 0 0          486 <---------- 0 0 0 0          407 <---------- 0 0 0 0          408 <---------- 0 0 0 0          200 <---------- E-RTD1 38755 0 0 0          ACK ----------> 38755 0        Pause [ 5000ms] 38755 0          BYE ----------> 38755 0 0          200 <---------- 38755 0 0 0          ACK ----------> 0 0 ------------------------------ Test Terminated -------------------------------- ----------------------------- Statistics Screen ------- [1-9]: Change Screen --   Start Time | 2011-07-01 22:09:48:875 1309576188.875640   Last Reset Time | 2011-07-01 22:20:16:769 1309576816.769457   Current Time | 2011-07-01 22:20:16:769 1309576816.769612 -------------------------+---------------------------+--------------------------   Counter Name | Periodic value | Cumulative value -------------------------+---------------------------+--------------------------   Elapsed Time | 00:00:00:000 | 00:10:27:893   Call Rate | 0.000 cps | 61.722 cps -------------------------+---------------------------+--------------------------   Incoming call created | 0 | 0   OutGoing call created | 0 | 38755   Total Call created | | 38755   Current Call | 0 | -------------------------+---------------------------+--------------------------   Successful call | 0 | 38755   Failed call | 0 | 0 -------------------------+---------------------------+--------------------------   Response Time 1 | 00:00:00:000 | 00:00:00:018   Call Length | 00:00:00:000 | 00:00:05:033 ------------------------------ Test Terminated --------------------------------

Min Xie added a comment - 07/Jul/11 10:56 AM @Anthony, We used the latest GIT last night (git-9cf44f3 2011-07-06 16-45-36 -0500) to isolate the memory leak problem and find some different result. Actually under our traffic test, memory leaks happen in scenarios WITH and WITHOUT conference. 1. Test Setup Answer then Sleep dialplan as the following,     <extension name="loadtest-2">       <condition field="destination_number" expression="^(557[0-9])$">         <action application="answer"/>         <action application="sleep" data="5000"/>       </condition>     </extension> Answer then Conference dialplan as the following,     <extension name="loadtest-2">       <condition field="destination_number" expression="^(557[0-9])$">         <action application="answer"/>         <action application="conference" data="${uuid}@default"/>       </condition>     </extension> For both scenarios we used 100cps SIPP traffic with the scenario files you shared with us. Test call duration is 3s. ./sipp 192.168.41.219 -sf dft_cap.xml -d 3000 -s 5575 -i 192.168.41.206 -r 0 -rtp_echo 2. Test Result For answer-sleep case, FreeSwitch VM usage hit 6.27GB after 1.825 million calls. Stop the traffic, the VM usage dropped back just a tiny bit. For answer-conference case, FreeSwitch VM usage hit 9.35GB after 2.18 million calls. Stop the traffic, the VM usage dropped back a tiny bit as well. We made a excel workbook with chart to show the history of VM in the traffic. It's in the attachment (freeswitch-100cps-traffic-vm-usage.xlsx) The traffic runs clean. ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host 75.0(3000 ms)/1.000s 5060 40023.47 s 2191955 192.168.41.219:5080(UDP)   0 new calls during 1.002 s period 1 ms scheduler resolution   0 calls (limit 675) Peak was 241 calls, after 18919 s   0 Running, 2 Paused, 4 Woken up   0 dead call msg (discarded) 0 out-of-call msg (discarded)   3 open sockets   0 Total echo RTP pckts 1st stream 0.000 last period RTP rate (kB/s)   0 Total echo RTP pckts 2nd stream 0.000 last period RTP rate (kB/s)                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 2191955 0 0          100 <---------- 2191955 0 0 0          101 <---------- 0 0 0 0          180 <---------- 0 0 0 0          183 <---------- 0 0 0 0          480 <---------- 0 0 0 0          487 <---------- 0 0 0 0          404 <---------- 0 0 0 0          488 <---------- 0 0 0 0          502 <---------- 0 0 0 0          503 <---------- 0 0 0 0          486 <---------- 0 0 0 0          407 <---------- 0 0 0 0          408 <---------- 0 0 0 0          200 <---------- E-RTD1 2191955 0 0 0          ACK ----------> 2191955 0        Pause [ 3000ms] 2191955 0          BYE ----------> 2191955 0 0          200 <---------- 2191955 0 0 0          ACK ----------> 0 0 ----------------- Traffic Paused - Press [p] again to resume ------------------

Min Xie added a comment - 07/Jul/11 11:06 AM @Anthony, Just some corrections, 1. ./freeswitch -nosql -nc 2. "answer-sleep" scenario, 100cps traffic, 3s call duration. 3. "answer-conference" scenario, 75cps traffic, 3s call duration

Anthony Minessale II added a comment - 07/Jul/11 5:02 PM forget about -nosql try unloading mod_sofia and not stopping FS to compare. try memstat to see which files are using what memorry try slower cps to compare. This is pushing the bar on my tolerance of load testing related issues. If you want to continue much farther please acquire a support contract.

Min Xie added a comment - 07/Jul/11 6:29 PM @Anthony, Yes we are working on different options as well. But meanwhile, we used much effort to figure out the problem as well. We might need to put this aside to working on other area for a while (the team has worked on this problem for quite some time). Hope we could see you at ClueCon. The following might be useful for the developer to see if it's the cause. We managed to run it under Valgrind and rerpoduced the problem as well. 1. 3cps for 50% of the traffic but FS/Valgrind is too slow. 2cps is fairly stable. Only 27000 calls 2. The memory grow from 380MB to 1.34GB nonstop, stopping the traffic, FS status shows no calls remaining and FS memory stays at 1.34GB. 3. Same SIPP setup and Valgrind command as below 4. Uploaded the VG log file for this JIRA (vg-answer-conf.log) ./sipp 192.168.41.219:5080 -sf dft_cap.xml -d 3000 -s 5575 -i 192.168.41.206 -r 0 -rtp_echo valgrind --tool=memcheck --log-file=vg-answer-conf.log --leak-check=full --leak-resolution=high --show-reachable=yes .libs/freeswitch -vg -nosql freeswitch@freeswitch219 > status UP 0 years, 0 days, 3 hours, 25 minutes, 6 seconds, 716 milliseconds, 193 microseconds FreeSWITCH is ready 27005 session(s) since startup 0 session(s) 0/1000 5000 session(s) max min idle cpu 0.00/94.00 freeswitch@freeswitch219 > version FreeSWITCH Version 1.0.head (git-9cf44f3 2011-07-06 16-45-36 -0500) ------------------------------ Scenario Screen -------- [1-9]: Change Screen --   Call-rate(length) Port Total-time Total-calls Remote-host 2.0(3000 ms)/1.000s 5060 18835.71 s 27005 192.168.41.219:5080(UDP)   0 new calls during 1.002 s period 1 ms scheduler resolution   0 calls (limit 18) Peak was 27 calls, after 4321 s   0 Running, 2 Paused, 3 Woken up   11513 dead call msg (discarded) 338 out-of-call msg (discarded)   3 open sockets   0 Total echo RTP pckts 1st stream 0.000 last period RTP rate (kB/s)   0 Total echo RTP pckts 2nd stream 0.000 last period RTP rate (kB/s)                                  Messages Retrans Timeout Unexpected-Msg       INVITE ----------> 27005 15333 0          100 <---------- 27005 6109 0 0          101 <---------- 0 0 0 0          180 <---------- 0 0 0 0          183 <---------- 0 0 0 0          480 <---------- 0 0 0 0          487 <---------- 0 0 0 0          404 <---------- 0 0 0 0          488 <---------- 0 0 0 0          502 <---------- 0 0 0 0          503 <---------- 0 0 0 0          486 <---------- 0 0 0 0          407 <---------- 0 0 0 0          408 <---------- 0 0 0 0          200 <---------- E-RTD1 27005 14800 0 0          ACK ----------> 27005 14800        Pause [ 3000ms] 27005 0          BYE ----------> 27005 22754 0          200 <---------- 27005 0 0 0          ACK ----------> 0 0 ----------------- Traffic Paused - Press [p] again to resume ------------------

Min Xie added a comment - 07/Jul/11 10:38 PM @Anthony, I lost the previous VG log file. Re-ran the case, same result. 2 files attached. Valgrind log: vg-answer-conf.log Other info: fs-vg-other-msg The leak reported is definitely less than what we observed. So there are leaks not detected by VG. Please take a look at this when you have time. This memory leak is common to all the users, not just our scenarios. We are not sure if support contract is the best way to solve such common problems of FreeSwitch.

Git added a comment - 08/Jul/11 2:53 PM Repository: freeswitch Branch: refs/heads/master Commit: e420e17 http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=e420e17 Updated By: anthm@freeswitch.org Comment: FS-3386 Jeff Lenk found this one, Good Catch!

Min Xie added a comment - 08/Jul/11 2:57 PM Would you describe what is this fix for? Conference or Sofia? And we got "Changeset not found" for http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=e420e17

Jeff Lenk added a comment - 08/Jul/11 3:00 PM It fixes an issue in the switch_event.c handling code. sometimes fisheye takes a while to catch the commit.

Min Xie added a comment - 08/Jul/11 5:59 PM @Jeff, is this related to the memory leak problem we are working on here? On the other hand, should we rerun the traffic?

Anthony Minessale II added a comment - 08/Jul/11 6:13 PM Of course its related, why do you think the commit is filed against this bug. And based on your last comment trying to trivialize this problem by calling it "common". I am not much more interested in helping you unless you retract that statement. Any problem exacerbated by 100cps over millions of calls is far from common. And still takes precious billable time to fix.  

Min Xie added a comment - 08/Jul/11 6:20 PM @Anthony, We updated the system with latest fix and running the test now. Okay, I agree that 100cps or even 75cps is not common. But we do see the VM increasing under 2cps with Valgrind. We do believe that we are putting much effort to test and isolate the problem as well. The effort will benefit not only us. It's for the whole community. So far, this JIRA has resulted 3 valuable fixes (conference memory pool, sofia memory and switch_event). I wouldn't say it means nothing.

Anthony Minessale II added a comment - 08/Jul/11 6:47 PM That is a valid point, these issues would eventually be solved by our own Q/A testing but you have accelerated this and contributed to the solution to 3 issues. Thank you. My concern, however, is your comment that this is a "common" issue which is a way to make it sound simple and how it is not worthy of a support contract. Despite the details of the problem, the timing on when we fix it is based on a number of factors related to our own time and how much we can contribute to others. Based on your persistence in this issue and your intentions of using FreeSWITCH and Sangoma to build commercial services. I find your company an ideal candidate for commercial support services. You clearly have an ongoing list of urgent tasks to complete and expect reliability. There is a difference between helping someone who is trying to learn and helping someone who simply is trying to complete a task for their commercial goals. It is fortunate for our community that your current problem helps make FreeSWITCH more stable and removes a 3 headed memory leak. But that is also the purpose of open source and these regressions are very young only introduced in the last month. Perhaps I am just still annoyed by your colleague and his tweeted comment disparaging FreeSWITCH and questioning its reliability because he uncovered some edge case in our 3rd party SIP library. http://jira.freeswitch.org/browse/FS-3394 Too bad I spend a long time explaining several FreeSWITCH concepts to him when he first arrived. Bottom Line: I feel like your company is my customer regardless of your status with our company.

Min Xie added a comment - 10/Jul/11 10:25 AM @Anthony, Thanks for the comments. Yes, we are learning the FreeSwitch and trying to see what it is capable of (in the real production environment, not just some learning project). We found FreeSwitch very tempting and that's why we keep working on this case. Some of the points I am not agree with. As open source project like any others, the community is your QA and DEV and that's how the open projects thrive. If we consider this is a closed and license based product, then it's no difference that what our current vendors are. I also agree the support services are important to the open source project. We have to get through the evaluation phase to get there. Anyway, I grabbed the latest and ran some "normal" condition tests. Just want to share the test result. All the following case runs within the same subnet, FS SIPP scenario file, clean runs (not even single SIP message retrans). FS status shows 0 calls and 0 conferences left after stopping traffic. The final VM usages were taken after 5 minutes of stopping test traffic. For all scenarios, the VM usage has small drop back. If we keep the traffic running, the VM just keeps going up, as you could see in the results. 1. 50cps, answer then conf, SQL on, SIPP UAC sends BYE 559129 calls completed, VM stopped at 2.052GB. 2. 30cps, answer then sleep 5 seconds, SQL on, SIPP UAC sends BYE 1035988 calls completed, VM stopped at 3.136GB. 3. 30cps, answer then sleep 5 seconds, SQL off, SIPP UAC sends BYE 501303 calls completed, VM stopped at 1.809GB. 4. 30cps, answer then sleep 5 seconds, SQL on, FS SOFIA sends BYE 323292 calls completed, VM stopped at 1.334GB. 5. 30cps, answer then sleep 5 seconds, SQL off, FS SOFIA sends BYE 476037 calls completed, VM stopped at 1.686GB

Anthony Minessale II added a comment - 10/Jul/11 1:17 PM Please run these tests longer and make new valgrind req. There are memory pools in sofia sip and in apr that may need time to level out. Forget the support contract. We are not interested.

Min Xie added a comment - 13/Jul/11 9:49 AM 30cps, 48-hour run. SIPP scenario file: dft_cap.xml, call duration 3s. SIPP releases the call. FreeSwitch dialplan: answer then conference the incoming call to new conference. 4848561 sessons processed. FreeSwitch VMPeak 17.140GB FreeSwitch VMSize 16.944GB Got CRIT messages from FreeSwitch. 2011-07-13 04:34:47.875969 [CRIT] switch_event.c:366 Out of event dispatch threads! Slowing things down. 2011-07-13 04:34:48.875973 [CRIT] switch_event.c:366 Out of event dispatch threads! Slowing things down. 2011-07-13 04:34:49.876037 [CRIT] switch_event.c:343 Event system overloading. Taking a 10 second break, Reducin g max_sessions to 213 1sps 2011-07-13 04:34:50.006058 [CRIT] sofia.c:862 No more sessions allowed at this time. 2011-07-13 04:34:50.306053 [CRIT] sofia.c:862 No more sessions allowed at this time. 2011-07-13 04:34:50.306053 [CRIT] sofia.c:862 No more sessions allowed at this time. There are too many identical messages about "No more sessions" or "Out of event dispatch threads". Eventually, the max_sessions was reduced to 13.

Anthony Minessale II added a comment - 13/Jul/11 10:02 AM repeat on latest

Git added a comment - 13/Jul/11 12:39 PM Repository: freeswitch Branch: refs/heads/master Commit: e339b54 http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=e339b54 Updated By: anthm@freeswitch.org Comment: FS-3386 this is probably relevant, try this revision

Min Xie added a comment - 13/Jul/11 5:52 PM @Anthony, The last fix ( http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=e339b54 ) works. 50cps, answer+conference, SIPP release call. Exact same test setup. The following result captured after stopping traffic for 5 minutes. Without last fix: 507735 calls, VmPeak 2.12GB, VmSize 1.99GB With last fix: 507700 calls, VmPeak 707MB, VmSize 633MB (The VmPeak is very stable). We believe the root cause of the reported issue has been captured and fixed. Thanks Anthony and the team.

Min Xie added a comment - 13/Jul/11 5:54 PM Attach fixed version: FreeSWITCH Version 1.0.head (git-e339b54 2011-07-13 10-37-38 -0500)

Anthony Minessale II added a comment - 14/Jul/11 2:18 AM sadly i think this is not quite it, there is some ref counting errs related to this change.

Git added a comment - 14/Jul/11 2:19 AM Repository: freeswitch Branch: refs/heads/master Commit: 2932c1f http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=2932c1f Updated By: anthm@freeswitch.org Comment: FS-3386 Try this revision please

Min Xie added a comment - 14/Jul/11 1:05 PM @Anthony, From the test the latest fix (Commit: 2932c1f http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=2932c1f) reverted the good fix we tested yesterday. 50cps, 3s call duration, answer then conference. 500K calls. After stopping the traffic for 5 minutes, we got: Thu Jul 14 13:04:18 CDT 2011 VmPeak: 1848288 kB VmSize: 1757868 kB ~~~ UP 0 years, 0 days, 3 hours, 10 minutes, 30 seconds, 845 milliseconds, 349 microseconds FreeSWITCH is ready 500000 session(s) since startup 0 session(s) 0/1000 5000 session(s) max min idle cpu 0.00/100.00 Memory usage jumps from time to time during the testing like the following, hope it helps: ------------------------------------- Thu Jul 14 11:36:31 CDT 2011 VmPeak: 1280080 kB VmSize: 1280080 kB ~~~ UP 0 years, 0 days, 1 hour, 42 minutes, 43 seconds, 175 milliseconds, 433 microseconds FreeSWITCH is ready 303156 session(s) since startup 152 session(s) 5/1000 5000 session(s) max min idle cpu 0.00/100.00 ------------------------------------- Thu Jul 14 11:36:36 CDT 2011 VmPeak: 1411152 kB VmSize: 1345616 kB ~~~ UP 0 years, 0 days, 1 hour, 42 minutes, 48 seconds, 200 milliseconds, 496 microseconds FreeSWITCH is ready 303407 session(s) since startup 151 session(s) 5/1000 5000 session(s) max min idle cpu 0.00/91.00

Git added a comment - 15/Jul/11 2:25 AM Repository: freeswitch Branch: refs/heads/master Commit: 30ebe5d http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=30ebe5d Updated By: anthm@freeswitch.org Comment: FS-3386 this should do it

Min Xie added a comment - 15/Jul/11 9:16 AM Latest GIT crashes immediately after starting traffic (SQL is on). FreeSWITCH Version 1.0.head (git-c7c1cf7 2011-07-15 15-42-01 +0200) Core was generated by `./freeswitch'. Program terminated with signal 6, Aborted. #0 0x0000003db5430265 in raise () from /lib64/libc.so.6 (gdb) (gdb) (gdb) (gdb) where #0 0x0000003db5430265 in raise () from /lib64/libc.so.6 #1 0x0000003db5431d10 in abort () from /lib64/libc.so.6 #2 0x0000003db546a99b in __libc_message () from /lib64/libc.so.6 #3 0x0000003db54733d0 in _int_malloc () from /lib64/libc.so.6 #4 0x0000003db5474e2e in malloc () from /lib64/libc.so.6 #5 0x0000003db546964e in vasprintf () from /lib64/libc.so.6 #6 0x00002b65ec92250d in switch_event_add_header (event=0x2aaab0033940, stack=SWITCH_STACK_BOTTOM,     header_name=0x2aaab8561494 "Energy-Level",     fmt=0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>) at src/switch_event.c:1081 #7 0x00002aaab854f5d9 in conference_add_event_member_data (member=0x408df7c0, event=0x2aaab0033940)     at /root/freeswitch/git/freeswitch/src/mod/applications/mod_conference/mod_conference.c:503 #8 0x00002aaab855b569 in conference_del_member (conference=0x2aaaac086e60, member=0x408df7c0)     at /root/freeswitch/git/freeswitch/src/mod/applications/mod_conference/mod_conference.c:938 #9 0x00002aaab8561004 in conference_function (session=0x2aaab00e51c8, data=<value optimized out>)     at /root/freeswitch/git/freeswitch/src/mod/applications/mod_conference/mod_conference.c:6140 #10 0x00002b65ec906982 in switch_core_session_exec (session=0x2aaab00e51c8,     application_interface=0x2aaab001f648, arg=0x2aaab00f16d8 "${uuid}@default")     at src/switch_core_session.c:2115 #11 0x00002b65ec906faf in switch_core_session_execute_application_get_flags (session=0xeff,     app=0x2aaab00f16c8 "conference", arg=0x2aaab00f16d8 "${uuid}@default", flags=0x0)     at src/switch_core_session.c:2000 #12 0x00002b65ec909952 in switch_core_session_run (session=0x2aaab00e51c8)     at src/switch_core_state_machine.c:176 #13 0x00002b65ec903db0 in switch_core_session_thread (thread=<value optimized out>, obj=0x2aaab00e51c8)     at src/switch_core_session.c:1309 #14 0x0000003db640673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003db54d40cd in clone () from /lib64/libc.so.6 (gdb) 2011-07-15 09:11:13.286984 [NOTICE] switch_core_session.c:1346 Close Channel sofia/external/sipp@192.168.41.2 06:5060 [CS_DESTROY] *** glibc detected *** ./freeswitch: corrupted double-linked list: 0x00002aaaac048130 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3db5470793] /lib64/libc.so.6[0x3db5472bbc] /lib64/libc.so.6(__libc_malloc+0x6e)[0x3db5474e2e] /usr/local/freeswitch/lib/libfreeswitch.so.1(sqlite3MallocX+0x19)[0x2b418fb7e8a9] /usr/local/freeswitch/lib/libfreeswitch.so.1(sqlite3ParserAlloc+0xe)[0x2b418fba2fae] /usr/local/freeswitch/lib/libfreeswitch.so.1(sqlite3RunParser+0x48)[0x2b418fb7ada8] /usr/local/freeswitch/lib/libfreeswitch.so.1(sqlite3Prepare+0xc8)[0x2b418fb77aa8] /usr/local/freeswitch/lib/libfreeswitch.so.1(sqlite3_exec+0x7e)[0x2b418fb87a8e] /usr/local/freeswitch/lib/libfreeswitch.so.1(switch_core_db_exec+0x61)[0x2b418faf99a1] /usr/local/freeswitch/lib/libfreeswitch.so.1(switch_cache_db_execute_sql_callback+0xca)[0x2b418fae2a5a] /usr/local/freeswitch/mod/mod_sofia.so[0x2aaaaba52c07] /usr/local/freeswitch/mod/mod_sofia.so[0x2aaaaba6aed9] /lib64/libpthread.so.0[0x3db640673d] /lib64/libc.so.6(clone+0x6d)[0x3db54d40cd] ======= Memory map: ======== 00400000-00404000 r-xp 00000000 fd:06 59179020 /usr/local/freeswitch/bin/freeswitch 00604000-00605000 rw-p 00004000 fd:06 59179020 /usr/local/freeswitch/bin/freeswitch 0c679000-0cf29000 rw-p 0c679000 00:00 0 [heap] 4006f000-40070000 ---p 4006f000 00:00 0 40070000-400ab000 rwxp 40070000 00:00 0 400ab000-400ac000 ---p 400ab000 00:00 0 400ac000-400e7000 rwxp 400ac000 00:00 0 400ed000-400ee000 ---p 400ed000 00:00 0 400ee000-40129000 rwxp 400ee000 00:00 0 40129000-4012a000 ---p 40129000 00:00 0 4012a000-40165000 rwxp 4012a000 00:00 0 4019c000-4019d000 ---p 4019c000 00:00 0 4019d000-401d8000 rwxp 4019d000 00:00 0 401d8000-401d9000 ---p 401d8000 00:00 0 401d9000-40214000 rwxp 401d9000 00:00 0 4021d000-4021e000 ---p 4021d000 00:00 0 4021e000-40259000 rwxp 4021e000 00:00 0 40259000-4025a000 ---p 40259000 00:00 0 4025a000-40295000 rwxp 4025a000 00:00 0 40295000-40296000 ---p 40295000 00:00 0 40296000-402d1000 rwxp 40296000 00:00 0 402d1000-402d2000 ---p 402d1000 00:00 0 402d2000-4030d000 rwxp 402d2000 00:00 0 4030d000-4030e000 ---p 4030d000 00:00 0 4030e000-40349000 rwxp 4030e000 00:00 0 40349000-4034a000 ---p 40349000 00:00 0 4034a000-40385000 rwxp 4034a000 00:00 0 403ad000-403ae000 ---p 403ad000 00:00 0 403ae000-403e9000 rwxp 403ae000 00:00 0 4040c000-4040d000 ---p 4040c000 00:00 0 4040d000-40448000 rwxp 4040d000 00:00 0 40448000-40449000 ---p 40448000 00:00 0 40449000-40484000 rwxp 40449000 00:00 0 40484000-40485000 ---p 40484000 00:00 0 40485000-404c0000 rwxp 40485000 00:00 0 404d7000-404d8000 ---p 404d7000 00:00 0 404d8000-40513000 rwxp 404d8000 00:00 0 40513000-40514000 ---p 40513000 00:00 0 40514000-4054f000 rwxp 40514000 00:00 0 4055d000-4055e000 ---p 4055d000 00:00 0 4055e000-40599000 rwxp 4055e000 00:00 0 40599000-4059a000 ---p 40599000 00:00 0 4059a000-405d5000 rwxp 4059a000 00:00 0 405d5000-405d6000 ---p 405d5000 00:00 0 405d6000-40611000 rwxp 405d6000 00:00 0 40611000-40612000 ---p 40611000 00:00 0 40612000-4064d000 rwxp 40612000 00:00 0 40682000-40683000 ---p 40682000 00:00 0 40683000-406be000 rwxp 40683000 00:00 0 406be000-406bf000 ---p 406be000 00:00 0 406bf000-406fa000 rwxp 406bf000 00:00 0 40705000-40706000 ---p 40705000 00:00 0 40706000-40741000 rwxp 40706000 00:00 0 4074b000-4074c000 ---p 4074b000 00:00 0 4074c000-40787000 rwxp 4074c000 00:00 0 40787000-40788000 ---p 40787000 00:00 0 40788000-407c3000 rwxp 40788000 00:00 0 407c3000-407c4000 ---p 407c3000 00:00 0 407c4000-407ff000 rwxp 407c4000 00:00 0 407ff000-40800000 ---p 407ff000 00:00 0 40800000-4083b000 rwxp 40800000 00:00 0 40843000-40844000 ---p 40843000 00:00 0 40844000-4087f000 rwxp 40844000 00:00 0 4087f000-40880000 ---p 4087f000 00:00 0 40880000-408bb000 rwxp 40880000 00:00 0 408d6000-408d7000 ---p 408d6000 00:00 0 408d7000-40912000 rwxp 408d7000 00:00 0 40912000-40913000 ---p 40912000 00:00 0 40913000-4094e000 rwxp 40913000 00:00 0 4094e000-4094f000 ---p 4094e000 00:00 0 4094f000-4098a000 rwxp 4094f000 00:00 0 409c2000-409c3000 ---p 409c2000 00:00 0 409c3000-409fe000 rwxp 409c3000 00:00 0 409fe000-409ff000 ---p 409fe000 00:00 0 409ff000-40a3a000 rwxp 409ff000 00:00 0 40a49000-40a4a000 ---p 40a49000 00:00 0 40a4a000-40a85000 rwxp 40a4a000 00:00 0 40a85000-40a86000 ---p 40a85000 00:00 0 40a86000-40ac1000 rwxp 40a86000 00:00 0 40ac1000-40ac2000 ---p 40ac1000 00:00 0 40ac2000-40afd000 rwxp 40ac2000 00:00 0 40afe000-40aff000 ---p 40afe000 00:00 0 40aff000-40b3a000 rwxp 40aff000 00:00 0 40b3a000-40b3b000 ---p 40b3a000 00:00 0 40b3b000-40b76000 rwxp 40b3b000 00:00 0 40b8a000-40b8b000 ---p 40b8a000 00:00 0 40b8b000-40bc6000 rwxp 40b8b000 00:00 0 40bc6000-40bc7000 ---p 40bc6000 00:00 0 40bc7000-40c02000 rwxp 40bc7000 00:00 0 40c02000-40c03000 ---p 40c02000 00:00 0 40c03000-40c3e000 rwxp 40c03000 00:00 0 40c3e000-40c3f000 ---p 40c3e000 00:00 0 40c3f000-40c7a000 rwxp 40c3f000 00:00 0 40ca7000-40ca8000 ---p 40ca7000 00:00 0 40ca8000-40ce3000 rwxp 40ca8000 00:00 0 40ce3000-40ce4000 ---p 40ce3000 00:00 0 40ce4000-40d1f000 rwxp 40ce4000 00:00 0 40d1f000-40d20000 ---p 40d1f000 00:00 0 40d20000-40d5b000 rwxp 40d20000 00:00 0 40d5b000-40d5c000 ---p 40d5b000 00:00 0 40d5c000-40d97000 rwxp 40d5c000 00:00 0 40d97000-40d98000 ---p 40d97000 00:00 0 40d98000-40dd3000 rwxp 40d98000 00:00 0 40dd3000-40dd4000 ---p 40dd3000 00:00 0 40dd4000-40e0f000 rwxp 40dd4000 00:00 0 40e14000-40e15000 ---p 40e14000 00:00 0 40e15000-40e50000 rwxp 40e15000 00:00 0 40e6a000-40e6b000 ---p 40e6a000 00:00 0 40e6b000-40ea6000 rwxp 40e6b000 00:00 0 40eaa000-40eab000 ---p 40eaa000 00:00 0 40eab000-40ee6000 rwxp 40eab000 00:00 0 40f18000-40f19000 ---p 40f18000 00:00 0 40f19000-40f54000 rwxp 40f19000 00:00 0 40f54000-40f55000 ---p 40f54000 00:00 0 40f55000-40f90000 rwxp 40f55000 00:00 0 40f90000-40f91000 ---p 40f90000 00:00 0 40f91000-40fcc000 rwxp 40f91000 00:00 0 40fcc000-40fcd000 ---p 40fcc000 00:00 0 40fcd000-41008000 rwxp 40fcd000 00:00 0 41011000-41012000 ---p 41011000 00:00 0 41012000-4104d000 rwxp 41012000 00:00 0 4104d000-4104e000 ---p 4104d000 00:00 0 4104e000-41089000 rwxp 4104e000 00:00 0 410b4000-410b5000 ---p 410b4000 00:00 0 410b5000-410f0000 rwxp 410b5000 00:00 0 410f0000-410f1000 ---p 410f0000 00:00 0 410f1000-4112c000 rwxp 410f1000 00:00 0 4112c000-4112d000 ---p 4112c000 00:00 0 4112d000-41168000 rwxp 4112d000 00:00 0 41168000-41169000 ---p 41168000 00:00 0 41169000-411a4000 rwxp 41169000 00:00 0 411a4000-411a5000 ---p 411a4000 00:00 0 411a5000-411e0000 rwxp 411a5000 00:00 0 411e0000-411e1000 ---p 411e0000 00:00 0 411e1000-4121c000 rwxp 411e1000 00:00 0 41237000-41238000 ---p 41237000 00:00 0 41238000-41273000 rwxp 41238000 00:00 0 41276000-41277000 ---p 41276000 00:00 0 41277000-412b2000 rwxp 41277000 00:00 0 412b2000-412b3000 ---p 412b2000 00:00 0 412b3000-412ee000 rwxp 412b3000 00:00 0 412ee000-412ef000 ---p 412ee000 00:00 0 412ef000-4132a000 rwxp 412ef000 00:00 0 4132a000-4132b000 ---p 4132a000 00:00 0 4132b000-41366000 rwxp 4132b000 00:00 0 41366000-41367000 ---p 41366000 00:00 0 41367000-413a2000 rwxp 41367000 00:00 0 413a2000-413a3000 ---p 413a2000 00:00 0 413a3000-413de000 rwxp 413a3000 00:00 0 413de000-413df000 ---p 413de000 00:00 0 413df000-4141a000 rwxp 413df000 00:00 0 4141a000-4141b000 ---p 4141a000 00:00 0 4141b000-41456000 rwxp 4141b000 00:00 0 41481000-41482000 ---p 41481000 00:00 0 41482000-414bd000 rwxp 41482000 00:00 0 414bd000-414be000 ---p 414bd000 00:00 0 414be000-414f9000 rwxp 414be000 00:00 0 414f9000-414fa000 ---p 414f9000 00:00 0 414fa000-41535000 rwxp 414fa000 00:00 0 41535000-41536000 ---p 41535000 00:00 0 41536000-41571000 rwxp 41536000 00:00 0 41591000-41592000 ---p 41591000 00:00 0 41592000-415cd000 rwxp 41592000 00:00 0 415cd000-415ce000 ---p 415cd000 00:00 0 415ce000-41609000 rwxp 415ce000 00:00 0 41609000-4160a000 ---p 41609000 00:00 0 4160a000-41645000 rwxp 4160a000 00:00 0 41645000-41646000 ---p 41645000 00:00 0 41646000-41681000 rwxp 41646000 00:00 0 41681000-41682000 ---p 41681000 00:00 0 41682000-416bd000 rwxp 41682000 00:00 0 416d3000-416d4000 ---p 416d3000 00:00 0 416d4000-4170f000 rwxp 416d4000 00:00 0 4170f000-41710000 ---p 4170f000 00:00 0 41710000-4174b000 rwxp 41710000 00:00 0 4174b000-4174c000 ---p 4174b000 00:00 0 4174c000-41787000 rwxp 4174c000 00:00 0 417b3000-417b4000 ---p 417b3000 00:00 0 417b4000-417ef000 rwxp 417b4000 00:00 0 417ef000-417f0000 ---p 417ef000 00:00 0 417f0000-4182b000 rwxp 417f0000 00:00 0 41846000-41847000 ---p 41846000 00:00 0 41847000-41882000 rwxp 41847000 00:00 0 4188c000-4188d000 ---p 4188c000 00:00 0 4188d000-418c8000 rwxp 4188d000 00:00 0 418c8000-418c9000 ---p 418c8000 00:00 0 418c9000-41904000 rwxp 418c9000 00:00 0 41904000-41905000 ---p 41904000 00:00 0 41905000-41940000 rwxp 41905000 00:00 0 41940000-41941000 ---p 41940000 00:00 0 41941000-4197c000 rwxp 41941000 00:00 0 4197c000-4197d000 ---p 4197c000 00:00 0 4197d000-419b8000 rwxp 4197d000 00:00 0 419b8000-419b9000 ---p 419b8000 00:00 0 419b9000-419f4000 rwxp 419b9000 00:00 0 419f4000-419f5000 ---p 419f4000 00:00 0 419f5000-41a30000 rwxp 419f5000 00:00 0 41a42000-41a43000 ---p 41a42000 00:00 0 41a43000-41a7e000 rwxp 41a43000 00:00 0 41a7e000-41a7f000 ---p 41a7e000 00:00 0 41a7f000-41aba000 rwxp 41a7f000 00:00 0 41aca000-41acb000 ---p 41aca000 00:00 0 41acb000-41b06000 rwxp 41acb000 00:00 0 41b06000-41b07000 ---p 41b06000 00:00 0 41b07000-41b42000 rwxp 41b07000 00:00 0 41b42000-41b43000 ---p 41b42000 00:00 0 41b43000-41b7e000 rwxp 41b43000 00:00 0 41bb5000-41bb6000 ---p 41bb5000 00:00 0 41bb6000-41bf1000 rwxp 41bb6000 00:00 0 41c22000-41c23000 ---p 41c22000 00:00 0 41c23000-41c5e000 rwxp 41c23000 00:00 0 41c5e000-41c5f000 ---p 41c5e000 00:00 0 41c5f000-41c9a000 rwxp 41c5f000 00:00 0 41c9a000-41c9b000 ---p 41c9a000 00:00 0 41c9b000-41cd6000 rwxp 41c9b000 00:00 0 41d02000-41d03000 ---p 41d02000 00:00 0 41d03000-41d3e000 rwxp 41d03000 00:00 0 41d3e000-41d3f000 ---p 41d3e000 00:00 0 41d3f000-41d7a000 rwxp 41d3f000 00:00 0 41d84000-41d85000 ---p 41d84000 00:00 0 41d85000-41dc0000 rwxp 41d85000 00:00 0 41dc0000-41dc1000 ---p 41dc0000 00:00 0 41dc1000-41dfc000 rwxp 41dc1000 00:00 0 41dfc000-41dfd000 ---p 41dfc000 00:00 0 41dfd000-41e38000 rwxp 41dfd000 00:00 0 41e38000-41e39000 ---p 41e38000 00:00 0 41e39000-41e74000 rwxp 41e39000 00:00 0 41e74000-41e75000 ---p 41e74000 00:00 0 41e75000-41eb0000 rwxp 41e75000 00:00 0 41ee0000-41ee1000 ---p 41ee0000 00:00 0 41ee1000-41f1c000 rwxp 41ee1000 00:00 0 41f33000-41f34000 ---p 41f33000 00:00 0 41f34000-41f6f000 rwxp 41f34000 00:00 0 41f6f000-41f70000 ---p 41f6f000 00:00 0 41f70000-41fab000 rwxp 41f70000 00:00 0 41fe5000-41fe6000 ---p 41fe5000 00:00 0 41fe6000-42021000 rwxp 41fe6000 00:00 0 42021000-42022000 ---p 42021000 00:00 0 42022000-4205d000 rwxp 42022000 00:00 0 4205d000-4205e000 ---p 4205d000 00:00 0 4205e000-42099000 rwxp 4205e000 00:00 0 42099000-4209a000 ---p 42099000 00:00 0 4209a000-420d5000 rwxp 4209a000 00:00 0 420d5000-420d6000 ---p 420d5000 00:00 0 420d6000-42111000 rwxp 420d6000 00:00 0 42111000-42112000 ---p 42111000 00:00 0 42112000-4214d000 rwxp 42112000 00:00 0 4214d000-4214e000 ---p 4214d000 00:00 0 4214e000-42189000 rwxp 4214e000 00:00 0 42189000-4218a000 ---p 42189000 00:00 0 4218a000-421c5000 rwxp 4218a000 00:00 0 421c5000-421c6000 ---p 421c5000 00:00 0 421c6000-42201000 rwxp 421c6000 00:00 0 42201000-42202000 ---p 42201000 00:00 0 42202000-4223d000 rwxp 42202000 00:00 0 4223d000-4223e000 ---p 4223d000 00:00 0 4223e000-42279000 rwxp 4223e000 00:00 0 42279000-4227a000 ---p 42279000 00:00 0 4227a000-422b5000 rwxp 4227a000 00:00 0 422b5000-422b6000 ---p 422b5000 00:00 0 422b6000-422f1000 rwxp 422b6000 00:00 0 422f1000-422f2000 ---p 422f1000 00:00 0 422f2000-4232d000 rwxp 422f2000 00:00 0 4232d000-4232e000 ---p 4232d000 00:00 0 4232e000-42369000 rwxp 4232e000 00:00 0 42369000-4236a000 ---p 42369000 00:00 0 4236a000-423a5000 rwxp 4236a000 00:00 0 423a5000-423a6000 ---p 423a5000 00:00 0 423a6000-423e1000 rwxp 423a6000 00:00 0 423e1000-423e2000 ---p 423e1000 00:00 0 423e2000-4241d000 rwxp 423e2000 00:00 0 4241d000-4241e000 ---p 4241d000 00:00 0 4241e000-42459000 rwxp 4241e000 00:00 0 42459000-4245a000 ---p 42459000 00:00 0 4245a000-42495000 rwxp 4245a000 00:00 0 42495000-42496000 ---p 42495000 00:00 0 42496000-424d1000 rwxp 42496000 00:00 0 424d1000-424d2000 ---p 424d1000 00:00 0 424d2000-4250d000 rwxp 424d2000 00:00 0 4250d000-4250e000 ---p 4250d000 00:00 0 4250e000-42549000 rwxp 4250e000 00:00 0 42549000-4254a000 ---p 42549000 00:00 0 4254a000-42585000 rwxp 4254a000 00:00 0 42585000-42586000 ---p 42585000 00:00 0 42586000-425c1000 rwxp 42586000 00:00 0 425c1000-425c2000 ---p 425c1000 00:00 0 425c2000-425fd000 rwxp 425c2000 00:00 0 425fd000-425fe000 ---p 425fd000 00:00 0 425fe000-42639000 rwxp 425fe000 00:00 0 42639000-4263a000 ---p 42639000 00:00 0 4263a000-42675000 rwxp 4263a000 00:00 0 42675000-42676000 ---p 42675000 00:00 0 42676000-426b1000 rwxp 42676000 00:00 0 426b1000-426b2000 ---p 426b1000 00:00 0 426b2000-426ed000 rwxp 426b2000 00:00 0 426ed000-426ee000 ---p 426ed000 00:00 0 426ee000-42729000 rwxp 426ee000 00:00 0 42729000-4272a000 ---p 42729000 00:00 0 4272a000-42765000 rwxp 4272a000 00:00 0 42765000-42766000 ---p 42765000 00:00 0 42766000-427a1000 rwxp 42766000 00:00 0 427a1000-427a2000 ---p 427a1000 00:00 0 427a2000-427dd000 rwxp 427a2000 00:00 0 427dd000-427de000 ---p 427dd000 00:00 0 427de000-42819000 rwxp 427de000 00:00 0 42819000-4281a000 ---p 42819000 00:00 0 4281a000-42855000 rwxp 4281a000 00:00 0 42855000-42856000 ---p 42855000 00:00 0 42856000-42891000 rwxp 42856000 00:00 0 42891000-42892000 ---p 42891000 00:00 0 42892000-428cd000 rwxp 42892000 00:00 0 428cd000-428ce000 ---p 428cd000 00:00 0 428ce000-42909000 rwxp 428ce000 00:00 0 42909000-4290a000 ---p 42909000 00:00 0 4290a000-42945000 rwxp 4290a000 00:00 0 42945000-42946000 ---p 42945000 00:00 0 42946000-42981000 rwxp 42946000 00:00 0 42981000-42982000 ---p 42981000 00:00 0 Aborted (core dumped)

Min Xie added a comment - 15/Jul/11 9:25 AM @Anthony, The call scenario is still answer the incoming call from SIPP then join to new conference. Then SIPP UAC releases the call.

Min Xie added a comment - 15/Jul/11 9:31 AM Answer + Bridge scenario crashes as well. FreeSWITCH Version 1.0.head (git-c7c1cf7 2011-07-15 15-42-01 +0200) (gdb) (gdb) where #0 0x00002aaab454069b in su_port_decref (rmsg=<value optimized out>) at su_port.h:231 #1 su_msg_destroy (rmsg=<value optimized out>) at su_root.c:1191 #2 0x00002aaab446ce58 in sofia_process_dispatch_event (dep=<value optimized out>) at sofia.c:1175 #3 0x00002aaab4450ea8 in sofia_receive_message (session=0xc57fff8, msg=0x4183d030) at mod_sofia.c:1384 #4 0x00002b2716bdf715 in switch_core_session_perform_receive_message (session=0xc57fff8, message=0x0,     file=<value optimized out>, func=<value optimized out>, line=<value optimized out>)     at src/switch_core_session.c:678 #5 0x00002b2716c2a26f in switch_ivr_parse_all_signal_data (session=0xc57fff8) at src/switch_ivr.c:727 #6 0x00002b2716bc192e in switch_channel_check_signal (channel=0xc584b38, in_thread_only=SWITCH_FALSE)     at src/switch_channel.c:1708 #7 0x00002b2716bc3501 in switch_channel_event_set_basic_data (channel=0xc584b38, event=0x2aaaac14d360)     at src/switch_channel.c:2126 #8 0x00002b2716bc37b8 in switch_channel_event_set_data (channel=0xc584b38, event=0x2aaaac14d360)     at src/switch_channel.c:2261 #9 0x00002b2716c0a804 in switch_ivr_multi_threaded_bridge (session=0xc57fff8, peer_session=0xc594258,     input_callback=<value optimized out>, session_data=0x0, peer_session_data=0x0)     at src/switch_ivr_bridge.c:1303 #10 0x00002aaab49d6c08 in audio_bridge_function (session=0xc57fff8, data=<value optimized out>)     at /root/freeswitch/git/freeswitch/src/mod/applications/mod_dptools/mod_dptools.c:2844 #11 0x00002b2716be0982 in switch_core_session_exec (session=0xc57fff8, application_interface=0xc552738,     arg=0xc58a448 "sofia/external/user@192.168.41.200") at src/switch_core_session.c:2115 #12 0x00002b2716be0faf in switch_core_session_execute_application_get_flags (session=0x2aaaac082f10,     app=0xc58a440 "bridge", arg=0xc58a448 "sofia/external/user@192.168.41.200", flags=0x0)     at src/switch_core_session.c:2000 #13 0x00002b2716be3952 in switch_core_session_run (session=0xc57fff8) at src/switch_core_state_machine.c:176 #14 0x00002b2716bdddb0 in switch_core_session_thread (thread=<value optimized out>, obj=0xc57fff8)     at src/switch_core_session.c:1309 #15 0x0000003db640673d in start_thread () from /lib64/libpthread.so.0 #16 0x0000003db54d40cd in clone () from /lib64/libc.so.6 (gdb)

Min Xie added a comment - 15/Jul/11 9:46 AM @Anthony, both cases happened when FreeSwitch received 2nd BYE message from SIPP of the 2nd call. The first call was fine. Just FYI.

David Bolen added a comment - 15/Jul/11 6:19 PM This may be redundant, but I found this issue while investigating some core dumps of my own. I'm new to FreeSWITCH and have been working from git - after updating today I began core dumping immediately after a call is bridged from a SIP handset to a PSTN number over a SIP gateway (flowroute), whether or not the b-leg/PSTN is answered. This is a test setup, so the switch was quiescent aside from my single test call (or I suppose any regular SIP background traffic). I was going to include some stack traces (and still can) but they vary a lot, and the issue appears to be earlier general memory corruption as the varying failure points are always in memory management activities, such as su_port_decref (as in Min Xie's), and also su_alloc as well as the libc malloc. So I suspect the actual dumps are past the point of corruption. I can assure a dump on any attempt to make the handset call. Reverting commit 30ebe5d individually appears sufficient to resolve the problem, at least in my test case. I haven't attempted to dig into the commit itself yet.

Anthony Minessale II added a comment - 15/Jul/11 6:36 PM yes this whole thing is a big can of worms, stay on old version and stay tuned for update.

Git added a comment - 16/Jul/11 3:06 AM Repository: freeswitch Branch: refs/heads/master Commit: b0e076a http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=b0e076a Updated By: anthm@freeswitch.org Comment: FS-3386 add some more debug defines to sofia and avoid double destroy in nh

Git added a comment - 16/Jul/11 3:06 AM Repository: freeswitch Branch: refs/heads/master Commit: 1675981 http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=1675981 Updated By: anthm@freeswitch.org Comment: FS-3386 roll back a few revs then remove some refs and reroll patches that were in between

Min Xie added a comment - 16/Jul/11 11:26 AM @Anthony, Latest GIT fixed the crash problem and now the traffic is running. after 400K calls the VM is stable at 760MB for answer+conference scenario. However, there are changes for the sofia_reg.c and bridge scenario is broken. Before the crash, the bridge runs fine.     <extension name="loadtest-5573">       <condition field="destination_number" expression="^(5573)$">         <action application="bridge" data="sofia/external/user@192.168.41.200"/>       </condition>     </extension> 2011-07-16 10:46:05.757326 [DEBUG] switch_core_session.c:855 Send signal sofia/external/user@192.168.41.200 [BREAK] 2011-07-16 10:46:05.757326 [ERR] sofia_reg.c:2011 Cannot locate any authentication credentials to complete an authentication request for realm '"192.168.41.219"' 2011-07-16 10:46:05.757326 [DEBUG] switch_channel.c:2739 (sofia/external/user@192.168.41.200) Callstate Change RINGING -> HANGUP

Git added a comment - 16/Jul/11 1:33 PM Repository: freeswitch Branch: refs/heads/master Commit: 73ba5a5 http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=73ba5a5 Updated By: anthm@freeswitch.org Comment: FS-3386

Git added a comment - 16/Jul/11 1:33 PM Repository: freeswitch Branch: refs/heads/master Commit: 1729078 http://fisheye.freeswitch.org/changelog/freeswitch.git/?cs=1729078 Updated By: anthm@freeswitch.org Comment: FS-3386

Mike Jerris added a comment - 16/Jul/11 1:34 PM This looks like your calling something your being challenged for not specifying the gateway. Can you look at the latest rev, and enable sip trace as well and see whats going on.

Anthony Minessale II added a comment - 16/Jul/11 1:58 PM create a gateway either name it 192.168.41.219 or add a realm param of 192.168.41.219   <gateway name="192.168.41.219">     <params>       <param name="proxy" value=""/>       <param name="username" value="1001"/>       <param name="password" value="1234"/>       <param name="expire-seconds" value="30"/>       <param name="retry-seconds" value="30"/>       <param name="register" value="true"/>     </params>   </gateway> even better name it what you want like foo then use the gateway syntax sofia/gateway/foo/user

David Bolen added a comment - 16/Jul/11 2:27 PM Rebuilding with the latest master branch (f1de377, so one commit beyond 1729078 from above) also no longer crashes in my test scenario.

Min Xie added a comment - 16/Jul/11 3:42 PM Thanks Anthony. We found the problem. Now running traffic for bridge scenario. Will update the result soon.

Min Xie added a comment - 17/Jul/11 10:24 AM 500K calls (50cps) of answer+conference and the system VM usage is stable. 3000K calls (50cps) of answer+bridge, no crash, VM usage is stable. Next test is to involve playback. Looks good so far.

Min Xie added a comment - 18/Jul/11 9:31 AM 1000K calls (50cps) involving answer, playback and bridge. VM usage is very stable. @Anthony, this reported issue is fixed. Please close it. If you need me to close it, let me know. Thanks!

Auto Admin added a comment - 31/Jul/11 8:17 PM Due to a long period of inactivity (13 or more days), this issue is due to be automatically close within 24 hours. If this issue is not actually resolved, please reopen it and add appropriate comments to help developers fix the issue. Thanks,   Jira Admin

Auto Admin added a comment - 01/Aug/11 8:17 PM Since there has been no change to the status of this issue, it is being closed for inactivity Thanks,   Jira Admin