FreeSWITCH

FreeSWITCH crashes without coredump - Sofia bug?

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 1.0.7
  • Component/s: mod_sofia
  • Labels:
    None
  • Environment:
    OpenSUSE 11.2
  • Platform:
    Linux x86_64/gcc
  • Uname:
    Linux 2.6.31.12-0.2-default #1 SMP 2010-03-16 21:25:39 +0100 x86_64 x86_64 x86_64 GNU/Linux
  • CPU Info:
    Hide
    processor : 0
    vendor_id : GenuineIntel
    cpu family : 15
    model : 4
    model name : Intel(R) Xeon(TM) CPU 3.00GHz
    stepping : 3
    cpu MHz : 2992.505
    cache size : 2048 KB
    physical id : 0
    siblings : 2
    core id : 0
    cpu cores : 1
    apicid : 0
    initial apicid : 0
    fpu : yes
    fpu_exception : yes
    cpuid level : 5
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr
    bogomips : 5985.01
    clflush size : 64
    cache_alignment : 128
    address sizes : 36 bits physical, 48 bits virtual
    power management:

    processor : 1
    vendor_id : GenuineIntel
    cpu family : 15
    model : 4
    model name : Intel(R) Xeon(TM) CPU 3.00GHz
    stepping : 3
    cpu MHz : 2992.505
    cache size : 2048 KB
    physical id : 3
    siblings : 2
    core id : 0
    cpu cores : 1
    apicid : 6
    initial apicid : 6
    fpu : yes
    fpu_exception : yes
    cpuid level : 5
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr
    bogomips : 5985.41
    clflush size : 64
    cache_alignment : 128
    address sizes : 36 bits physical, 48 bits virtual
    power management:

    processor : 2
    vendor_id : GenuineIntel
    cpu family : 15
    model : 4
    model name : Intel(R) Xeon(TM) CPU 3.00GHz
    stepping : 3
    cpu MHz : 2992.505
    cache size : 2048 KB
    physical id : 0
    siblings : 2
    core id : 0
    cpu cores : 1
    apicid : 1
    initial apicid : 1
    fpu : yes
    fpu_exception : yes
    cpuid level : 5
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr
    bogomips : 5985.38
    clflush size : 64
    cache_alignment : 128
    address sizes : 36 bits physical, 48 bits virtual
    power management:

    processor : 3
    vendor_id : GenuineIntel
    cpu family : 15
    model : 4
    model name : Intel(R) Xeon(TM) CPU 3.00GHz
    stepping : 3
    cpu MHz : 2992.505
    cache size : 2048 KB
    physical id : 3
    siblings : 2
    core id : 0
    cpu cores : 1
    apicid : 7
    initial apicid : 7
    fpu : yes
    fpu_exception : yes
    cpuid level : 5
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr
    bogomips : 5985.41
    clflush size : 64
    cache_alignment : 128
    address sizes : 36 bits physical, 48 bits virtual
    power management:
    Show
    processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) CPU 3.00GHz stepping : 3 cpu MHz : 2992.505 cache size : 2048 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr bogomips : 5985.01 clflush size : 64 cache_alignment : 128 address sizes : 36 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) CPU 3.00GHz stepping : 3 cpu MHz : 2992.505 cache size : 2048 KB physical id : 3 siblings : 2 core id : 0 cpu cores : 1 apicid : 6 initial apicid : 6 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr bogomips : 5985.41 clflush size : 64 cache_alignment : 128 address sizes : 36 bits physical, 48 bits virtual power management: processor : 2 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) CPU 3.00GHz stepping : 3 cpu MHz : 2992.505 cache size : 2048 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr bogomips : 5985.38 clflush size : 64 cache_alignment : 128 address sizes : 36 bits physical, 48 bits virtual power management: processor : 3 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) CPU 3.00GHz stepping : 3 cpu MHz : 2992.505 cache size : 2048 KB physical id : 3 siblings : 2 core id : 0 cpu cores : 1 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr bogomips : 5985.41 clflush size : 64 cache_alignment : 128 address sizes : 36 bits physical, 48 bits virtual power management:
  • GCC Version:
    gcc version 4.4.1 [gcc-4_4-branch revision 150839] (SUSE Linux)
  • FreeSWITCH GIT Revision:
    FreeSWITCH Version 1.0.head (git-2f0105d 2010-12-11 16-53-52 -0600)
  • Reproduced with GIT HEAD?:
    yes
  • lsb_release:
    Hide
    LSB Version: n/a
    Distributor ID: SUSE LINUX
    Description: openSUSE 11.2 (x86_64)
    Release: 11.2
    Codename: n/a
    Show
    LSB Version: n/a Distributor ID: SUSE LINUX Description: openSUSE 11.2 (x86_64) Release: 11.2 Codename: n/a

Description

Freeswitch crashes without dumping core when specific conditions are met:

1. <action application="bridge" data="${lcr_auto_route}"/>
2. Multiple carriers in lcr_auto_route
3. number dialed fails with multiple carriers
4. carrier sends early media then sends USER_BUSY or NORMAL_TEMPORARY_FAILURE
5. sofia tries to bridge next carrier and then crashes without core dump.


In the two log snippets below you can se things failing over until
Sending Early Media
Pre-Answer sofia....
Then when that call gets an error, it is destroyed, and sofia creates the new channel.
Then boom. down. that's the last entry in the log and no core file is produced.

=======
2010-12-11 15:54:08.462281 [NOTICE] switch_channel.c:779 New Channel sofia/external/0114721308540 [d2593cf4-1f4e-40fc-b8b0-bf832cb26ef0]
2010-12-11 15:54:12.306475 [INFO] sofia.c:709 sofia/external/0114721308540 Update Callee ID to "0114721308540" <0114721308540>
2010-12-11 15:54:12.309488 [NOTICE] sofia_glue.c:3470 Pre-Answer sofia/external/0114721308540!
2010-12-11 15:54:12.310467 [INFO] switch_ivr_originate.c:1079 Sending early media
2010-12-11 15:54:12.312538 [NOTICE] mod_sofia.c:2176 Pre-Answer sofia/internal/19543788324@customer!
2010-12-11 15:54:14.203334 [NOTICE] sofia.c:5141 Hangup sofia/external/0114721308540 [CS_CONSUME_MEDIA] [USER_BUSY]
2010-12-11 15:54:14.210369 [NOTICE] switch_core_session.c:1228 Session 965 (sofia/external/0114721308540) Ended
2010-12-11 15:54:14.210369 [NOTICE] switch_core_session.c:1230 Close Channel sofia/external/0114721308540 [CS_DESTROY]
2010-12-11 15:54:14.211491 [NOTICE] switch_channel.c:779 New Channel sofia/external/4721308540 [6095c846-cf36-4807-aca9-0aac548dbc39]
=======

=======
2010-12-13 19:19:26.532573 [NOTICE] switch_channel.c:784 New Channel sofia/external/011442074999000 [c8e48efc-cded-4bd7-8ce4-c55585f0957d]
2010-12-13 19:19:30.085883 [NOTICE] sofia.c:5253 Hangup sofia/external/011442074999000 [CS_CONSUME_MEDIA] [NORMAL_TEMPORARY_FAILURE]
2010-12-13 19:19:30.087836 [NOTICE] switch_channel.c:784 New Channel sofia/external/011442074999000 [14735252-75ae-478c-82c9-90ccbedb6e39]
2010-12-13 19:19:30.089828 [NOTICE] switch_core_session.c:1273 Session 117 (sofia/external/011442074999000) Ended
2010-12-13 19:19:30.089828 [NOTICE] switch_core_session.c:1275 Close Channel sofia/external/011442074999000 [CS_DESTROY]
2010-12-13 19:19:30.224733 [NOTICE] sofia.c:5253 Hangup sofia/external/011442074999000 [CS_CONSUME_MEDIA] [NORMAL_TEMPORARY_FAILURE]
2010-12-13 19:19:30.225759 [NOTICE] switch_core_session.c:1273 Session 118 (sofia/external/011442074999000) Ended
2010-12-13 19:19:30.225759 [NOTICE] switch_core_session.c:1275 Close Channel sofia/external/011442074999000 [CS_DESTROY]
2010-12-13 19:19:30.226857 [NOTICE] switch_channel.c:784 New Channel sofia/external/442074999000 [ff6ec003-5930-46d5-b451-30e01eb0a510]
2010-12-13 19:19:41.288466 [INFO] sofia.c:720 sofia/external/442074999000 Update Callee ID to "442074999000" <442074999000>
2010-12-13 19:19:41.291564 [NOTICE] sofia_glue.c:3630 Pre-Answer sofia/external/442074999000!
2010-12-13 19:19:41.292570 [INFO] switch_ivr_originate.c:1094 Sending early media
2010-12-13 19:19:41.294594 [NOTICE] mod_sofia.c:2218 Pre-Answer sofia/internal/18005551212@customer!
2010-12-13 19:19:48.089401 [NOTICE] sofia.c:5253 Hangup sofia/external/442074999000 [CS_CONSUME_MEDIA] [NORMAL_TEMPORARY_FAILURE]
2010-12-13 19:19:48.101387 [NOTICE] switch_core_session.c:1273 Session 119 (sofia/external/442074999000) Ended
2010-12-13 19:19:48.101387 [NOTICE] switch_core_session.c:1275 Close Channel sofia/external/442074999000 [CS_DESTROY]
2010-12-13 19:19:48.101387 [NOTICE] switch_channel.c:784 New Channel sofia/external/442074999000 [26c8e15f-ec37-4a8a-b1de-0e1563e46c5b]
=======


Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Anthony Minessale II added a comment -
FYI your build is 4 days older than your post so the answer is no not reproduced on GIT HEAD.
run "make current" and retest please

Does it log a crash in dmesg?

are you running FS as a non-root user, do you not have priv to write the core file?
is the dumping of cores disabled by the OS at some level?

sysctl -a | grep kernel.core

run FS in the foreground so you can see what happens when you reproduce.

Show
Anthony Minessale II added a comment - FYI your build is 4 days older than your post so the answer is no not reproduced on GIT HEAD. run "make current" and retest please Does it log a crash in dmesg? are you running FS as a non-root user, do you not have priv to write the core file? is the dumping of cores disabled by the OS at some level? sysctl -a | grep kernel.core run FS in the foreground so you can see what happens when you reproduce.
Hide
Permalink
Bill Weidman added a comment -
There is no crash in dmesg.

I am running freeswitch as non-root user. I tested that the freeswitch user can produce cores by running 'top' as that user then doing a kill -11. It does produce a core dump, in the current directory.

I just did this to make sure the core file goes to a specific place:
echo "/tmp/core.%p" > /proc/sys/kernel/core_pattern

Old:
sysctl -a | grep kernel.core
kernel.core_uses_pid = 1
kernel.core_pattern = core

New:
kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core.%p

Also, FYI,
I set this in the dialplan before the bridge command
   <action application="set" data="ignore_early_media=ring_ready"/>
and FS hasn't crashed since.

I'll update to current and test some more


Show
Bill Weidman added a comment - There is no crash in dmesg. I am running freeswitch as non-root user. I tested that the freeswitch user can produce cores by running 'top' as that user then doing a kill -11. It does produce a core dump, in the current directory. I just did this to make sure the core file goes to a specific place: echo "/tmp/core.%p" > /proc/sys/kernel/core_pattern Old: sysctl -a | grep kernel.core kernel.core_uses_pid = 1 kernel.core_pattern = core New: kernel.core_uses_pid = 1 kernel.core_pattern = /tmp/core.%p Also, FYI, I set this in the dialplan before the bridge command    <action application="set" data="ignore_early_media=ring_ready"/> and FS hasn't crashed since. I'll update to current and test some more
Hide
Permalink
Bill Weidman added a comment -
Okay I think the reason freeswitch isn't dumping core is becuase the freeswitch init script for suse uses startproc, and it isn't accepting the ulimit settings.

At any rate, that's a separate problem. I was able to produce a core dump.

http://pastebin.freeswitch.org/14797
Show
Bill Weidman added a comment - Okay I think the reason freeswitch isn't dumping core is becuase the freeswitch init script for suse uses startproc, and it isn't accepting the ulimit settings. At any rate, that's a separate problem. I was able to produce a core dump. http://pastebin.freeswitch.org/14797
Hide
Permalink
Anthony Minessale II added a comment -
commit 51e1de39f2d7071e7ee8ad55c0574e2908744b26
Author: Anthony Minessale <anthm@freeswitch.org>
Date: Wed Dec 15 20:01:34 2010 -0600
Show
Anthony Minessale II added a comment - commit 51e1de39f2d7071e7ee8ad55c0574e2908744b26 Author: Anthony Minessale < anthm@freeswitch.org > Date: Wed Dec 15 20:01:34 2010 -0600

People

  • Assignee:
    Anthony Minessale II
    Reporter:
    Bill Weidman
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: