FreeSWITCH

Segfault in switch_event after long run

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Cannot Reproduce
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: freeswitch-core
  • Labels:
    None
  • Environment:
    using freeswitch git version downloaded 4 days ago
    RHEL 6.2
    2.6.32-220.7.1.el6.x86_64 #1 SMP Fri Feb 10 15:22:22 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
  • Platform:
    Linux x86_64/gcc
  • CPU Info:
    AMD Opteron(tm) Processor 6176 SE 24-core
  • FreeSWITCH GIT Revision:
    April 25 HEAD
  • Reproduced with GIT HEAD?:
    yes, as of 4 days ago.

Description

During a mult-hour run averaging 250 simultaneous calls, got a segfault.
This is a production system making real calls, and this condition cannot be easily replicated.
Do not ask me to replicate with git HEAD.
gdb backtrace:

#0 0x0000003e4c47fb21 in __strlen_sse2 () from /lib64/libc.so.6
#1 0x00007f6eb9431686 in my_dup (s=0x3933323533312f65 <Address 0x3933323533312f65 out of bounds>) at src/switch_event.c:97
#2 0x00007f6eb94339a0 in switch_event_add_header_string (event=0x7f6bb046f800, stack=<value optimized out>, header_name=<value optimized out>, data=<value optimized out>) at src/switch_event.c:1137
#3 0x00007f6eb942414a in switch_scheduler_add_task (task_runtime=1335829448, func=<value optimized out>, desc=<value optimized out>, group=<value optimized out>, cmd_id=0, cmd_arg=0x7f6c2a5c7250, flags=2) at src/switch_scheduler.c:219
#4 0x00007f6eb944ed24 in switch_ivr_schedule_hangup (runtime=1335829448, uuid=0x7f6c291f24d3 "makecall_49_from_CTCLIENT1_on_warhel33.e-r-solutions.com_t_1335829363", cause=SWITCH_CAUSE_ALLOTTED_TIMEOUT, bleg=SWITCH_FALSE) at src/switch_ivr_async.c:3799
#5 0x00007f6eaac595cc in sched_hangup_function (cmd=<value optimized out>, session=<value optimized out>, stream=0x7f6e094edd40) at /var/tmp/freeswitch_apr_25_2012/src/mod/applications/mod_commands/mod_commands.c:2664
#6 0x00007f6eb9426294 in switch_api_execute (cmd=0x7f6de0063ee0 "sched_hangup", arg=0x7f6de0063ef0 "+0 makecall_49_from_CTCLIENT1_on_warhel33.e-r-solutions.com_t_1335829363", session=0x0, stream=0x7f6e094edd40) at src/switch_loadable_module.c:2261
#7 0x00007f6eabdf7aee in api_exec (thread=<value optimized out>, obj=0x7f6de0063db0) at /var/tmp/freeswitch_apr_25_2012/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:1445
#8 0x0000003e4cc077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e4c4e592d in clone () from /lib64/libc.so.6

More details availabe on request.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Anthony Minessale II added a comment -
switch to frame 3

frame 3

and

print *tp
Show
Anthony Minessale II added a comment - switch to frame 3 frame 3 and print *tp
Hide
Permalink
Doug Campbell added a comment -
(gdb) frame 3
#3 0x00007f6eb942414a in switch_scheduler_add_task (task_runtime=1335829448, func=<value optimized out>, desc=<value optimized out>, group=<value optimized out>, cmd_id=0, cmd_arg=0x7f6c2a5c7250, flags=2) at src/switch_scheduler.c:219
(gdb) print *tp
$1 = {task = {created = 140102508159216, runtime = 140100528079216, cmd_id = 775434298, group = 0x554245445b203937 <Address 0x554245445b203937 out of bounds>, cmd_arg = 0x6374697773205d47, task_id = 1868783464}, executed = 7146771083473154917, in_thread = 858796346, destroyed = 1699971127, func = 0x616e67697320646e, pool = 0x2f6169666f73206c, flags = 1852794992, desc = 0x3933323533312f65 <Address 0x3933323533312f65 out of bounds>, next = 0x70}
(gdb)
Show
Doug Campbell added a comment - (gdb) frame 3 #3 0x00007f6eb942414a in switch_scheduler_add_task (task_runtime=1335829448, func=<value optimized out>, desc=<value optimized out>, group=<value optimized out>, cmd_id=0, cmd_arg=0x7f6c2a5c7250, flags=2) at src/switch_scheduler.c:219 (gdb) print *tp $1 = {task = {created = 140102508159216, runtime = 140100528079216, cmd_id = 775434298, group = 0x554245445b203937 <Address 0x554245445b203937 out of bounds>, cmd_arg = 0x6374697773205d47, task_id = 1868783464}, executed = 7146771083473154917, in_thread = 858796346, destroyed = 1699971127, func = 0x616e67697320646e, pool = 0x2f6169666f73206c, flags = 1852794992, desc = 0x3933323533312f65 <Address 0x3933323533312f65 out of bounds>, next = 0x70} (gdb)
Hide
Permalink
Anthony Minessale II added a comment -
I'll have to check this tomorrow, there is some memory corruption here.
Is sched_hangup something you use a lot, any other things you do a lot here so i can investigate in the morning?
Show
Anthony Minessale II added a comment - I'll have to check this tomorrow, there is some memory corruption here. Is sched_hangup something you use a lot, any other things you do a lot here so i can investigate in the morning?
Hide
Permalink
Doug Campbell added a comment -
The only things we do (but we do them a lot) are:
  via Event Socket In:
    - bgapi originate
    - bgapi uuid_bridge
    - bgapi sched_hangup
  via Event Socket out (all via sendmsg call-command: execute):
    - answer
    - playback
    - break

Show
Doug Campbell added a comment - The only things we do (but we do them a lot) are:   via Event Socket In:     - bgapi originate     - bgapi uuid_bridge     - bgapi sched_hangup   via Event Socket out (all via sendmsg call-command: execute):     - answer     - playback     - break
Hide
Permalink
Anthony Minessale II added a comment -
I missed the declaration "do not ask me to reproduce with GIT HEAD" Kind of rude, no?
Show
Anthony Minessale II added a comment - I missed the declaration "do not ask me to reproduce with GIT HEAD" Kind of rude, no?
Hide
Permalink
Doug Campbell added a comment -
I agree it was a poor choice of wording.
I meant to convey that it is not easy to reproduce.
I did not mean to convey that I am not willing to help; I am willing to help if I can.
Show
Doug Campbell added a comment - I agree it was a poor choice of wording. I meant to convey that it is not easy to reproduce. I did not mean to convey that I am not willing to help; I am willing to help if I can.
Hide
Permalink
Doug Campbell added a comment -
A very similar crash just happened again.
The system had run with about 500 simultaneous calls for about 10 minutes.
Is there any hope of making progress on this issue?

The gdb backtrace:

(gdb) bt
#0 0x0000003e4c47fb21 in __strlen_sse2 () from /lib64/libc.so.6
#1 0x00007fe80a316686 in my_dup (s=0x6e00746e756f635f <Address 0x6e00746e756f635f out of bounds>) at src/switch_event.c:97
#2 0x00007fe80a3189a0 in switch_event_add_header_string (event=0x7fe4f7bbcae0, stack=<value optimized out>, header_name=<value optimized out>, data=<value optimized out>) at src/switch_event.c:1137
#3 0x00007fe80a30914a in switch_scheduler_add_task (task_runtime=1336145045, func=<value optimized out>, desc=<value optimized out>, group=<value optimized out>, cmd_id=0, cmd_arg=0x7fe68e4362a0, flags=2) at src/switch_scheduler.c:219
#4 0x00007fe80a333d24 in switch_ivr_schedule_hangup (runtime=1336145045, uuid=0x7fe68cf99513 "makecall_147_from_CTCLIENT1_on_warhel33.e-r-solutions.com_t_1336145003", cause=SWITCH_CAUSE_ALLOTTED_TIMEOUT, bleg=SWITCH_FALSE) at src/switch_ivr_async.c:3799
#5 0x00007fe773b385cc in sched_hangup_function (cmd=<value optimized out>, session=<value optimized out>, stream=0x7fe4f35aed40) at /var/tmp/freeswitch_apr_25_2012/src/mod/applications/mod_commands/mod_commands.c:2664
#6 0x00007fe80a30b294 in switch_api_execute (cmd=0x7fe7282cdd10 "sched_hangup", arg=0x7fe7282cdd20 "+0 makecall_147_from_CTCLIENT1_on_warhel33.e-r-solutions.com_t_1336145003", session=0x0, stream=0x7fe4f35aed40) at src/switch_loadable_module.c:2261
#7 0x00007fe808e37aee in api_exec (thread=<value optimized out>, obj=0x7fe7282cdbe0) at /var/tmp/freeswitch_apr_25_2012/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:1445
#8 0x0000003e4cc077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e4c4e592d in clone () from /lib64/libc.so.6
(gdb)
Show
Doug Campbell added a comment - A very similar crash just happened again. The system had run with about 500 simultaneous calls for about 10 minutes. Is there any hope of making progress on this issue? The gdb backtrace: (gdb) bt #0 0x0000003e4c47fb21 in __strlen_sse2 () from /lib64/libc.so.6 #1 0x00007fe80a316686 in my_dup (s=0x6e00746e756f635f <Address 0x6e00746e756f635f out of bounds>) at src/switch_event.c:97 #2 0x00007fe80a3189a0 in switch_event_add_header_string (event=0x7fe4f7bbcae0, stack=<value optimized out>, header_name=<value optimized out>, data=<value optimized out>) at src/switch_event.c:1137 #3 0x00007fe80a30914a in switch_scheduler_add_task (task_runtime=1336145045, func=<value optimized out>, desc=<value optimized out>, group=<value optimized out>, cmd_id=0, cmd_arg=0x7fe68e4362a0, flags=2) at src/switch_scheduler.c:219 #4 0x00007fe80a333d24 in switch_ivr_schedule_hangup (runtime=1336145045, uuid=0x7fe68cf99513 "makecall_147_from_CTCLIENT1_on_warhel33.e-r-solutions.com_t_1336145003", cause=SWITCH_CAUSE_ALLOTTED_TIMEOUT, bleg=SWITCH_FALSE) at src/switch_ivr_async.c:3799 #5 0x00007fe773b385cc in sched_hangup_function (cmd=<value optimized out>, session=<value optimized out>, stream=0x7fe4f35aed40) at /var/tmp/freeswitch_apr_25_2012/src/mod/applications/mod_commands/mod_commands.c:2664 #6 0x00007fe80a30b294 in switch_api_execute (cmd=0x7fe7282cdd10 "sched_hangup", arg=0x7fe7282cdd20 "+0 makecall_147_from_CTCLIENT1_on_warhel33.e-r-solutions.com_t_1336145003", session=0x0, stream=0x7fe4f35aed40) at src/switch_loadable_module.c:2261 #7 0x00007fe808e37aee in api_exec (thread=<value optimized out>, obj=0x7fe7282cdbe0) at /var/tmp/freeswitch_apr_25_2012/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:1445 #8 0x0000003e4cc077f1 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003e4c4e592d in clone () from /lib64/libc.so.6 (gdb)
Hide
Permalink
Anthony Minessale II added a comment -
Well I have been looking at this issue but I have yet to reproduce it. I would say there is "hope" but i'd put you at 0 for 2 on tact when it comes to wording.

Are you always using sched_hangup with +0 maybe that's what I'm missing....

Can you explain how you are using it so I can trying to find the bad code path you seem to be landing on.

BTW, we offer commercial support contracts where issues like this are covered and have priority (See consulting@freeswitch.org or # on homepage)

Show
Anthony Minessale II added a comment - Well I have been looking at this issue but I have yet to reproduce it. I would say there is "hope" but i'd put you at 0 for 2 on tact when it comes to wording. Are you always using sched_hangup with +0 maybe that's what I'm missing.... Can you explain how you are using it so I can trying to find the bad code path you seem to be landing on. BTW, we offer commercial support contracts where issues like this are covered and have priority (See consulting@freeswitch.org or # on homepage)
Hide
Permalink
Anthony Minessale II added a comment -
WIth the data I can get from this trace I tried to lab up a script doing a constant flow of originates with origination_uuid to very similar long custom uuid strings.
The script was running in an endless loop doing bgapi originate then a sched_hangup +0 on that uuid.

The script was generating a steady flow of 140 calls per second and I could not replicate any crash.
Subsequently, I added a patch to make sched_hangup +0 do the same thing as uuid_kill <uuid> [<cause>]

Can you do a clean build "make current" to rule out build skew and see if you can replicate? It should at least do something different with the new version.


Show
Anthony Minessale II added a comment - WIth the data I can get from this trace I tried to lab up a script doing a constant flow of originates with origination_uuid to very similar long custom uuid strings. The script was running in an endless loop doing bgapi originate then a sched_hangup +0 on that uuid. The script was generating a steady flow of 140 calls per second and I could not replicate any crash. Subsequently, I added a patch to make sched_hangup +0 do the same thing as uuid_kill <uuid> [<cause>] Can you do a clean build "make current" to rule out build skew and see if you can replicate? It should at least do something different with the new version.
Hide
Permalink
Doug Campbell added a comment -
First, let me try to make amends for my previous tact failures. I am extremely grateful for both the freeswitch project and for the wonderful support that accompanies it. I am a member of a team that codes and provides support for a free software project myself, and I understand at least to a degree how challenging it can be to provide software that meets as many different needs as there are users. The reasons that we are trying to use freeswitch for a demanding telephony application are that we appreciate its good design and quality implementation, and we believe it is up to the task.

I am also extremely grateful for your assistance on this particular issue. I believe there is nothing harder to track down than a very occasional fault that cannot be intentionally reproduced. Your test run and your code change are excellent steps toward a discovery and hence resolution of the problem. I will at once get the new version and install it. I will also try to be aware of any particular conditions present should it happen again.

Humbly and gratefully,
Doug
Show
Doug Campbell added a comment - First, let me try to make amends for my previous tact failures. I am extremely grateful for both the freeswitch project and for the wonderful support that accompanies it. I am a member of a team that codes and provides support for a free software project myself, and I understand at least to a degree how challenging it can be to provide software that meets as many different needs as there are users. The reasons that we are trying to use freeswitch for a demanding telephony application are that we appreciate its good design and quality implementation, and we believe it is up to the task. I am also extremely grateful for your assistance on this particular issue. I believe there is nothing harder to track down than a very occasional fault that cannot be intentionally reproduced. Your test run and your code change are excellent steps toward a discovery and hence resolution of the problem. I will at once get the new version and install it. I will also try to be aware of any particular conditions present should it happen again. Humbly and gratefully, Doug
Hide
Permalink
Jeff Lenk added a comment -
Any update here? Please close if no longer occurring.
Show
Jeff Lenk added a comment - Any update here? Please close if no longer occurring.
Hide
Permalink
Doug Campbell added a comment -
Since running 1.2rc2, I have not seen this issue.
Much thanks to the freeswitch team.
Show
Doug Campbell added a comment - Since running 1.2rc2, I have not seen this issue. Much thanks to the freeswitch team.

People

  • Assignee:
    Anthony Minessale II
    Reporter:
    Doug Campbell
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: