FreeSWITCH

Protection form DoS attack

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Not A Bug
  • Affects Version/s: 1.0.7
  • Fix Version/s: None
  • Component/s: freeswitch-core
  • Labels:
    None
  • Environment:
    Windows Server 2003
  • Platform:
    Win32/VS
  • CPU Info:
    PIII
  • FreeSWITCH GIT Revision:
    4630e3b
  • Reproduced with GIT HEAD?:
    4630e3b

Description

FreeSWITCH is very vulnerable for DoS attacks.

Today leaving my home for a scheduled meeting I've noticed unusual network activity linked to my server, running FreeSWITCH. I was lucky - I've noticed that. The bad news is - I was too late for the meeting... I tried to investigate what's going on and it took 45 min to fix the problem. Why it took so long? Server was unresponsive and close to complete halt, swapping memory to and from HD... Reason? FS was in huge memory leak mode, requesting more and more memory form the system. Processor was very busy too. Finally I killed FS (it did not respond to the common '...' exit request from console).

What happened? A rogue host from the Internet was sending a flood of UDP packets to the server via opened 5060 port. And FS was endlessly allocating memory... From FS log and/or its console there was no indication which external host (no IP whatsoever) was doing that. There was no indication from FS, that it making the huge allocation of memory or even any attempts to stop it. If I did not block access to it from the Internet, server would crash...

Unfortunately I've seen similar behavior from FreeSWITCH more then once. And I have to say, this is completely unacceptable behavior, taking into consideration, that FS currently runs on a server, that serves other services too (web, mail, etc) and I can't afford to loose all of those services for a long time because FS is wide open to DoS.

What I expect form the FS:
1. Never endlessly allocate the memory for serving requests coming from a host, when there is a clear indication that it's attacking FS (it doesn't care about responses, it's just sending a flood of requests, tens or may be hundreds of packets per sec)

2. Indicate the IP address of attacker (in console and/or in log). It will help person, who tires to stop the DoS attack to see which IP should be blocked immediately (with firewall or with other means). I had problems to find out IP of attacker because there is no tool in Windows, that will show IP of arriving packets and Wireshark simply crashed, when I tried to launch it...

3. FS should monitor (routine time to time check) its memory allocation and stops accepting new requests, when some specified limit is reached (until memory is de-allocated, or FS is reconfigured to take more). It will at least protect the server (running other important services) from a complete halt.

With FreeSWITCH I have experienced two types of attacks:
1. Logical attacks, when attacker tries to register, to make calls or do other tasks, which make some sense. In 2011-11-11 I've opened ticket, looking for protection from such attacks (http://jira.freeswitch.org/browse/FS-3588). It still unresolved.
2. DoS attacks, when attacker doesn't care about any SIP logic. It simply sends a flood of UDP packets to crash server. This request is exactly about such scenario.

The problem again - big flood of UDP requests (I don't know content of any of them) makes FS to endlessly allocate system memory without any limits. Packets are sent via purposely opened SIP port 5060. All packets are sent by attacker without checking any replies form FS (attack continued after host was blocked with firewall). In other instances the IP's of the attackers were different. By now all of them are blocked with router, but it's just a matter of time when other will try to do the same. Please help to stop it.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Anthony Minessale II added a comment -
You're kidding right?

So you just have a list of demands and present them as if you are blaming on us that you have an inadequate security measure in place.


Patches welcome.......
Show
Anthony Minessale II added a comment - You're kidding right? So you just have a list of demands and present them as if you are blaming on us that you have an inadequate security measure in place. Patches welcome.......
Hide
Permalink
Yuri Ost added a comment -
No, I'm not. Believe me, its not a fun to discover, that FreeSWITCH allocates all memory from system, making server completely unresponsive for other tasks, and eventually will crash it...

Checking IP address, where requests are coming from, and denying accepting further requests, in case if there is a hundred or may be a thousand requests from that host already in place, will certainly prevent FreeSWITCH from halting the whole server. Logging such event (indicating the IP) will help administrator to prevent it in future.
Show
Yuri Ost added a comment - No, I'm not. Believe me, its not a fun to discover, that FreeSWITCH allocates all memory from system, making server completely unresponsive for other tasks, and eventually will crash it... Checking IP address, where requests are coming from, and denying accepting further requests, in case if there is a hundred or may be a thousand requests from that host already in place, will certainly prevent FreeSWITCH from halting the whole server. Logging such event (indicating the IP) will help administrator to prevent it in future.
Hide
Permalink
Brian West added a comment -
This type of feature is outside the scope of FreeSWITCH, You should be using the appropriate tools provide by your OS to lessen this. You could investigate using Linux or trying to see if Windows has something similar to fail2ban that would help track and block these types of attacks. You're on the public internet expect these types of things and plan accordingly.

/b
Show
Brian West added a comment - This type of feature is outside the scope of FreeSWITCH, You should be using the appropriate tools provide by your OS to lessen this. You could investigate using Linux or trying to see if Windows has something similar to fail2ban that would help track and block these types of attacks. You're on the public internet expect these types of things and plan accordingly. /b
Hide
Permalink
Michael Collins added a comment -
"What happened? A rogue host from the Internet was sending a flood of UDP packets to the server via opened 5060 port."

If you are letting this flood get to your server then you've already lost the battle. This is like leaving your house unlocked, then calling up Sony and complaining that your big screen TV got stolen because Sony doesn't include anti-theft security features in its TV's. Like Brian said, the Internet is a dangerous place. However, there are security devices explicitly designed to mitigate or prevent these kinds of attacks. Furthermore, using Windows has its advantages and disadvantages. You've obviously found one of those disadvantages with this scenario. At least in Linux there is fail2ban and similar tools. Perhaps a Windows security expert could suggest something similar for your configuration?

-MC
Show
Michael Collins added a comment - "What happened? A rogue host from the Internet was sending a flood of UDP packets to the server via opened 5060 port." If you are letting this flood get to your server then you've already lost the battle. This is like leaving your house unlocked, then calling up Sony and complaining that your big screen TV got stolen because Sony doesn't include anti-theft security features in its TV's. Like Brian said, the Internet is a dangerous place. However, there are security devices explicitly designed to mitigate or prevent these kinds of attacks. Furthermore, using Windows has its advantages and disadvantages. You've obviously found one of those disadvantages with this scenario. At least in Linux there is fail2ban and similar tools. Perhaps a Windows security expert could suggest something similar for your configuration? -MC
Hide
Permalink
Ken Rice added a comment -
This is not a bug but a problem with lack of proper (D)DoS mitigation in place. Please see the other comment
Show
Ken Rice added a comment - This is not a bug but a problem with lack of proper (D)DoS mitigation in place. Please see the other comment
Hide
Permalink
Yuri Ost added a comment -
Closing the ticket converts FreeSWICTH's memory leak from the bug to a "feature". That means, that FreeSWITCH could not be installed in a production environment without jeopardizing the whole server... It's easy to deprive of memory resources and eventually halt the server by simply sending a stream of requests to FS. Therefore, FS couldn't be trusted and left unattended for a while... :(

Now, with regards to fail2ban:
1. there is no such tool in Windows Server
2. default log, generated by FreeSWITCH at that time, contains nothing about attacker. It means that fail2ban would be useless in this scenario

What log options do you use in FS to show/recognize IP of attacker?

BTW, regarding to fail2ban as a panacea for DoS attacks, here is related article: "Why Fail2Ban Can Fail With VoIP"
http://infosecisland.com/blogview/11384-Why-Fail2Ban-Can-Fail-With-VoIP.html
Show
Yuri Ost added a comment - Closing the ticket converts FreeSWICTH's memory leak from the bug to a "feature". That means, that FreeSWITCH could not be installed in a production environment without jeopardizing the whole server... It's easy to deprive of memory resources and eventually halt the server by simply sending a stream of requests to FS. Therefore, FS couldn't be trusted and left unattended for a while... :( Now, with regards to fail2ban: 1. there is no such tool in Windows Server 2. default log, generated by FreeSWITCH at that time, contains nothing about attacker. It means that fail2ban would be useless in this scenario What log options do you use in FS to show/recognize IP of attacker? BTW, regarding to fail2ban as a panacea for DoS attacks, here is related article: "Why Fail2Ban Can Fail With VoIP" http://infosecisland.com/blogview/11384-Why-Fail2Ban-Can-Fail-With-VoIP.html
Hide
Permalink
Anthony Minessale II added a comment -
You are not proposing a solution, you are arrogantly demanding a fix to a problem that is easier to address on your operating system. Each bug report you open is very rude and demanding and this time you are even blaming us for your own misfortune. Most servers can be DoS by an endless supply of reasons will you go and send a 10 page complaint letter to each of them? I think you need to learn how to communicate in a fashion that does not put people instantly on the defense if you expect to make any progress with them.......

The most important thing is the attack is very low level and triggering code in the sofia-sip stack. The problem is happening before we even get notice up in FreeSWITCH. The best thing to do is to watch for attack patterns at the router or OS level. The software will always be under durest if you let it make it that far. The ideal system would block the traffic at the lowest level possible (kernel) and its out of the scope of FS to perform such an operation.

You should consider making a new app with libpcap that can independently watch for attack patterns and send os-specific block commands to the kernel.
Show
Anthony Minessale II added a comment - You are not proposing a solution, you are arrogantly demanding a fix to a problem that is easier to address on your operating system. Each bug report you open is very rude and demanding and this time you are even blaming us for your own misfortune. Most servers can be DoS by an endless supply of reasons will you go and send a 10 page complaint letter to each of them? I think you need to learn how to communicate in a fashion that does not put people instantly on the defense if you expect to make any progress with them....... The most important thing is the attack is very low level and triggering code in the sofia-sip stack. The problem is happening before we even get notice up in FreeSWITCH. The best thing to do is to watch for attack patterns at the router or OS level. The software will always be under durest if you let it make it that far. The ideal system would block the traffic at the lowest level possible (kernel) and its out of the scope of FS to perform such an operation. You should consider making a new app with libpcap that can independently watch for attack patterns and send os-specific block commands to the kernel.
Hide
Permalink
Yuri Ost added a comment -
I think I've already expressed proposed solution in my initial post. Currently FS allocates memory from system with every incoming request and without any limits. In a while it leads to potential possibility to deplete server from all its memory and eventually to crash it. Proposed solution was in calling a function, that checks how many active requests FS already has from that particular IP. If returned number is bigger, than some threshold, do nothing (print alert in log, do it once). Otherwise proceed with request as usual. It will stop FS from allocating unnecessary system memory resources and indicate to administrator, that FS was under attack from a particular IP.
Show
Yuri Ost added a comment - I think I've already expressed proposed solution in my initial post. Currently FS allocates memory from system with every incoming request and without any limits. In a while it leads to potential possibility to deplete server from all its memory and eventually to crash it. Proposed solution was in calling a function, that checks how many active requests FS already has from that particular IP. If returned number is bigger, than some threshold, do nothing (print alert in log, do it once). Otherwise proceed with request as usual. It will stop FS from allocating unnecessary system memory resources and indicate to administrator, that FS was under attack from a particular IP.
Hide
Permalink
Anthony Minessale II added a comment -
I am not going to continuously repeat myself. Just re-read my other posts.
Show
Anthony Minessale II added a comment - I am not going to continuously repeat myself. Just re-read my other posts.
Hide
Permalink
Umberto Mautone added a comment -
Yuri, there are tools you can use at the OS level that give you control in mitigating floods. For the time being, DDoS protection is outside the current scope of the FreeSWITCH project.

Just to give you an example, iptables can be used to mitigate such attacks as described here:

http://etel.wiki.oreilly.com/wiki/index.php/SIP_DoS/DDoS_Mitigation

There are more sophisticated tools and methods available but the example above is a good base from which to start. The added advantage of using OS-level tools is that they can be deployed at the network edge rather than the individual servers, which offloads the effort from the FreeSWITCH server(s).
Show
Umberto Mautone added a comment - Yuri, there are tools you can use at the OS level that give you control in mitigating floods. For the time being, DDoS protection is outside the current scope of the FreeSWITCH project. Just to give you an example, iptables can be used to mitigate such attacks as described here: http://etel.wiki.oreilly.com/wiki/index.php/SIP_DoS/DDoS_Mitigation There are more sophisticated tools and methods available but the example above is a good base from which to start. The added advantage of using OS-level tools is that they can be deployed at the network edge rather than the individual servers, which offloads the effort from the FreeSWITCH server(s).
Hide
Permalink
Umberto Mautone added a comment -
Here's a simple iptables rule to limit the rate of incoming SIP requests from all sources:

iptables -A INPUT -p udp --dport 5060 -m limit --limit 10/s --limit-burst 10 -i eth0 -j REJECT

This one rule will limit incoming SIP requests to 10 per second. Adjust the number to suit your needs.
Show
Umberto Mautone added a comment - Here's a simple iptables rule to limit the rate of incoming SIP requests from all sources: iptables -A INPUT -p udp --dport 5060 -m limit --limit 10/s --limit-burst 10 -i eth0 -j REJECT This one rule will limit incoming SIP requests to 10 per second. Adjust the number to suit your needs.
Hide
Permalink
Umberto Mautone added a comment -
More advice on FreeSWITCH's QoS Wiki:

http://wiki.freeswitch.org/wiki/QoS
Show
Umberto Mautone added a comment - More advice on FreeSWITCH's QoS Wiki: http://wiki.freeswitch.org/wiki/QoS
Hide
Permalink
Yuri Ost added a comment -
Umberto, thank you for your posts here and I appreciate your efforts to help. Unfortunately Windows Server OS doesn't have iptables. I may try to use ipsec policies (to filter incoming traffic), but they're very limited and can't be effectively used in such case.

It looks like the only solution to protect server OS from crashing could be a script/program, that will watch current memory allocation for FreeSWITCH process and then will kill / start it again in case, if FS takes more memory, than it's allotted... Obviously it will disrupt all VoIP communications for a while, but it's better, than to allow FreeSWICTH to halt the whole server (which is unacceptable, of course).

Is there any way to make FreeSWITCH to watch its own memory allocation and somehow limit it?
Show
Yuri Ost added a comment - Umberto, thank you for your posts here and I appreciate your efforts to help. Unfortunately Windows Server OS doesn't have iptables. I may try to use ipsec policies (to filter incoming traffic), but they're very limited and can't be effectively used in such case. It looks like the only solution to protect server OS from crashing could be a script/program, that will watch current memory allocation for FreeSWITCH process and then will kill / start it again in case, if FS takes more memory, than it's allotted... Obviously it will disrupt all VoIP communications for a while, but it's better, than to allow FreeSWICTH to halt the whole server (which is unacceptable, of course). Is there any way to make FreeSWITCH to watch its own memory allocation and somehow limit it?
Hide
Permalink
João Mesquita added a comment -
Yuri, howcome you don't have some kind of firewall that will do the task in front of that Windows Server? I don't know any sane windows server admin that would simply put a but naked windows server out to the world like that and expect everything to be in its place forever.

Even if the core team solves this particular issue you are having, I think that you need to rethink how you are deploying your servers because security threats like that are vast on the internet. Just my 2 cents.
Show
João Mesquita added a comment - Yuri, howcome you don't have some kind of firewall that will do the task in front of that Windows Server? I don't know any sane windows server admin that would simply put a but naked windows server out to the world like that and expect everything to be in its place forever. Even if the core team solves this particular issue you are having, I think that you need to rethink how you are deploying your servers because security threats like that are vast on the internet. Just my 2 cents.
Hide
Permalink
Yuri Ost added a comment -
João, of course I have firewall (please read me earlier posts where I mentioned about it). With firewall I can block any host/range of hosts from access to my server. It's not a problem, when I know about the attack. The problem is - at any time FreeSWITCH may suddenly start to allocate memory at big pace if a new (and unknown for me) host starts to send stream of malicious requests to FS. As you may guess, I can't predict when it'll happen and unless I watch network traffic on my router - I don't have indication of the attack.. It means, if I left FreeSWITCH unattended for a while (as it should be all the time, BTW), there is always a chance, that FreeSWITCH (by depleting it from memory resources) will bring the whole server down...
Show
Yuri Ost added a comment - João, of course I have firewall (please read me earlier posts where I mentioned about it). With firewall I can block any host/range of hosts from access to my server. It's not a problem, when I know about the attack. The problem is - at any time FreeSWITCH may suddenly start to allocate memory at big pace if a new (and unknown for me) host starts to send stream of malicious requests to FS. As you may guess, I can't predict when it'll happen and unless I watch network traffic on my router - I don't have indication of the attack.. It means, if I left FreeSWITCH unattended for a while (as it should be all the time, BTW), there is always a chance, that FreeSWITCH (by depleting it from memory resources) will bring the whole server down...
Hide
Permalink
Umberto Mautone added a comment -
Does your firewall allow you to set a rate limit on a particular port?
Show
Umberto Mautone added a comment - Does your firewall allow you to set a rate limit on a particular port?
Hide
Permalink
Jeff Lenk added a comment -
try something like this - Anti DDoS Guardian
Show
Jeff Lenk added a comment - try something like this - Anti DDoS Guardian
Hide
Permalink
Yuri Ost added a comment -
Umberto -- no, it does not support any rate limits. It would be helpful to have it though...

Jeff -- thank you for suggestion. I guess it could work. But it's pricey (one license costs $100). Do you use it?
Show
Yuri Ost added a comment - Umberto -- no, it does not support any rate limits. It would be helpful to have it though... Jeff -- thank you for suggestion. I guess it could work. But it's pricey (one license costs $100). Do you use it?

People

  • Assignee:
    Anthony Minessale II
    Reporter:
    Yuri Ost
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: