Just a popular stackoverflow question & answer which was deleted as not constructive even with 10+ score and thousands of views (by a mod, so can’t vote to revert - thanks!) Keep in mind that it’s from 2009 without later reviews.
Question:
I am searching for a tool that tests SIP calls. A platform that makes a call from SIP device A to SIP device B and reports results…
Any idea? A simulation platform would be ideal.
My answer:
There are many solutions. Some more broken than others. Here’s a quick summary of what I’ve found while looking for a base for a proper automated testing solution.
SIPp
It’s ok if you want only a single dialog at a time. What doesn’t work here is complex scenarios where you need to synchronise 2 call legs, do registration, call and presence in the same scenario. If you go this way, you’ll end up with running multiple sipp scenarios for each conversation element separately. Sipp also doesn’t scale at all for media transfers. Even though it’s multithreaded, something stops it from running concurrently - if you look at htop for example, you’ll see that sipp never crosses the 100% line. Around 50 media calls it starts to cut audio and take all CPU of the machine.
It can sometimes lose track of what’s happening, some packets which don’t even belong to the call really can result in a failed test. It’s got some silly bugs too, like case-sensitive comparing of the headers.
SIPr/sipper
Ruby-based solution where you have to write your own scenarios in Ruby. It’s got its own SIP stack and lots of tests. While it’s generally good and handles a lot of complex scenarios nicely, its design is terrible in my opinion. Bugs are hard to track down and after a week I had >10 patches that I needed just to make it do basic stuff. Later I learned that some of the scenarios should have been written in a slightly different way, but SIPr developers were not really responsive and it took a lot of time to find out about it. There was no good documentation either. Synchronising actions of many agents is a hard problem, since they’d rather use an event-based, but still single-threaded approach… it just makes you concentrate too much on “what order can this happen in and do I handle it correctly” and worrying about sync/async actions, rather than writing the actual test.
WinSIP
Commercial solution. Never tested it properly since the basic functionality is missing from the evaluation version and it’s hard to spend that much money on something you’re not sure works…
SipUnit
Java-based solution reusing Jain-SIP stack. It can do almost any scenario and is fairly good. It tries to make everything non-blocking / action based leading to a similar situation SIPr has, but in this case it’s trivial to make it parallel / threaded so it’s much easier to deal with. It has its own share of bugs, so not everything works well in the vanilla package, but most of the stuff is patchable. The developers seem to be busy with other projects, so it’s not updated for a long time. If you need transfers, presence, dialog-info, custom messages, RTP handling, etc. - you’ll have to write your own modifications to support them. It is not good for performance testing though - each session on the test client side is rather heavy compared to the typical server side processing. That means you’ll most likely need a number of hosts doing the testing per one server/proxy.
If you’re a Java-hater like me, it can be used in a simple way from Jython, JRuby or any other JVM language.
In the end, I chose SIPunit as the least broken/evil/unusable solution. It is by no means perfect, but… it works in most cases. If I was doing the project once again with all this knowledge, I’d probably reuse SIPp configurations and try to write my own solution that uses proper threading - but I guesstimate it’s at least a ½ year project for one person, to make it good enough for production.