Astrad Version 12

André Guimarães, 2012-03-28

Switzernet

 

This Astrad version requires version 7 of DBA. Incompatible with Porta-SIP: it rejects all incoming calls from a Porta-SIP server due to authentication issues. Porta-SIP should be modified to send a Remote-Party-ID header with the caller-id and replace the URI user to “nodes”.

This version was renamed from Astrad 11 to 12 and a new version 11 was released.

 

Changes and new functionalities

New Asterisk version

Instead of using the Asterisk version provided in Debian Etch, a new version (1.8.7.1) was compiled with all available modules. Two packages were created: one with the modules we currently use and other with the extra modules that we may possible use. This new version as several security fixes, stability corrections and additional features.

 

It provides User Agent information for Billing when a customer’s phone registers.

 

Asterisk default sounds in the following languages were added: English, French, German, Italian, Spanish and Portuguese.

 

All sounds used for errors were added in French and German. All previous missing sound files are now present.

 

The new version also provides music on hold if a customer puts the call on hold or is put on hold.

 

The procedure for compiling asterisk with all modules, sounds and music on hold was the following:

 

apt-get install --assume-yes  build-essential ncurses-dev libncurses-dev libnewt-dev libz-dev  libssl-dev  libxml2-dev bison uw-mailutils libc-client2007b-dev

cd /usr/src

wget -c http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-1.8-current.tar.gz

tar xvzf asterisk-1.8-current.tar.gz

# To unblock maximum options in Asterisk

pushd /usr/src

cd asterisk-1.8.*

./contrib/scripts/install_prereq install

./contrib/scripts/install_prereq install-unpackaged

popd

# Resource Modules > res_calendar_caldav res_calendar_exchange res_calendar_icalendar

pushd /usr/src

wget -O libical-0.44.tar.gz http://sourceforge.net/projects/freeassociation/files/libical/libical-0.44/libical-0.44.tar.gz/download

tar zxvf libical-0.44.tar.gz

cd libical-0.44

./configure --enable-shared && make && make install

popd

# Resource Modules > res_calendar_ews

pushd /usr/src

wget -c http://www.webdav.org/neon/neon-0.29.5.tar.gz

tar xvzf neon-0.29.5.tar.gz

cd neon-0.29.5

./configure --enable-shared --with-ssl && make && make install

popd

# app_osplookup

pushd /usr/src

wget -O OSPToolkit-3.6.1.tar.gz http://sourceforge.net/projects/osp-toolkit/files/osp-toolkit/OSPToolkit%203.6.1/OSPToolkit-3.6.1.tar.gz/download

tar xvzf OSPToolkit-3.6.1.tar.gz

cd TK-3_6_1-20100107/src

make clean && make build && make install

# to add support to compilation of this module compile with: ./configure --with-osptk=/usr/local/include/osp

popd

#Asterisk compilation:

pushd /usr/src

cd asterisk-1.8.*

contrib/scripts/get_mp3_source.sh

contrib/scripts/get_ilbc_source.sh

./configure --without-srtp

#Select all modules

make menuselect

make

checkinstall --requires "lua5.1,popt,wget,libxml,libogg,libvorbis,curl,iksemel,spandsp,speex,iodbc,freetds-0.63p1-msdblib,mysql-client,gmime,sqlite,sqlite3,jack,libxml2,openssl,bison,uw-mailutils" --maintainer "admin@switzernet.com"checkinstall --requires "lua5.1,popt,wget,libxml,libogg,libvorbis,curl,iksemel,spandsp,speex,iodbc,freetds-0.63p1-msdblib,mysql-client,gmime,sqlite,sqlite3,jack,libxml2,openssl,bison,uw-mailutils" --maintainer "admin@switzernet.com" --pkgversion="1:1.8.7.1"

#make install

make samples

make config

popd

cd /usr/src

mv  asterisk-1.8*/*.deb .

#Procedure to edit the package and add new files

mkdir asterisk_1.8.7.1-1_amd64

cd asterisk_1.8.7.1-1_amd64

ar -x ../asterisk*.deb

tar xvfz data.tar.gz

mkdir -p etc/asterisk/default-confs/

cp -a /etc/asterisk/default-confs/* etc/asterisk/

cp -a /etc/asterisk/default-confs etc/asterisk/

mkdir -p var/spool/asterisk/

cp -a /var/spool/asterisk/voicemail var/spool/asterisk/

chown asterisk.asterisk var/spool/asterisk -R

mkdir -p var/lib/asterisk/

cp -a  /var/lib/asterisk/moh /var/lib/asterisk/agi-bin /var/lib/asterisk/firmware /var/lib/asterisk/images /var/lib/asterisk/keys /var/lib/asterisk/phoneprov  /var/lib/asterisk/sounds /var/lib/asterisk/static-http var/lib/asterisk

pushd .

cd var/lib/asterisk/sounds/

wget http://switzernet.com/2/public/100719-asterisk-update/data/astrad-prompt/astrad-prompt_astradv10.tgz

wget http://sip.startel.pt/downloads/sounds/sons_asterisk.zip

tar xvfz *.tgz

ls -la

rm *.tgz

chown asterisk.asterisk . -R

chmod 755 *

chmod 644 ??/*.* -R

popd

pushd .

cd usr/lib/asterisk/modules/

wget http://asterisk.hosting.lv/bin/codec_g723-ast18-gcc4-glibc-x86_64-core2.so

wget http://asterisk.hosting.lv/bin/codec_g729-ast18-gcc4-glibc-x86_64-core2.so

mv codec_g723*.so codec_g723.so

mv codec_g729*.so codec_g729.so

chmod 755 *.*

popd

tar cvfz data.tar.gz etc usr var

rm etc usr var

mkdir DEBIAN

cd DEBIAN

tar xvfz ../control.tar.gz

rm ../control.tar.gz

cd /usr/src

dpkg --build asterisk_1.8.7.1-1_amd64

dpkg --purge asterisk

dpkg -i asterisk_1.8.7.1-1_amd64.deb

 

The packages were later separated in the following files using the procedure described above to edit Debian packages:

asterisk                            1:1.8.7.1-2                Asterisk 1.8.7.1

asterisk-moh                        1:1.8.7.1-1                Provides music on hold to Asterisk 1.8.7.1.

asterisk-sounds                     1:1.8.7.1-1                Music and sound files for Asterisk 1.8.7.1 in english.

asterisk-sounds-de                  1:1.8.7.1-1                Music and sound files for Asterisk 1.8.7.1 in german.

asterisk-sounds-es                  1:1.8.7.1-1                Music and sound files for Asterisk 1.8.7.1 in spanish.

asterisk-sounds-fr                  1:1.8.7.1-1                Music and sound files for Asterisk 1.8.7.1 in french.

asterisk-sounds-it                  1:1.8.7.1-1                Music and sound files for Asterisk 1.8.7.1 in italian.

asterisk-sounds-pt                  1:1.8.7.1-1                Music and sound files for Asterisk 1.8.7.1 in portuguese.

 

These packages have the same content and file structure except for the larger repeated sound files which where removed to allow installation from Puppet which has a bug that prevents large files from being transferred. In these cases it always times out.

Asterisk Configuration files

All asterisk configuration files were updated to the new asterisk version format. Major changes in each of the configuration files are described next:

/etc/asterisk/cdr_custom.conf

[download] [diff]

All fields are now quote safe

/etc/asterisk/extconfig.conf

[download] [diff]

Only the table sipdevices2 is used now for authentication instead of sippeers2 and sipusers. This change optimizes almost by half the content of the database and decreases the amount of data replicated.

/etc/asterisk/extensions.conf

[download] [diff]

Variable names and command format had to be changed to be compatible with the new version and deprecated commands were replaced.

 

Numbers beginning by ‘+’ or ‘#’ are now supported. A customer can call international numbers using + INT.PREFIX NUMBER or 00 INT.PREFIX NUMBER.

 

The new asterisk version has better support for playing messages for phones without answering the call. The restrictions to play the messages only to Linksys SPA921 were removed and now any phone can listen to those messages.

 

A new hang-up cause was added to play an error when a customer is not online. It now plays the message “Not found” instead of “An error as occurred”.

 

A problem that prevented some Linksys models (i.e. Linksys/SPA3102) from hearing the ring tone after hearing the “Free call” message and existed in previous versions was corrected.

 

Fail2ban can now ban IP addresses that try to make unauthorized calls. When an unauthorized phone tries to make a call it leaves an entry in a log which will be then processed by fail2ban.

/etc/asterisk/modules.conf

[download] [diff]

About 130 asterisk modules that weren’t used are prevented from loading now to keep the installation lighter. Only functions modules, applications that we currently use and codecs are loaded as well as CDR modules and some core asterisk modules.

/etc/asterisk/sip.conf

[download] [diff]

Support for the following codecs:

The additional following codecs are also supported but not active in the current version:

Support for video calls using the following codecs:

Database

[sql]

Only the table sipdevices2 is used now for authentication instead of sippeers2 and sipusers. This change optimizes almost by half the content of the database and decreases the amount of data replicated. The previous tables were dropped and the new one was created. All triggers and functions using the previous tables were updated to work with the new one.

 

Replication configuration was changed to replicate only authentication table sipdevices now available in DBA v6+.

 

The new Asterisk version is incompatible with Porta-SIP. As the new asterisk always authenticates first by username and then by IP address it always asks authentication to the Porta-SIP. Due to this request the Porta-SIP cancels the call as the customers authentication was already consumed by it. For Astrads the problem was solved by masking the username and sending the caller id in the RPID field.

 

As an alternative solution the module chan_sip was modified and recompiled to authenticate first based on IP address. However this solution was not chosen as it would imply that it would be needed to make this change in each future Asterisk update.

 

In previous versions the introduction of a field Deny, to set an ACL, prevented the authentication by username if the IP address matched the ACL. In the current version, that was changed and the ACL is only used to verify if that IP address can register, i.e. if the IP address is denied Asterisk will not search again in the table for other possible matches. For that reason the field Deny is not used anymore in this version. The size of the database is decreased by a large amount by this change because this field contained a list of every node and provider and it was replicated in each customer.

Authentication

/etc/astrad/script/agi-rad-auth.pl

[download] [diff]

In the script /etc/astrad/script/agi-rad-auth.pl, which is responsible for the authentication of every call, support for additional dialing symbols and number verification were added. Now a customer can dial numbers beginning with + (for international numbers) or # (for enabling disabling features). We now verify all the numbers that are dialed to guarantee that only the numbers 0 through 9 and the symbols +, # and * are dialed. This way Asterisk is protected from code injection as the variables used are now filtered.

 

Previous versions add a problem which prevented some phone models from making calls despite having registered if the field SIP field “Authorization Digest” was not written exactly this way. The match is now case insensitive to correct this problem.

Billing

/etc/astrad/script/ast-rad-acc.pl

[download] [diff]

To cope with the changes in the Asterisk version the script /etc/astrad/script/ast-rad-acc.pl responsible for sending the Radius packets to the Master had to be changed. In this version some of the messages don’t exist anymore and the content of other has modified. Due to this the script was almost entirely rewritten. Before the changes no CDRs appeared in Billing for internal, incoming or outgoing calls.

 

It was modified to prevent duplication of sent Radius packets when Master is down, a problem that makes customers to be billed repeatedly for the same call. Timeout for sending Radius message is 10 seconds. More Radius headers where added to replicate better the packet sent by Porta-SIP.

 

Bellow image showing one of the repeated calls. It can be seen that there are several repeated messages received and they are grouped under the same CDR. Afterwards some time Billing creates a new CDR even though the received packet is exactly the same.

 

Bellow logs from calls to from and to the same number in Astrad v11 and Porta-SIP.

[Astradv12]  [Porta-SIP]

 

[Astradv12]  [Porta-SIP]

 

[Astradv12]  [Porta-SIP]

 

This script now uses its own AMI user with read only permissions to connect to Asterisk.

/etc/astrad/script/ast-resend-lost.pl

[download] [diff]

To prevent repeated billing of the same call when Master is down, the script was modified to:

- have an increased timeout. Only after 10 seconds without answer a packet is marked as unsent

- after a failed attempt, the next attempt for each call will be increasingly slower. The next try will use the formula: NOW + 2Tries. This will help decrease the number of repeated billed calls in the case where the Master accepts a packet but doesn’t reply in time and also will decrease the number of simultaneous packets received by the Master when it returns.

- a correction in the previous code was made to prevent multiple sending of the same requests. In the previous version an array kept the unsent data and for each retry the data was added again to that array which would make the same call to be sent multiple times.

 

At this moment the script doesn’t verify if there is already a call with the same h323-id. This might happen in a normal call that passes through multiple servers for instance and in this case we should have multiple CDRs with same h323-id but different CLI and CLD.

 

The problem is this way minimized in Astrad v11 but can still be replicated by blocking all incoming connections from the Master in the firewall:

iptables -A INPUT  -s  66.234.138.71  -j DROP

Astrad

/etc/files/script/ast-tools.pl

[download] [diff]

Script was updated to work with the changes made in the database.

/etc/cron.d/astrad

[download] [diff]

Removed unneeded astrad reload at 3 AM. Asterisk in previous versions sometimes crashed during this reload.

/etc/logrotate.d/asterisk

[download] [diff]

/var/log/asterisk.log is now rotated daily instead of by size. This way it will be possible to filter by day. Using daily filters it will be possible to see which customers have tried to register in the last days or which IPs are attacking our servers and react accordingly. The logs are kept for 2 months.

Puppet

This version supports update or fresh installation. If it is an update it will uninstall previous asterisk version packages and files before installing new one. It will remove the previous asterisk user and create a new one to force the same asterisk id across all servers.

Wish list

Add following features:

•     Authentication: Use only Master for Billing. All authentications, features and routing should be local.

•     Authentication: Optimize call routing based on cost / priority

•     Authentication: Each Astrad should have all the information needed to authenticate and route calls without having to connect to other servers for each call. All that information should be replicated from DBAs in a simplified format.

•     Authentication: Automatically replace + for the international prefix defined in Billing site instead of assuming it is 00

•     Authentication: When receiving calls the number should be delivered in the format chosen in the web interface (+41, 0041, 41, etc)

•     Customer: Fax support. T30 and T38

•     Customer: Voicemail. Customers could receive email with voicemail or consult it using their phone account

•     Customer: Allow customers to choose desired codec

•     Customer: Create a table with all default sip configurations and another for customer’s customization. Allow customers to configure these setting from a web interface that should be created. Some of these options could be: type of NAT, type of DTMF, preferred codecs.

•     Customer: Show available credit in phones that support the option

•     DNS: TTL - the value we use is too high preventing us from being able to switch quickly to another server when there are problems

•     DNS: balancing

•     DB: Disabled and removed accounts shouldn’t appear in the registration tables, only active ones (DBA).

•     DB: Separate database replication configuration and scripts configuration, to allow configuration of new servers without having to make a new full dump of the MySQL database in each server. (DBA)

•     DB: Optimize registration tables. Only after a phone tries to register itself for the first time on a server, should its data appear on sipusers/sippeers. Now all accounts appear in all servers even though each Astrad only use a small percentage of those servers.

•     DB: Optimize queries that lock the database

•     Follow Me: Enable full Follow Me support with Radius authentication (time rules are not used yet)

•     Follow Me: Enable full Follow Me support when Master is down

•     Follow Me: Incoming calls should have a timeout superior to 30 seconds. This value should come from Billing follow me configuration if available.

•     Statistics: Recover call quality information from Asterisk Manager and send this information to Radius. Using this information:

1.      Choose automatically best codecs for this user

2.      Report server/destination problems

3.      Refund customers for bad quality calls

•     Asterisk: All error messages should have a version at least in English, French, German and Italian (maybe also in Spanish, Portuguese and Russian). At this moment there are only messages in English, French and German.

•     Asterisk: Use Kamailio to be able to add fields to calls or filter unwanted calls and messages before they reach asterisk

•     Maintenance: Fail2ban should write on a database instead of sending emails

 

Problems detected

 

Correct the following problems:

•     Simultaneous call limits don’t work when Master is down, as the table Active_Calls is on the Master and is updated by Radius packets.

•     The free call message is only played to internal destinations, not destinations where the cost is 0 when authentication is made by DBA instead of Master Radius

•     Procedures set_route and call_route can be optimized. The way it is done now causes table sipdevices2 to get lots of repeated UA entries. It also has a possible concurrency problem which can possibly make a user call the wrong customer. The table sipdevices2 should be cleaned after the call is made or after a delay. (there’s a maximum number of UA_XXX entries that limits the problem)

•     When Master is down there are multiple billing of the same call. This can be replicated in Astrad V12 by dropping incoming packets from Master in the firewall (Astrad will send but won't receive any reply). This shows that is a timeout problem. The timeout was set to 10 seconds. Most of the CDRs are grouped but some create a new CDR with potential new costs.

•     When Master is down call duration is null for calls between Porta-SIP and Astrad V010. (V12 cannot receive calls from Porta-SIP)

•     When Master is down Astrads cannot call phones on Porta- SIPs (location of those phones is in table location which is not replicated to Astrads or DBAs). PBSs can be used to get this value.

•     Correct Puppet to restart Perl scripts when they are modified and to reload configurations of those scripts when the corresponding configuration file is modified.

•     When Asterisk send Notifys to keep NAT open, it adds port 0 instead of using the port where it is running which prevents the User Agents from answering. Due to that no answer is received and Asterisk shows a Warning message in the logs. Bug: https://issues.asterisk.org/view.php?id=19276

•     Following message appears several times in /var/log/syslog due to a problem with OVH kernel:

Dec 22 15:52:04 db3 kernel: IPv6 addrconf: prefix with wrong length 56

References

 

Master Mysql Astrad DBA V007

http://ftp.switzernet.com/3/public/120308-dba-v7

 

Master MySQL-Astrad versions (DBA)

http://switzernet.com/3/public/110317-db3-versions

 

Astrad Versioning

http://switzernet.com/3/public/110126-astrad-versions

 

List of functionalities to add to Astrad

http://switzernet.com/3/public/110523-astrad-wish-list

 

Astrad v12 test procedure

http://ftp.switzernet.com/3/public/111121-astrad-tests

 

Astrad v12 voting

(to be added)

 

Porta-Switch Interfaces Manual

http://www.portaone.com/resources/docs/PortaSwitch/m-r-24/PortaSwitch_Interfaces_MR24.pdf