Make a full Billing Master replication
Created on 111116 by André Guimarães, Switzernet
This document describes how to make a full Billing Master replication.
Puppet installation and MySQL replication
Installation of pbs1.switzernet.com
WARNING:
Master’s replication takes much time during which all other servers will be
affected (some of them will not be able to accept calls). Be sure to do it at a
time of low call volume.
Pre-Installation
Make these steps before starting the installation of the new server.
Billing Master
In the Billing Master execute the following lines in MySQL (login as root) replacing IPADDRESS by the new server’s IP address.
INSERT IGNORE INTO user (Host,User,Password) VALUES ('IPADDRESS','astrob','');
GRANT PROCESS, FILE, SUPER, REPLICATION CLIENT, REPLICATION SLAVE, RELOAD,LOCK TABLES ON *.* TO 'astrob'@'IPADDRESS' IDENTIFIED BY '';
GRANT all privileges ON *.* TO 'astrob'@'IPADDRESS' IDENTIFIED BY '';
FLUSH PRIVILEGES;
If you’re going to replicate from an existing replication execute only the following lines in the server from where you’re doing the database dump.
GRANT all privileges ON *.* TO 'astrob'@'IPADDRESS' IDENTIFIED BY '';
FLUSH PRIVILEGES;
Reverse DNS
Replace the name of the server by its pbsXX name in Dedicated servers > Services > Reverse IPV4:
DNS
Create an entry in DNS for pbsXX.switzernet.com in server ns1.switzernet.com. Change serial value YYYYMMDD01 to the current date. Add the lines in blue changing the values on the example to the new FQDNs of the server. Replace pbsYY by the name of your new server and SERVERIP by its IP address. Reload bind and try to ping the new server by each of its new names.
vi /var/cache/bind/db.switzernet.com
; The $TTL directive at the top of the zone file (before the SOA)
; gives a default TTL for every RR without a specific TTL set.
$TTL 3600
switzernet.com. IN SOA ns1.switzernet.com. hostmaster (
YYYYMMDD01 ; Serial
30 ; Refresh
30M ; Retry
1W ; Expiry
3H ; Minimum
)
...
pbsYY IN A SERVERIP
...
/etc/init.d/bind9 restart
ping pbsYY.switzernet.com -c4
Installation
Install a new server Debian Lenny 5.0 64 bits. Make sure its root partition is very large.
Create an user sona
Access the new server by ssh as root using the password sent by email. Use the command adduser sona to create a new user. Replace password0 by the correct password. Leave all other options by default.
ks34189:~# adduser sona
Adding user `sona' ...
Adding new group `sona' (1000) ...
Adding new user `sona' (1000) with group `sona' ...
Creating home directory `/home/sona' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: password0
Retype new UNIX password: password0
passwd: password updated successfully
Changing the user information for sona
Enter the new value, or press ENTER for the default
Full Name []:
Room Number []:
Work Phone []:
Home Phone []:
Other []:
Is the information correct? [Y/n] y
ks34280:~#
Change the root password
Type the command passwd root to modify its password. Replace password1 by the desired password.
ks34280:~# passwd root
Enter new UNIX password: password1
Retype new UNIX password: password1
passwd: password updated successfully
Modify Hostname
Find a free pbs name (increase the number of the last one) and set the pbsXX to that name with the following command:
newfqdn=pbsXX.switzernet.com; echo $newfqdn;
Change the hostname of the machine in /etc/hostname and in /etc/hosts to the chosen name by executing the following commands:
hostname $newfqdn
echo $newfqdn > /etc/hostname
sed -i -r 's/(.*)ks.*/\1'$newfqdn'/g' /etc/hosts
sed -i -e '$d' /etc/motd;sed -i -e '/^hostname/d' /etc/motd; echo -e "hostname : `hostname`\ncompany : Switzernet@2011\n" >> /etc/motd
/etc/init.d/networking restart
Example
ks386980:~# newfqdn=pbs1.switzernet.com; echo $newfqdn;
pbs1.switzernet.com
ks386980:~# echo $newfqdn > /etc/hostname
ks386980:~# sed -i -r 's/(.*)ks.*/\1'$newfqdn'/g' /etc/hosts
ks386980:~# sed -i -e '$d' /etc/motd;sed -i -e '/^hostname/d' /etc/motd; echo -e "hostname : `hostname`\ncompany : Switzernet@2011\n" >> /etc/motd
ks386980:~# /etc/init.d/networking restart
Reconfiguring network interfaces...done.
Validation
Logout and login again. Verify the results seeing the content of /etc/hosts, /etc/hostname and executing hostname, comparing your results with the following example. Also notice the prompt as changed to the new name. Instead of pbs1.switzernet.com you should see your domain.
login as: sona
sona@176.31.247.50's password:
Debian GNU/Linux 5.0
Linux ks386980.kimsufi.com 2.6.38.2-grsec-xxxx-grs-ipv6-64 #2 SMP Thu Aug 25 16:40:22 UTC 2011 x86_64 GNU/Linux
server : 176453
ip : 176.31.247.50
hostname : pbs1.switzernet.com
company : Switzernet@2011
Last login: Wed Oct 5 10:16:34 2011 from 212-147-8-99.fix.access.vtx.ch
sona@ pbs1:~$ su -
Password:
pbs1:~# cat /etc/hostname
pbs1.switzernet.com
pbs1:~# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
176.31.247.50 pbs1.switzernet.com
# The following lines are desirable for IPv6 capable hosts
#(added automatically by netbase upgrade)
::1 ip6-localhost ip6-loopback
feo0::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
pbs1:~# hostname
pbs1.switzernet.com
Puppet installation and MySQL replication
If this a server reinstallation you need to run the following line as root on the Puppet Master server (puppet.switzernet.com), replacing XX by the new server number:
puppetca --clean pbsXX.switzernet.com
Install puppet in the new server. Execute the following commands as root:
aptitude update
aptitude install -y puppet
/etc/init.d/puppet stop
sed -i -r 's/^pluginsync=.*/pluginsync=false/g' /etc/puppet/puppet.conf
puppetd --server puppet.switzernet.com --waitforcert 60 --test
Connect to the Puppet Master (puppet.switzernet.com). Run the following commands (replacing XX by the new server number).
ping pbsXX.switzernet. com -c4
puppetca --list
puppetca --sign pbsXX.switzernet.com
Example:
puppet:~# puppetca --list
pbs1.switzernet.com
puppet:~# puppetca --sign pbs1.switzernet.com
Signed pbs1.switzernet.com
puppet:~# puppetca --list
No certificates to sign
In /etc/puppet/manifests/nodes.pp create a new node for the new server. Increase server_id by one (it must be unique) and in ip_nodes add the IPs of all Astrads. Add the new DB ip to each of the other DBA servers in the dba_sync_ip variable.
vim /etc/puppet/manifests/nodes.pp
…
node "pbs1.switzernet.com" {
$server_id=81
$masteruser='astrob'
$mastertype='porta'
$mysql_db3_user='root'
$mysql_db3_pass='XXXXXXX'
$mysql_localhost_user='astrob'
$mysql_ast_user='astrob'
$ip_nodes='66.234.138.73, 91.121.101.126, 91.121.75.124, 82.103.128.3, 91.121.167.75, 91.121.70.119, 91.121.138.5, 213.251.169.218, 91.121.99.16, 94.23.50.74, 91.121.121.115, 213.251.174.129, 91.121.16.79, 91.121.178.108, 91.121.142.9, 91.121.147.45, 91.121.143.56, 91.121.25.159, 91.121.205.108, 91.121.151.75, 91.121.172.156, 91.121.151.58, 176.31.247.50, 213.251.169.218, 91.121.117.76, 91.121.122.64'
$dba_sync_ip=''
import "pbs001"
import "snmp"
include pbs001
include snmp
…
In the new PBS server execute:
puppetd --server puppet.switzernet.com --waitforcert 60 --test
After executing the first command Puppet will install and configure all needed services. Verify if there aren’t any yellow or purple lines while the command runs.
WARNING:
Running this command will start Master’s DB full dump. During this time all
other servers will be affected (some of them will not be able to accept calls).
Be sure to do it at a time of low call volume. To prevent automatic replication
from happening comment the lines that run /etc/pbs/setup_replication.pl in puppet configuration and run this command from crontab at the
desired time.
FEATURE REQUIRED:
The next puppet configuration should have support to synchronize from file and
for choosing the time of SQL dump.
Monitoring
Add the server to monitoring list
Log in to http://monitor.switzernet.com/cacti and follow the procedure described in 2.3 and 3.6 of the following document:
http://switzernet.com/public/100510-cacti-monitoring/
Follow the procedure in:
http://ftp.switzernet.com/3/support/110419-cacti-graph-page-manual/
to add the graphs to:
http://monitor.switzernet.com/view_graphs_list.php
To add the synchronization status between the new server and Billing Master server, edit /root/folders/110726-check-replication/check-rep.conf and add the new server to STAR1 group. Verify if in this web page appears a new line with the master->pbsXX (It may take 30 minutes to appear):
http://switzernet.com/3/public/110727-normality-monitoring/
Installation of pbs1.switzernet.com
For the installation of this server puppet run everything except for the SQL dumping and importing. These lines were commented out from /etc/pbs/setup_replication.pl. The script was then uncommented and run again in crontab at 1 AM.
The database dumping started at 1AM Swiss time and ended at 4AM, 3 hours later. During this period all servers were affected due to a read lock on the Master Billing Database. This lock:
- prevents phones from making calls and registering on Porta-SIPs
- delays calls in Astrads (without Master they authenticate on DB2 – it will be changed on next version)
- prevents access to the Billing site
- prevents replication of location2 and multiple_ua between all Astrads servers (the script aborts if cannot read from the Master).
While there’s a read lock on Master, radius is unable to authenticate requests
from Porta-SIPs. This cause a blockage on Openser which causes high CPU in
these machines:
(in the period between 3:20 and 4:40 these servers were unreachable due to an
unrelated problem)
There are replication problems in table location2 because the scripts stops
when cannot read from the Billing Master. The servers continued in red alarm
after the DB lock was released because all events accumulated in a table Registration
in each of the servers. This table is emptied by a script that replicates its
contents to location2 in each astrad and location in Master. As the number of
queries got very high during the time the Master was locked it took almost an
hour to send everything in the table, during which the data in location2 was
old giving the red alarm. The other replications were not affected because as
nothing can write on Master, the state of Master and Slaves is the same.
The DB dump of porta-billing and porta-sip took about 3 hours.
The SQL import and Master/PBS synchronization took an additional 6 hours.
Resources
http://switzernet.com/3/public/111114-dba-v6
http://switzernet.com/public/100510-cacti-monitoring/
http://switzernet.com/company/100512-insert-new-node