Load balancing your SIPIS installation across multiple SIPIS processes

On a single machine

If your SIPIS installation hosts between 15000 upto 50000 users we recommend that you split the load across multiple SIPIS processes which run concurrently on a single machine.

As a running example, we are going to split the load across four SIPIS processes. The steps to enable load balancing are as follows.

1. Check and update your current SIPIS settings file

Before we start, make sure your current /etc/sipis/Settings.xml file contains FilterInstancesAccordingToLoadBalancer2BackendSection="Yes" and Realm attributes in the Server element:

<Sipis>

  <Server Name="…" Port="…"
    FilterInstancesAccordingToLoadBalancer2BackendSection="Yes"
    Realm="…" />

</Sipis>

If the Realm attribute is not there, set its value to your server’s current name, e.g. if your /etc/sipis/Settings.xml file looks like this:

<Sipis>

  <Server Name="SipisOne" Port="…" />

</Sipis>

Then after your modifications, it should look like this:

<Sipis>

  <Server Name="SipisOne" Port="…"
    FilterInstancesAccordingToLoadBalancer2BackendSection="Yes"
    Realm="SipisOne" />

</Sipis>

These changes don’t make any difference in the initial single SIPIS process scenario, however, if you later decide to rollback from your load balanced solution back to single SIPIS, then these two attributes become important.

Note

In the absence of the Realm attribute, the server Name attribute value is used for the realm parameter of the digest authentication mechanism. In the following sections we are going to change the server Name attribute and this could cause already running applications to fail to authenticate with SIPIS. Application restart will correct this problem, but that solution is not ideal. It’s better to keep the realm unchanged by explicitly specifying the Realm attribute.

2. Prepare additional SIPIS settings files

The default SIPIS installation comes with a single /etc/sipis/Settings.xml file, which systemd then passes as a command line parameter to the SIPIS process.

In fact, the SIPIS systemd script starts a SIPIS process for each Settings*.xml file it finds in the /etc/sipis/ directory. Therefore, to start four SIPIS processes we need to create four settings files by renaming the original /etc/sipis/Settings.xml to Settings1.xml and then copying it as Settings2.xml, Settings3.xml and Settings4.xml.

In the next step we need to edit each of the four settings files, supplying unique values for the following attributes:

<Sipis>

  <Server Name="Sipis" Port="14998" />

  <HttpServer Port="5000" />

  <Lock FileName="/var/run/sipis/sipis.lock"/>

  <Log FileName="/var/log/sipis/sipis.log" />

</Sipis>

The following table shows the values used in each settings file.

Attribute	Settings1.xml	Settings2.xml	Settings3.xml	Settings4.xml
Server[Name]	Sipis1	Sipis2	Sipis3	Sipis4
Server[Port]	14998	14988	14978	14968
HttpServer[Port]	5000	5010	5020	5030
Lock[FileName]	…/sipis1.lock	…/sipis2.lock	…/sipis3.lock	…/sipis4.lock
Log[FileName]	…/sipis1.log	…/sipis2.log	…/sipis3.log	…/sipis4.log

Note

You may have noticed a somewhat arbitrary convention that Server[Port] is decreased and HttpServer[Port] is increased by 10 in each consecutive settings file.

2. Update LoadBalancer2 settings file

In the second step we’ll update the Backend section of the LoadBalancer2 settings file located at /etc/acrobits/LoadBalancer2/Settings.xml.

For our example, it should look like the following:

<LoadBalancer2>

  <Backend>

    <Sipis Name="Sipis1" SelectorPrefix="0/2" Address="127.0.0.1:14998" />

    <Sipis Name="Sipis2" SelectorPrefix="4/2" Address="127.0.0.1:14988" />

    <Sipis Name="Sipis3" SelectorPrefix="8/2" Address="127.0.0.1:14978" />

    <Sipis Name="Sipis4" SelectorPrefix="C/2" Address="127.0.0.1:14968" />

  </Backend>

</LoadBalancer2>

Make sure the Sipis[Name] attributes in the LoadBalancer2 settings file match exactly the Server[Name] attributes in the SIPIS settings files as these attributes are used by SIPIS processes to discover its selector prefix(es).

A selector is SHA1 value calculated from SIP account credentials and is used to identify a particular SIP account in a SIPIS installation. The selector is also used as a basis for load balancing scheme in which each SIPIS process is assigned SIP accounts based on a few most significant bits of the selector.

In our example we used two most significant selector bits (the /2 part) to split the “selector space” into four sub-spaces coresponding to 00, 01, 10 and 11 binary prefixes. The prefix value in the Sipis[SelectorPrefix] attribute needs to be a hexadecimal string, though, so we padded the binary prefixes with binary zeros to the nearest length divisible by four and converted the result into hexadecimal digits, i.e. 00b → 0000b → 0 hex, 01b → 0100b → 4 hex, 10b → 1000b → 8 hex and 11b → 1100b → C hex.

3. Restart SIPIS machine or SIPIS services

To apply all the changes we have made, we can invoke the following commands

systemctl stop sipis
systemctl deamon-reload
systemctl restart LoadBalancer2
systemctl start sipis

or restart the whole SIPIS machine.

At the end you can invoke

top -u sipis

command to check that there is one LoadBalancer2 and four sipis processes running on the machine.

On multiple machines

In the following example we will split the users between eight machines with two SIPIS processes running on each. The procedure is similar to the previous example, but it requires deeper understanding of all components and how they interact together.

Deeper overview of SIPIS workings

For basic information see the general overview.

Components

Typical SIPIS deployment consists of multiple components, including SIPIS itself, the app, “load balancers”, the registration server, and a shared database. Here’s how these components interact

SIPIS

SIPIS registers on the SIP registrar on behalf of the user after the Softphone application goes to the background. Accounts handled by SIPIS are known as “sipis instances.” An instance for the user is said to “live” on this SIPIS. Each SIPIS can manage thousands of such instances, identified by a selector (a hexadecimal string derived from user data).

Application (App)

The app communicates with SIPIS to manage registration and call handling.

Registration Server

When SIPIS does not recognize an app, the app communicates with the registration server over HTTPS to provide the necessary information. The registration server stores this data in a database, which SIPIS then uses.

LoadBalancer2 (LB2):

This component proxies app traffic to the appropriate SIPIS instance.

Working

Initialization

When the app starts, it tries to connect to SIPIS. If SIPIS does not recognize the app, the app sends its data to the registration server via HTTPS and retries until SIPIS can process the registration.

Background Handling

If the app goes to the background or remains inactive for one minute, SIPIS takes over the registration for that user.

Receiving Calls

When SIPIS receives a call for a registered user, it sends a push notification to the app. The app wakes up and connects to the correct SIPIS to retrieve the call.

App Configuration Parameters

The app is configured using the following parameters:

sipisHost

The address of the SIPIS server.

sipisRegServer

The server used for HTTPS registration.

sipisHostPrefixLength

Used to calculate the SIPIS address based on a selector value, helping to distribute users across SIPIS instances.

Depending on sipisHostPrefixLength, the SIPIS address is determined as follows:

• If sipisHostPrefixLength is zero, the app communicates directly with sipisHost for all interactions (except HTTPS registration).

• If sipisHostPrefixLength is greater than zero, the app constructs the SIPIS address using a prefix derived from the hexadecimal selector value:

  • If sipisHostPrefixLength is 1, the app uses the first hexadecimal digit X of its selector and constructs the address as s-X.sipisHost.

  • If sipisHostPrefixLength is 2, the app uses the first two hexadecimal digits XY and constructs the address as s-XY.sipisHost.

  • This pattern continues as per the value of sipisHostPrefixLength.

Note

Registration server does not modify sipisRegServer this way.

Role of LoadBalancer2 (LB2)

The app does not communicate directly with SIPIS; instead, all traffic is routed through LB2, which acts as a proxy to forward requests to the correct SIPIS instance. By default, LB2 routes SIPIS traffic to 127.0.0.1 on port 14998, but it can also direct traffic based on the selector.

Data Persistence and SIPIS State Management

SIPIS periodically saves its state to a shared PostgreSQL database. When SIPIS restarts, it loads the previously assigned instances based on configurations set in LB2.

• The FilterInstancesAccordingToLoadBalancer2BackendSection in SIPIS settings makes it easier to ensure that each SIPIS only loads instances assigned to it.

• SIPIS stores encrypted passwords in the database using a shared sipis.key file. This key must be consistent across all SIPISes.

Standard Multi-Server Setup

To handle high loads (e.g., multiple thousands of instances), a multi-server setup is recommended. Here is an example of a typical configuration:

Machines

Eight machines, each running two SIPIS processes (16 SIPIS processes in total, we recommend using powers of two but it is not required).

Database

A shared PostgreSQL database accessible by all SIPISes.

Configuration

SIPIS processes are differentiated by unique names and ports. Each process has a unique configuration file. Shared configuration is stored in Default.xml that is referenced by individual configuration files.

Load Balancer (LB2)

Each machine also runs an LB2 instance, which directs traffic to the correct SIPIS based on selectors. All LB2 instances share the same configuration file.

Nginx

Each machine uses an Nginx server to handle HTTPS traffic for SIPIS registration, proxying the requests to SIPIS HTTP interface. This means every machine can act as a Registration Server.

Note

The configuration files for SIPISes, LB2 and NGINX can be created by a template engine.

The breakdown of the indivitual components’ configuration follows.

Multi-Server Setup SIPIS Configuration

• SIPISes run on several machines with IP addresses S0IP, S1IP, …, S7IP.

• Every SIPIS process has a different Name attribute sipis0, sipis1, … sipisF.

• Each SIPIS process on given machine has a unique configuration with different ports for SIP traffic and HTTP interfaces.

• Each SIPIS process on given machine has a unique “lock” and “log” files.

• All instances reference a shared Defaults.xml file which is created by renaming the Settings.xml file included with the SIPIS installation, and changing the following properties

• Each SIPIS process will have a unique /etc/sipis/Settings*.xml for each process (so that’s two files per machine) created according to the following template:

  <Sipis Defaults="/etc/sipis/Defaults.xml">
   <Server Name="{{s["Name"]}}" Port="{{s["sipisport"]}}" PrivateAddress="{{s["simulatenatip"]}}" />
   <HttpServer Port="{{s["httpport"]}}" />
   <Lock FileName="/var/run/sipis/sipis{{s["name"]}}.lock"/>
   <Log FileName="/var/log/sipis/sipis{{s["name"]}}.log" >
   </Log>
  </Sipis>
  

  Name

  This represents the SIPIS instance name and must be unique for each process.

  Port

  Each process on a machine must have a different port for SIP communication. Commonly, ports like 14998 and 14988 are used.

  HttpServer Port

  Similarly, HTTP interfaces must use unique ports, such as 5000 and 5010.

  Lock and Log Files

  The lock and log file paths use the name attribute to create distinct filenames for each instance, e.g., sipis{{s[“name”]}}.lock.

  PrivateAddress (optional but recommended)

  Should be an IP address from a private range, e.g., 10.x.x.x. Though these addresses can be the same across different machines, using distinct IPs for each instance is preferable for easier debugging.

  See Settings.xml documentation for more details.

Multiple-Server setup LB2 Configuration

The LB2 configuration is similar to single server configuration except the Backend, which specifies each SIPIS by name, address, and selector range. This allows for the correct routing of app traffic to the assigned SIPIS. The backend section of /etc/acrobits/LoadBalancer2/Settings.xml will look as follows:

<Sipis Name="sipis0" SelectorPrefix="0/4" Address="S0IP:14998" />
<Sipis Name="sipis1" SelectorPrefix="1/4" Address="S0IP:14988" />
<Sipis Name="sipis2" SelectorPrefix="2/4" Address="S1IP:14998" />
<Sipis Name="sipis3" SelectorPrefix="3/4" Address="S1IP:14988" />
<Sipis Name="sipis4" SelectorPrefix="4/4" Address="S2IP:14998" />
<Sipis Name="sipis5" SelectorPrefix="5/4" Address="S2IP:14988" />
<Sipis Name="sipis6" SelectorPrefix="6/4" Address="S3IP:14998" />
<Sipis Name="sipis7" SelectorPrefix="7/4" Address="S3IP:14988" />
<Sipis Name="sipis8" SelectorPrefix="8/4" Address="S4IP:14998" />
<Sipis Name="sipis9" SelectorPrefix="9/4" Address="S4IP:14988" />
<Sipis Name="sipisA" SelectorPrefix="A/4" Address="S5IP:14998" />
<Sipis Name="sipisB" SelectorPrefix="B/4" Address="S5IP:14988" />
<Sipis Name="sipisC" SelectorPrefix="C/4" Address="S6IP:14998" />
<Sipis Name="sipisD" SelectorPrefix="D/4" Address="S6IP:14988" />
<Sipis Name="sipisE" SelectorPrefix="E/4" Address="S7IP:14998" />
<Sipis Name="sipisF" SelectorPrefix="F/4" Address="S7IP:14988" />

• If TLS is used for SIP traffic, TLS section in the configuration file should be set with proper certificate+key pair

• If TLS is used for SIP traffic, each LB2 requires a certificate for its given s-X.sipis… hostname.

Multiple-Server setup NGINX configuration

upstream anylocalsipis {
    server 127.0.0.1:5000;
    server 127.0.0.1:5010;
}
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate_key /path/to/key.pem;
    ssl_certificate /path/to/cert/chain.pem.pem;

    location /sipis/register {
        proxy_pass http://anylocalsipis;
    }


    location /sipis5000/ {
        proxy_pass http://127.0.0.1:5000/;
    }

    location /sipis5010/ {
        proxy_pass http://127.0.0.1:5010/;
    }
}

• Locations /sipis5000/ and /sipis5010/ are not required, but are nice to have . Optional access control directives can be added there.

  • In case these locations are added, and you decide to use HTTP authentication in NGINX, add the following in Defaults.xml for SIPIS to avoid SIPIS AuthenticationMethod from clashing with NGINX authentication and also to restrict direct access to SIPIS HTTP interface:

    <HttpServer Address="127.0.0.1" AuthenticationMethod=”None”>
    

Multiple-Server setup DNS configuration

Recommended DNS configuration:

• The first machine would be assigned the DNS names s-0.sipis... and s-1.sipis...

• The second machine would use s-2.sipis... and s-3.sipis..., and so on.

Additionally, there would be a DNS record called all.sipis... that includes all the machines.

Note

All NGINX instances need a certificate for all.sipis....

Multiple-Server setup Application configuration

• Set sipisRegServer to all.sipis...

• Set SetsipisHost to sipis...

• Set sipisHostPrefixLength to 1

Variations and Additional Considerations

Single Registration Server

It is possible to use a single SIPIS instance for all HTTPS registrations by setting sipisRegServer to the designated SIPIS’ address. This can be any of the eight servers from above or separate server with SIPIS without any instances on it.

Dedicated Load Balancers

You can also set up dedicated servers for LB2, with the app connecting to these servers for all SIPIS interactions. In that case LB2 does not need to be installed on the SIPIS machines but the file /etc/acrobits/LoadBalancer2/Settings.xml still needs to be present on them.

How to Handle Failures in Acrobits SIPIS

Database Failure

The database is the only component holding persistent data. The best practice is to maintain a backup database with streaming replication from the primary database. In the event of a failure (e.g., the primary database becomes inaccessible or damaged), SIPIS will continue partially functioning, such as sending call notifications. To restore full functionality

Promoting the Backup Database

• Promote the backup database to the primary role.

• Assign it the same address as the original primary database.

• SIPIS will automatically reconnect and continue operations

Alternative Scenario

• If the backup cannot assume the same address, promote it to the primary role.

• Update the database address in all SIPIS instances.

• Restart all SIPIS instances. This process involves downtime, as SIPIS needs to reload and start registering instances at the SIP registrar. Registration is throttled to avoid overloading the SIP Registrar.

SIPIS Machine Failure

If a SIPIS machine fails, it can be replaced with an identical spare machine. Here’s the procedure

Setting Up the Replacement Machine

• Ensure the new machine is configured identically, with SIPIS, LB2, and NGINX installed and properly set up.

• Assign the failed machine’s IP address to the new one.

• Start SIPIS, LB2, and NGINX services.

This approach minimizes disruption, affecting only users connected to the failed machine.

When IP Address Transfer Is Not Possible:

• Update all LB2 across all machines.

• Change the DNS record to reflect the new machine’s address.

• Reload the LB2 service on all machines.

• Finally, start SIPIS on the new machine.

Even in this scenario, only users linked to the affected machine or using it as the HTTPS sipisRegServer will experience interruptions.