Load balancing your SIPIS installation across multiple SIPIS processes
======================================================================

On a single machine
```````````````````

If your SIPIS installation hosts between 15000 upto 50000 users
we recommend that you split the load across multiple SIPIS processes
which run concurrently on a single machine.

As a running example, we are going to split the load across four SIPIS processes.
The steps to enable load balancing are as follows.

1. Check and update your current SIPIS settings file
----------------------------------------------------

Before we start, make sure your current ``/etc/sipis/Settings.xml`` file
contains ``FilterInstancesAccordingToLoadBalancer2BackendSection="Yes"``
and ``Realm`` attributes in the ``Server`` element:

.. code-block:: xml
  
  <Sipis>
  
    <Server Name="…" Port="…"
      FilterInstancesAccordingToLoadBalancer2BackendSection="Yes"
      Realm="…" />
        
  </Sipis>

If the ``Realm`` attribute is not there, set its value to your server's
current name, e.g. if your ``/etc/sipis/Settings.xml`` file looks like
this:

.. code-block:: xml
  
  <Sipis>

    <Server Name="SipisOne" Port="…" />
      
  </Sipis>

Then after your modifications, it should look like this:

.. code-block:: xml
  
  <Sipis>

    <Server Name="SipisOne" Port="…"
      FilterInstancesAccordingToLoadBalancer2BackendSection="Yes"
      Realm="SipisOne" />
      
  </Sipis>

These changes don't make any difference in the initial single SIPIS process
scenario, however, if you later decide to rollback from your load balanced
solution back to single SIPIS, then these two attributes become important.

.. note::

  In the absence of the ``Realm`` attribute, the server ``Name`` attribute
  value is used for the realm parameter of the digest authentication mechanism.
  In the following sections we are going to change the server ``Name``
  attribute and this could cause already running applications to fail
  to authenticate with SIPIS. Application restart will correct this problem,
  but that solution is not ideal. It's better to keep the realm unchanged
  by explicitly specifying the ``Realm`` attribute.


2. Prepare additional SIPIS settings files
------------------------------------------

The default SIPIS installation comes with a single ``/etc/sipis/Settings.xml``
file, which ``systemd`` then passes as a command line parameter to the SIPIS process.

In fact, the SIPIS ``systemd`` script starts a SIPIS process for each ``Settings*.xml``
file it finds in the ``/etc/sipis/`` directory. Therefore, to start four SIPIS processes
we need to create four settings files by renaming the original ``/etc/sipis/Settings.xml``
to ``Settings1.xml`` and then copying it as ``Settings2.xml``, ``Settings3.xml`` and
``Settings4.xml``.

In the next step we need to edit each of the four settings files, supplying unique
values for the following attributes:

.. code-block:: xml
  
  <Sipis>

    <Server Name="Sipis" Port="14998" />
    
    <HttpServer Port="5000" />
    
    <Lock FileName="/var/run/sipis/sipis.lock"/>
    
    <Log FileName="/var/log/sipis/sipis.log" />

  </Sipis>

The following table shows the values used in each settings file.

================  ==============  ==============  ==============  ==============
Attribute         Settings1.xml   Settings2.xml   Settings3.xml   Settings4.xml
================  ==============  ==============  ==============  ==============
Server[Name]      Sipis1          Sipis2          Sipis3          Sipis4
Server[Port]      14998           14988           14978           14968
HttpServer[Port]  5000            5010            5020            5030
Lock[FileName]    …/sipis1.lock   …/sipis2.lock   …/sipis3.lock   …/sipis4.lock
Log[FileName]     …/sipis1.log    …/sipis2.log    …/sipis3.log    …/sipis4.log
================  ==============  ==============  ==============  ==============

.. note::

    You may have noticed a somewhat arbitrary convention that ``Server[Port]``
    is *decreased* and ``HttpServer[Port]`` is *increased* by 10 in each
    consecutive settings file.


2. Update LoadBalancer2 settings file
-------------------------------------

In the second step we'll update the ``Backend`` section of the LoadBalancer2
settings file located at ``/etc/acrobits/LoadBalancer2/Settings.xml``.

For our example, it should look like the following:

.. code-block:: xml

  <LoadBalancer2>
    
    <Backend>
 
      <Sipis Name="Sipis1" SelectorPrefix="0/2" Address="127.0.0.1:14998" />
 
      <Sipis Name="Sipis2" SelectorPrefix="4/2" Address="127.0.0.1:14988" />
 
      <Sipis Name="Sipis3" SelectorPrefix="8/2" Address="127.0.0.1:14978" />
 
      <Sipis Name="Sipis4" SelectorPrefix="C/2" Address="127.0.0.1:14968" />
    
    </Backend>

  </LoadBalancer2>

Make sure the ``Sipis[Name]`` attributes in the LoadBalancer2 settings file
match exactly the ``Server[Name]`` attributes in the SIPIS settings files as
these attributes are used by SIPIS processes to discover its selector prefix(es).

A selector is SHA1 value calculated from SIP account credentials and is used
to identify a particular SIP account in a SIPIS installation. The selector
is also used as a basis for load balancing scheme in which each SIPIS process
is assigned SIP accounts based on a few most significant bits of the selector.

In our example we used two most significant selector bits (the ``/2`` part)
to split the "selector space" into four sub-spaces coresponding to ``00``,
``01``, ``10`` and ``11`` *binary* prefixes. The prefix value in the
``Sipis[SelectorPrefix]`` attribute needs to be a hexadecimal string, though,
so we padded the binary prefixes with binary zeros to the nearest length
divisible by four and converted the result into hexadecimal digits, i.e.
``00b`` → ``0000b`` → ``0 hex``, ``01b`` → ``0100b`` → ``4 hex``, ``10b`` → ``1000b``
→ ``8 hex`` and ``11b`` → ``1100b`` → ``C hex``.    


3. Restart SIPIS machine or SIPIS services
------------------------------------------

To apply all the changes we have made, we can  invoke the following commands

.. code-block:: bash

  systemctl stop sipis
  systemctl deamon-reload
  systemctl restart LoadBalancer2
  systemctl start sipis

or restart the whole SIPIS machine.

At the end you can invoke

.. code-block:: bash

  top -u sipis

command to check that there is one ``LoadBalancer2`` and four ``sipis`` processes
running on the machine.


On multiple machines
````````````````````

In the following example we will split the users between eight machines with two SIPIS processes running on each. The procedure is similar to the previous example, but it requires deeper understanding of all components and how they interact together.

Deeper overview of SIPIS workings
---------------------------------

For basic information see :doc:`the general overview <overview>`.

Components
**********

Typical SIPIS deployment consists of multiple components, including SIPIS itself, the app, “load balancers”,  the registration server, and a shared database. Here's how these components interact

SIPIS
    SIPIS registers on the SIP registrar on behalf of the user after the Softphone application goes to the background. Accounts handled by SIPIS are known as "sipis instances." An instance for the user is said to "live" on this SIPIS. Each SIPIS can manage thousands of such instances, identified by a selector (a hexadecimal string derived from user data).

Application (App)
    The app communicates with SIPIS to manage registration and call handling.

Registration Server
    When SIPIS does not recognize an app, the app communicates with the registration server over HTTPS to provide the necessary information. The registration server stores this data in a database, which SIPIS then uses.

LoadBalancer2 (LB2):
    This component proxies app traffic to the appropriate SIPIS instance.

Working
*******

Initialization
    When the app starts, it tries to connect to SIPIS. If SIPIS does not recognize the app, the app sends its data to the registration server via HTTPS and retries until SIPIS can process the registration.

Background Handling
    If the app goes to the background or remains inactive for one minute, SIPIS takes over the registration for that user.

Receiving Calls
    When SIPIS receives a call for a registered user, it sends a push notification to the app. The app wakes up and connects to the correct SIPIS to retrieve the call.

App Configuration Parameters
****************************

The app is configured using the following parameters:

``sipisHost``
    The address of the SIPIS server.

``sipisRegServer``
    The server used for HTTPS registration.

``sipisHostPrefixLength``
    Used to calculate the SIPIS address based on a selector value, helping to distribute users across SIPIS instances.

Depending on ``sipisHostPrefixLength``, the SIPIS address is determined as follows:

* If ``sipisHostPrefixLength`` is zero, the app communicates directly with ``sipisHost`` for all interactions (except HTTPS registration).

* If ``sipisHostPrefixLength`` is greater than zero, the app constructs the SIPIS address using a prefix derived from the hexadecimal selector value:

    * If ``sipisHostPrefixLength`` is 1, the app uses the first hexadecimal digit X of its selector and constructs the address as ``s-X.sipisHost``.

    * If ``sipisHostPrefixLength`` is 2, the app uses the first two hexadecimal digits XY and constructs the address as ``s-XY.sipisHost``.

    * This pattern continues as per the value of ``sipisHostPrefixLength``.

.. note::

    Registration server does not modify ``sipisRegServer`` this way.

Role of LoadBalancer2 (LB2)
***************************

The app does not communicate directly with SIPIS; instead, all traffic is routed through LB2, which acts as a proxy to forward requests to the correct SIPIS instance. By default, LB2 routes SIPIS traffic to 127.0.0.1 on port 14998, but it can also direct traffic based on the selector.

Data Persistence and SIPIS State Management
*******************************************

SIPIS periodically saves its state to a shared PostgreSQL database. When SIPIS restarts, it loads the previously assigned instances based on configurations set in LB2.

    * The ``FilterInstancesAccordingToLoadBalancer2BackendSection`` in SIPIS settings makes it easier to ensure that each SIPIS only loads instances assigned to it.

    * SIPIS stores encrypted passwords in the database using a shared ``sipis.key`` file. This key must be consistent across all SIPISes.

Standard Multi-Server Setup
---------------------------

To handle high loads (e.g., multiple thousands of instances), a multi-server setup is recommended.
Here is an example of a typical configuration:

Machines
    Eight machines, each running two SIPIS processes (16 SIPIS processes in total, we recommend using powers of two but it is not required).

Database
    A shared PostgreSQL database accessible by all SIPISes.

Configuration
    SIPIS processes are differentiated by unique names and ports. Each process has a unique configuration file. Shared configuration is stored in Default.xml that is referenced by individual configuration files.

Load Balancer (LB2)
    Each machine also runs an LB2 instance, which directs traffic to the correct SIPIS based on selectors. All LB2 instances share the same configuration file.

Nginx
    Each machine uses an Nginx server to handle HTTPS traffic for SIPIS registration, proxying the requests to SIPIS HTTP interface. This means every machine can act as a Registration Server.

.. note::

    The configuration files for SIPISes, LB2 and NGINX can be created by a template engine.

The breakdown of the indivitual components' configuration follows.

Multi-Server Setup SIPIS Configuration
**************************************

* SIPISes run on several machines with IP addresses S0IP, S1IP, …, S7IP.
* Every SIPIS process has a different Name attribute sipis0, sipis1, … sipisF.
* Each SIPIS process on given machine has a unique configuration with different ports for SIP traffic and HTTP interfaces.
* Each SIPIS process on given machine has a unique "lock" and "log" files.
* All instances reference a shared ``Defaults.xml`` file which is created by renaming the ``Settings.xml`` file included with the SIPIS installation, and changing the following properties

.. code:
    <Server Address="0.0.0.0" Realm="Sipis" FilterInstancesAccordingToLoadBalancer2BackendSection="Yes" />
    <Database OpenString="host=DBIP port=5432 dbname=DBNAME user=DBUSER password=DBPASSWORD" />

* Each SIPIS process will have a unique /etc/sipis/Settings*.xml for each process (so that's two files per machine) created according to the following template:

    .. code:: xml+jinja

        <Sipis Defaults="/etc/sipis/Defaults.xml">
         <Server Name="{{s["Name"]}}" Port="{{s["sipisport"]}}" PrivateAddress="{{s["simulatenatip"]}}" />
         <HttpServer Port="{{s["httpport"]}}" />
         <Lock FileName="/var/run/sipis/sipis{{s["name"]}}.lock"/>
         <Log FileName="/var/log/sipis/sipis{{s["name"]}}.log" >
         </Log>
        </Sipis>

    Name
        This represents the SIPIS instance name and must be unique for each process.

    Port
        Each process on a machine must have a different port for SIP communication. Commonly, ports like 14998 and 14988 are used.

    HttpServer Port
        Similarly, HTTP interfaces must use unique ports, such as 5000 and 5010.

    Lock and Log Files
        The lock and log file paths use the name attribute to create distinct filenames for each instance, e.g., sipis{{s["name"]}}.lock.

    PrivateAddress (optional but recommended)
        Should be an IP address from a private range, e.g., 10.x.x.x. Though these addresses can be the same across different machines, using distinct IPs for each instance is preferable for easier debugging.

    See :doc:`Settings.xml<settings>` documentation for more details.

Multiple-Server setup LB2 Configuration
***************************************

The LB2 configuration is similar to single server configuration except the ``Backend``, which specifies each SIPIS by name, address, and selector range. This allows for the correct routing of app traffic to the assigned SIPIS. The backend section of ``/etc/acrobits/LoadBalancer2/Settings.xml`` will look as follows:

.. code:: xml

    <Sipis Name="sipis0" SelectorPrefix="0/4" Address="S0IP:14998" />
    <Sipis Name="sipis1" SelectorPrefix="1/4" Address="S0IP:14988" />
    <Sipis Name="sipis2" SelectorPrefix="2/4" Address="S1IP:14998" />
    <Sipis Name="sipis3" SelectorPrefix="3/4" Address="S1IP:14988" />
    <Sipis Name="sipis4" SelectorPrefix="4/4" Address="S2IP:14998" />
    <Sipis Name="sipis5" SelectorPrefix="5/4" Address="S2IP:14988" />
    <Sipis Name="sipis6" SelectorPrefix="6/4" Address="S3IP:14998" />
    <Sipis Name="sipis7" SelectorPrefix="7/4" Address="S3IP:14988" />
    <Sipis Name="sipis8" SelectorPrefix="8/4" Address="S4IP:14998" />
    <Sipis Name="sipis9" SelectorPrefix="9/4" Address="S4IP:14988" />
    <Sipis Name="sipisA" SelectorPrefix="A/4" Address="S5IP:14998" />
    <Sipis Name="sipisB" SelectorPrefix="B/4" Address="S5IP:14988" />
    <Sipis Name="sipisC" SelectorPrefix="C/4" Address="S6IP:14998" />
    <Sipis Name="sipisD" SelectorPrefix="D/4" Address="S6IP:14988" />
    <Sipis Name="sipisE" SelectorPrefix="E/4" Address="S7IP:14998" />
    <Sipis Name="sipisF" SelectorPrefix="F/4" Address="S7IP:14988" />

* If TLS is used for SIP traffic, TLS section in the configuration file should be set with proper certificate+key pair
* If TLS is used for SIP traffic, each LB2 requires a certificate for its given s-X.sipis... hostname.

Multiple-Server setup NGINX configuration
*****************************************

.. code:: nginx

    upstream anylocalsipis {
        server 127.0.0.1:5000;
        server 127.0.0.1:5010;
    }
    server {
        listen 443 ssl;
        listen [::]:443 ssl;
        ssl_certificate_key /path/to/key.pem;
        ssl_certificate /path/to/cert/chain.pem.pem;

        location /sipis/register {
            proxy_pass http://anylocalsipis;
        }


        location /sipis5000/ {
            proxy_pass http://127.0.0.1:5000/;
        }

        location /sipis5010/ {
            proxy_pass http://127.0.0.1:5010/;
        }
    }

* Locations ``/sipis5000/`` and ``/sipis5010/`` are not required, but are nice to have . Optional access control directives can be added there.

    * In case these locations are added, and you decide to use HTTP authentication in NGINX, add the following in ``Defaults.xml`` for SIPIS to avoid SIPIS ``AuthenticationMethod`` from clashing with NGINX authentication and also to restrict direct access to SIPIS HTTP interface:

        .. code:: xml

            <HttpServer Address="127.0.0.1" AuthenticationMethod=”None”>

Multiple-Server setup DNS configuration
***************************************

Recommended DNS configuration:

* The first machine would be assigned the DNS names ``s-0.sipis...`` and ``s-1.sipis...``
* The second machine would use ``s-2.sipis...`` and ``s-3.sipis...``, and so on.

Additionally, there would be a DNS record called ``all.sipis...`` that includes all the machines.

.. note::
    
    All NGINX instances need a certificate for ``all.sipis...``.

Multiple-Server setup Application configuration
***********************************************

* Set ``sipisRegServer`` to ``all.sipis...``
* Set ``SetsipisHost`` to ``sipis...``
* Set ``sipisHostPrefixLength`` to ``1``

Variations and Additional Considerations
****************************************

Single Registration Server
    It is possible to use a single SIPIS instance for all HTTPS registrations by setting sipisRegServer to the designated SIPIS' address. This can be any of the eight servers from above or separate server with SIPIS without any instances on it.

Dedicated Load Balancers
    You can also set up dedicated servers for LB2, with the app connecting to these servers for all SIPIS interactions. In that case LB2 does not need to be installed on the SIPIS machines but the file ``/etc/acrobits/LoadBalancer2/Settings.xml`` still needs to be present on them.

How to Handle Failures in Acrobits SIPIS
----------------------------------------

Database Failure
****************

The database is the only component holding persistent data. The best practice is to maintain a backup database with streaming replication from the primary database. In the event of a failure (e.g., the primary database becomes inaccessible or damaged), SIPIS will continue partially functioning, such as sending call notifications. To restore full functionality

Promoting the Backup Database
    * Promote the backup database to the primary role.
    * Assign it the same address as the original primary database.
    * SIPIS will automatically reconnect and continue operations

Alternative Scenario
    * If the backup cannot assume the same address, promote it to the primary role.
    * Update the database address in all SIPIS instances.
    * Restart all SIPIS instances. This process involves downtime, as SIPIS needs to reload and start registering instances at the SIP registrar. Registration is throttled to avoid overloading the SIP Registrar.

SIPIS Machine Failure
*********************

If a SIPIS machine fails, it can be replaced with an identical spare machine. Here’s the procedure

Setting Up the Replacement Machine
    * Ensure the new machine is configured identically, with SIPIS, LB2, and NGINX installed and properly set up.
    * Assign the failed machine’s IP address to the new one.
    * Start SIPIS, LB2, and NGINX services.

    This approach minimizes disruption, affecting only users connected to the failed machine.

When IP Address Transfer Is Not Possible:
    * Update all LB2  across all machines.
    * Change the DNS record to reflect the new machine's address.
    * Reload the LB2 service on all machines.
    * Finally, start SIPIS on the new machine.

    Even in this scenario, only users linked to the affected machine or using it as the HTTPS sipisRegServer will experience interruptions.