ViciDial Multi-Server Cluster Setup Guide

A complete, step-by-step guide to building a distributed ViciDial call center across multiple servers. Consolidates knowledge from the ViciDial forum thread t=4545 (66 replies), ViciBox documentation, PoundTeam concepts, and real production cluster experience into one clear reference.

Tested with: ViciBox 12 (openSUSE Leap 15.6), ViciBox 11 (openSUSE Leap 15.5), AlmaLinux 9 scratch installs, Asterisk 16/18/20, MariaDB 10.5-10.11

Last updated: 2026-03-13

Table of Contents

1. Architecture Overview
2. Prerequisites
3. Database Server Setup
4. Web Server Setup
5. Dialer/Asterisk Server Setup
6. IAX Trunks Between Servers
7. ViciDial Admin Configuration
8. Extension Configuration
9. Phone Registration and Load Balancing
10. Testing and Validation
11. Adding More Servers
12. Monitoring and Maintenance
13. Troubleshooting

1. Architecture Overview

1.1 What Is a ViciDial Cluster?

A ViciDial cluster splits the three core functions -- database, web interface, and telephony/dialing -- across dedicated servers. A single-server ViciDial install runs all three on one machine. A cluster separates them for scale, reliability, and geographic distribution.

1.2 When You Need a Cluster

Rule of thumb: One Asterisk/dialer server handles approximately 50-80 simultaneous agent sessions comfortably (depending on hardware and codec). Beyond that, add more dialer servers.

1.3 Architecture Diagram

                        +-----------------------+
                        |    INTERNET / PSTN    |
                        +-----------+-----------+
                                    |
                          SIP Trunks (inbound/outbound)
                                    |
            +-----------------------+-----------------------+
            |                       |                       |
    +-------+-------+     +--------+-------+     +---------+------+
    | DIALER 1      |     | DIALER 2       |     | DIALER 3       |
    | (Asterisk)    |     | (Asterisk)     |     | (Asterisk)     |
    | 10.0.0.11     |     | 10.0.0.12      |     | 10.0.0.13      |
    | Conf: 8600001 |     | Conf: 8600201  |     | Conf: 8600401  |
    |   - 8600200   |     |   - 8600400    |     |   - 8600600    |
    +-------+-------+     +--------+-------+     +---------+------+
            |   \                  |                  /   |
            |    \---- IAX2 ------+------ IAX2 -----/    |
            |         Trunks      |       Trunks         |
            |                     |                      |
    +-------+---------------------+----------------------+-------+
    |                     NETWORK (LAN/VLAN)                     |
    +-------+---------------------+----------------------+-------+
            |                     |                      |
    +-------+-------+    +-------+--------+    +--------+-------+
    | DATABASE      |    | WEB SERVER     |    | WEB SERVER     |
    | (MariaDB)     |    | (Apache+PHP)   |    | (Apache+PHP)   |
    | 10.0.0.1      |    | 10.0.0.2       |    | 10.0.0.3       |
    |               |    | Agent UI       |    | Admin/Reports  |
    | Central hub   |    | Points to DB   |    | Points to DB   |
    | for ALL data  |    | at 10.0.0.1    |    | at 10.0.0.1    |
    +---------------+    +----------------+    +----------------+

1.4 Server Roles Explained

Database Server (DB)

• Runs MariaDB (the ONLY copy of the database)
• All other servers connect to it remotely
• Runs DB-only cron jobs (hopper, GMT adjustment, optimization, archiving)
• Does NOT run Asterisk
• Does NOT serve web pages (in a pure cluster; can be combined)

Web Server(s)

• Runs Apache + PHP + ViciDial web files
• Agent login page, admin interface, reports
• dbconnect_mysqli.php points to the remote DB server IP
• astguiclient.conf has VARDB_server set to the DB server IP
• Runs minimal cron jobs (keepalive for web-related processes only)
• Does NOT run Asterisk (in a pure cluster)

Dialer/Asterisk Server(s)

• Runs Asterisk (handles all telephony: SIP, IAX, conferences, AGI)
• Runs the Perl keepalive daemons (AST_update, AST_send_listen, etc.)
• astguiclient.conf has VARDB_server set to the DB server IP
• Runs dialer-specific cron jobs
• Each dialer has its own conference bridge range
• Connected to other dialers via IAX2 trunks for cross-server transfers

1.5 Hardware Requirements Per Role

Database Server (most critical for performance)

The DB server is I/O bound. Fast storage and large InnoDB buffer pool are the two most impactful optimizations.

Web Server

Web servers are mostly stateless. You can run two behind a load balancer for HA.

Dialer/Asterisk Server (per server)

Asterisk is CPU-intensive during transcoding and conference mixing. Each server handles 50-80 concurrent agent sessions. Plan one dialer server per 50-75 agents.

2. Prerequisites

2.1 Operating System Options

ViciDial clusters can run on:

Recommendation: Use ViciBox 12 for all servers in the cluster. It provides a consistent base with ViciDial pre-installed and the vicibox-express setup wizard. For AlmaLinux 9, use the community auto-installer scripts.

Important: All servers in a cluster MUST run the same ViciDial SVN revision and the same database schema version (ExpectedDBSchema in astguiclient.conf). Mixing versions causes unpredictable behavior.

2.2 Network Requirements

CRITICAL NETWORK RULES:

1. All servers MUST be on the same LAN or VLAN
   - Latency between any two nodes: < 5ms (ideally < 1ms)
   - DB queries happen thousands of times per second
   - High latency = slow agent screens, dropped conferences

2. If servers are in different data centers:
   - Use a private VLAN or VPN between them
   - Latency > 20ms will cause noticeable agent UI lag
   - Latency > 50ms will cause conference/transfer failures

3. Firewall rules between cluster nodes:
   - TCP 3306 (MariaDB) - DB server from all nodes
   - TCP 80/443 (Apache) - Web server from agents/admins
   - UDP 4569 (IAX2) - Between all dialer servers
   - UDP 5060 + 10000-20000 (SIP + RTP) - Dialer to SIP trunks
   - TCP 4577 (FastAGI) - Loopback on each dialer (127.0.0.1 only)
   - ICMP - Allow between all nodes for health checks

4. All servers MUST have static IPs
   - ViciDial uses IP addresses as server identifiers
   - Changing an IP requires reconfiguring the entire cluster

2.3 NTP Time Synchronization

Time synchronization is CRITICAL in a ViciDial cluster. Even a 2-3 second drift between servers causes:

• Conference bridge timing mismatches
• Agent session overlap errors
• Incorrect call duration calculations
• Hopper timing failures

Strategy: One master NTP, all others sync to it.

On the Database Server (NTP master):

# Install chrony (preferred over ntpd)
# openSUSE / ViciBox:
zypper install chrony

# AlmaLinux / Rocky:
dnf install chrony

Edit /etc/chrony.conf:

# Upstream NTP servers
pool pool.ntp.org iburst maxsources 4
pool time.google.com iburst maxsources 2

# Allow cluster nodes to sync from this server
allow 10.0.0.0/24

# Serve time even when not synchronized to upstream
local stratum 10

# Log statistics
logdir /var/log/chrony
log measurements statistics tracking

systemctl enable chronyd
systemctl start chronyd

# Verify
chronyc sources -v
chronyc tracking

On ALL other servers (web + dialers):

Edit /etc/chrony.conf:

# Sync ONLY from the DB server (cluster NTP master)
server 10.0.0.1 iburst prefer

# Fallback to public pool if master is down
pool pool.ntp.org iburst maxsources 2

# Step the clock if drift exceeds 1 second (important for first sync)
makestep 1.0 3

logdir /var/log/chrony
log measurements statistics tracking

systemctl enable chronyd
systemctl restart chronyd

# Verify sync to master
chronyc sources -v
# Should show 10.0.0.1 as preferred source with * indicator

Verify across all nodes:

# Run on each server -- times should be within 0.01s of each other
date +%s.%N

2.4 Hostname and DNS Setup

Each server needs a resolvable hostname. Add entries to /etc/hosts on ALL servers in the cluster:

10.0.0.1    vici-db      vici-db.local
10.0.0.2    vici-web1    vici-web1.local
10.0.0.11   vici-dial1   vici-dial1.local
10.0.0.12   vici-dial2   vici-dial2.local
10.0.0.13   vici-dial3   vici-dial3.local

Set the hostname on each server:

hostnamectl set-hostname vici-db      # On DB server
hostnamectl set-hostname vici-dial1   # On Dialer 1
# etc.

2.5 ViciDial Source Code

All servers (web and dialer) need the ViciDial source code installed. On ViciBox, this is pre-installed. On scratch installs:

# Install SVN client
# openSUSE:
zypper install subversion

# AlmaLinux:
dnf install subversion

# Check out the ViciDial source (trunk = latest)
mkdir -p /usr/src/astguiclient
cd /usr/src/astguiclient
svn checkout svn://svn.eflo.net/agc_2-X/trunk

# The source tree will be at /usr/src/astguiclient/trunk/

3. Database Server Setup

The database server is the hub of the entire cluster. Every other server connects to it. Set this up FIRST.

3.1 Install MariaDB

ViciBox 12 (openSUSE):

MariaDB comes pre-installed with ViciBox. If installing on clean openSUSE:

zypper install mariadb mariadb-client
systemctl enable mariadb
systemctl start mariadb

AlmaLinux 9:

dnf install mariadb-server mariadb
systemctl enable mariadb
systemctl start mariadb
mysql_secure_installation

3.2 Configure MariaDB for Network Access

By default, MariaDB binds to 127.0.0.1 (localhost only). For a cluster, it must listen on the network.

Edit the MariaDB config file:

# ViciBox / openSUSE: /etc/my.cnf
# AlmaLinux: /etc/my.cnf.d/mariadb-server.cnf

Full recommended [mysqld] section for a cluster DB server:

[mysqld]
# CRITICAL: Bind to all interfaces (or specific LAN IP)
bind-address = 0.0.0.0
# Alternative: bind to specific LAN IP only (more secure)
# bind-address = 10.0.0.1

# Do NOT have skip-networking enabled -- remove or comment it out
# skip-networking

# Server ID (required, unique per server, useful if adding replication later)
server-id = 1

# Port (default 3306)
port = 3306

# Character set
character-set-server = utf8
collation-server = utf8_unicode_ci

# InnoDB settings (tune based on available RAM)
# Set to 60-70% of total RAM on a dedicated DB server
innodb_buffer_pool_size = 20G
innodb_log_file_size = 512M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000

# Connection limits (must handle all cluster nodes)
max_connections = 500
max_connect_errors = 100000

# Query safety -- kill long-running queries from web/cron
max_statement_time = 300

# Timeouts
wait_timeout = 28800
interactive_timeout = 28800
net_read_timeout = 60
net_write_timeout = 120

# Temp tables
tmp_table_size = 256M
max_heap_table_size = 256M

# Query cache (MariaDB 10.x)
query_cache_type = 1
query_cache_size = 128M

# Thread handling
thread_cache_size = 64
table_open_cache = 4096

# Logging
log-error = /var/log/mysql/mysqld.log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 5

# Binary logging (recommended for backups and future replication)
log_bin = mysql-bin
binlog_format = MIXED
expire_logs_days = 7

systemctl restart mariadb

Verify it is listening on the network:

ss -tlnp | grep 3306
# Should show: 0.0.0.0:3306 or 10.0.0.1:3306

3.3 Create the ViciDial Database

If this is a fresh install (not ViciBox where the DB already exists):

cd /usr/src/astguiclient/trunk

# Run the install script, it creates the 'asterisk' database and all tables
perl install.pl

During the install, you will be prompted for the server IP. Enter the DB server's LAN IP (e.g., 10.0.0.1).

If the database already exists (ViciBox), skip this step.

3.4 Create Database Users for Remote Access

This is the most critical step. Each remote server needs a MySQL user that can connect from its IP address. ViciDial uses two users: cron (for Perl scripts and cron jobs) and custom (for custom integrations).

Connect to MariaDB:

mysql -u root -p

Create users for EACH remote server:

-- ============================================================
-- GRANT STATEMENTS FOR VICIDIAL CLUSTER
-- Replace passwords with strong, unique values!
-- ============================================================

-- User: cron (primary ViciDial operations user)
-- Needs full access to the 'asterisk' database from each server

-- From Web Server 1 (10.0.0.2)
CREATE USER 'cron'@'10.0.0.2' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.2';

-- From Dialer 1 (10.0.0.11)
CREATE USER 'cron'@'10.0.0.11' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.11';

-- From Dialer 2 (10.0.0.12)
CREATE USER 'cron'@'10.0.0.12' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.12';

-- From Dialer 3 (10.0.0.13)
CREATE USER 'cron'@'10.0.0.13' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.13';

-- User: custom (for custom API/integration scripts)
CREATE USER 'custom'@'10.0.0.2' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.2';

CREATE USER 'custom'@'10.0.0.11' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.11';

CREATE USER 'custom'@'10.0.0.12' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.12';

CREATE USER 'custom'@'10.0.0.13' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.13';

-- OPTIONAL: Wildcard grant for entire subnet (less secure, but simpler)
-- Use this if you frequently add/remove servers
-- CREATE USER 'cron'@'10.0.0.%' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
-- GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.%';

-- Keep the localhost user for the DB server itself
-- (ViciDial scripts on the DB server use localhost)
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'localhost';

-- Apply changes
FLUSH PRIVILEGES;

IMPORTANT: The cron user needs ALL PRIVILEGES because ViciDial scripts perform INSERTs, UPDATEs, DELETEs, CREATEs, and ALTERs on the database. Do not restrict to SELECT only.

3.5 Configure astguiclient.conf on the DB Server

If the DB server also runs cron jobs (recommended for hopper, archiving, etc.), it needs its own astguiclient.conf:

vi /etc/astguiclient.conf

# /etc/astguiclient.conf - Database Server

# Paths
PATHhome => /usr/share/astguiclient
PATHlogs => /var/log/astguiclient
PATHagi => /var/lib/asterisk/agi-bin
PATHweb => /srv/www/htdocs
PATHsounds => /var/lib/asterisk/sounds
PATHmonitor => /var/spool/asterisk/monitor
PATHDONEmonitor => /var/spool/asterisk/monitorDONE

# THIS server's IP (the DB server's LAN IP)
VARserver_ip => 10.0.0.1

# Database connection -- localhost since DB is local
VARDB_server => localhost
VARDB_database => asterisk
VARDB_user => cron
VARDB_pass => Str0ng_Cr0n_P@ss!
VARDB_custom_user => custom
VARDB_custom_pass => Str0ng_Cust0m_P@ss!
VARDB_port => 3306

# Keepalives -- DB server does NOT run Asterisk
# X = no keepalive processes
VARactive_keepalives => X

# Asterisk version (even though not running Asterisk here)
VARasterisk_version => 18

# FastAGI settings (not needed on DB-only server, but required in file)
VARfastagi_log_min_servers => 3
VARfastagi_log_max_servers => 16
VARfastagi_log_min_spare_servers => 2
VARfastagi_log_max_spare_servers => 8
VARfastagi_log_max_requests => 1000
VARfastagi_log_checkfordead => 30
VARfastagi_log_checkforwait => 60

# DB Schema version -- MUST match across all servers
ExpectedDBSchema => 1736

3.6 Cron Jobs for the Database Server

The DB server runs the cron jobs that are database-intensive and should only run on ONE server in the cluster.

Create /var/spool/cron/tabs/root (openSUSE) or crontab -e (AlmaLinux):

### ==========================================================
### CRON JOBS FOR DATABASE SERVER ONLY
### ==========================================================

### Keepalive -- runs VARactive_keepalives processes
### On DB-only server with X, this is a no-op but keep it for consistency
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### ViciDial Hopper -- MUST run on exactly ONE server
### Populates the lead hopper for auto-dialing
* * * * * /usr/share/astguiclient/AST_VDhopper.pl -q

### Adjust GMT offsets on leads
1 1,7 * * * /usr/share/astguiclient/ADMIN_adjust_GMTnow_on_leads.pl --debug --list-settings

### Database optimization (runs during off-hours)
3 1 * * * /usr/share/astguiclient/AST_DB_optimize.pl

### Agent time log reports
2 0 * * 0 /usr/share/astguiclient/AST_agent_week.pl
22 0 * * * /usr/share/astguiclient/AST_agent_day.pl

### Archive old log tables daily (high-volume systems)
20 1 * * * /usr/share/astguiclient/ADMIN_archive_log_tables.pl --daily

### Clean up dead callbacks
25 0 * * * /usr/share/astguiclient/AST_DB_dead_cb_purge.pl --purge-non-cb --quiet

### Flush queue DB table every hour
11 * * * * /usr/share/astguiclient/AST_flush_DBqueue.pl -q

### Reset temporary info tables
2 1 * * * /usr/share/astguiclient/AST_reset_mysql_vars.pl

### Fix agent log once per hour and full day at night
33 * * * * /usr/share/astguiclient/AST_cleanup_agent_log.pl
50 0 * * * /usr/share/astguiclient/AST_cleanup_agent_log.pl --last-24hours

### Compress and rotate astguiclient logs
25 2 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz -9 >/dev/null 2>&1
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +30 -print | xargs rm -f

### Email processing -- MUST run on exactly ONE server
### Uncomment if using inbound email features
#* * * * * /usr/share/astguiclient/AST_inbound_email_parser.pl

Checkpoint -- Verify Database Server:

# 1. MariaDB is running and listening on network
ss -tlnp | grep 3306

# 2. Can connect locally
mysql -u cron -p'Str0ng_Cr0n_P@ss!' -e "SELECT COUNT(*) FROM asterisk.servers;"

# 3. NTP is synchronized
chronyc tracking

# 4. Cron is active
crontab -l | grep -c "astguiclient"
# Should show 10+ entries

4. Web Server Setup

The web server hosts the ViciDial agent interface, admin panel, and reports. It does NOT run Asterisk.

4.1 Install Apache and PHP

ViciBox 12 (pre-installed):

Apache and PHP are already configured. Verify:

systemctl status apache2
php -v

AlmaLinux 9 (scratch install):

dnf install httpd php php-mysqlnd php-cli php-gd php-xml php-mbstring php-json php-opcache
systemctl enable httpd
systemctl start httpd

openSUSE Leap (manual install):

zypper install apache2 php8 php8-mysql php8-gd php8-mbstring php8-xmlwriter php8-opcache
a2enmod php8
systemctl enable apache2
systemctl start apache2

4.2 Required PHP Modules

Verify these PHP modules are loaded:

php -m | grep -iE 'mysql|gd|mbstring|xml|json|curl|openssl|session'

Required modules:

• mysqli or mysqlnd -- database connectivity
• gd -- graph generation for reports
• mbstring -- multi-byte string handling
• xml -- XML processing
• json -- JSON handling (built-in on PHP 8)
• curl -- API calls
• session -- login sessions

4.3 Install ViciDial Web Files

The ViciDial web interface files come from the SVN source tree. On ViciBox they are pre-installed in /srv/www/htdocs/. On scratch installs:

cd /usr/src/astguiclient/trunk

# Run the install script
# When prompted, specify:
#   - Server IP: this web server's IP (e.g., 10.0.0.2)
#   - DB server: the database server IP (e.g., 10.0.0.1)
perl install.pl

# The installer copies web files to PATHweb (default: /srv/www/htdocs/)
# and creates /etc/astguiclient.conf

4.4 Configure Database Connection

ViciDial web files connect to the database through two mechanisms:

4.4.1 astguiclient.conf

vi /etc/astguiclient.conf

# /etc/astguiclient.conf - Web Server

# Paths
PATHhome => /usr/share/astguiclient
PATHlogs => /var/log/astguiclient
PATHagi => /var/lib/asterisk/agi-bin
PATHweb => /srv/www/htdocs
PATHsounds => /var/lib/asterisk/sounds
PATHmonitor => /var/spool/asterisk/monitor
PATHDONEmonitor => /var/spool/asterisk/monitorDONE

# THIS server's IP
VARserver_ip => 10.0.0.2

# CRITICAL: Point to the REMOTE database server
VARDB_server => 10.0.0.1
VARDB_database => asterisk
VARDB_user => cron
VARDB_pass => Str0ng_Cr0n_P@ss!
VARDB_custom_user => custom
VARDB_custom_pass => Str0ng_Cust0m_P@ss!
VARDB_port => 3306

# Web server does NOT run Asterisk -- no keepalive processes
VARactive_keepalives => X

VARasterisk_version => 18

VARfastagi_log_min_servers => 3
VARfastagi_log_max_servers => 16
VARfastagi_log_min_spare_servers => 2
VARfastagi_log_max_spare_servers => 8
VARfastagi_log_max_requests => 1000
VARfastagi_log_checkfordead => 30
VARfastagi_log_checkforwait => 60

ExpectedDBSchema => 1736

4.4.2 dbconnect_mysqli.php

This file is the PHP database connection include used by all ViciDial web pages. It reads from astguiclient.conf, but verify it exists and is correct:

# ViciBox default location:
cat /srv/www/htdocs/vicidial/dbconnect_mysqli.php

# AlmaLinux default:
cat /var/www/html/vicidial/dbconnect_mysqli.php

The file should contain code that reads VARDB_server, VARDB_database, VARDB_user, and VARDB_pass from /etc/astguiclient.conf. It should NOT have hardcoded localhost values. If it does, update them to match astguiclient.conf.

4.5 Apache Configuration

ViciBox default vhost (/etc/apache2/vhosts.d/vicidial.conf or similar):

<VirtualHost *:80>
    DocumentRoot /srv/www/htdocs
    ServerName vici-web1.local

    <Directory /srv/www/htdocs>
        Options -Indexes +FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    <Directory /srv/www/htdocs/vicidial>
        Options -Indexes
        AllowOverride All
        Require all granted
    </Directory>

    # PHP settings for ViciDial
    <IfModule mod_php.c>
        php_value max_execution_time 330
        php_value max_input_time 300
        php_value memory_limit 256M
        php_value post_max_size 64M
        php_value upload_max_filesize 64M
        php_value session.gc_maxlifetime 28800
        php_value error_reporting 22519
    </IfModule>

    ErrorLog /var/log/apache2/vicidial_error.log
    CustomLog /var/log/apache2/vicidial_access.log combined
</VirtualHost>

For AlmaLinux, the paths differ:

DocumentRoot /var/www/html
ErrorLog /var/log/httpd/vicidial_error.log
CustomLog /var/log/httpd/vicidial_access.log combined

systemctl restart apache2   # openSUSE
# or
systemctl restart httpd     # AlmaLinux

4.6 Cron Jobs for the Web Server

The web server runs minimal cron jobs. Most are NOT needed here because the web server does not run Asterisk.

### ==========================================================
### CRON JOBS FOR WEB SERVER ONLY
### ==========================================================

### Keepalive -- with VARactive_keepalives => X, this is a no-op
### Keep it for consistency in case you add web-related processes later
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### Compress/rotate astguiclient logs
25 2 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz -9 >/dev/null 2>&1
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +30 -print | xargs rm -f

That is all. The heavy cron jobs run on the DB server and dialer servers.

4.7 Recording Access

Agents and managers need to play back call recordings through the web interface. Recordings are stored on the dialer servers where the calls were processed. You have several options:

Option A: NFS Mount (recommended for LAN clusters)

Mount each dialer's recording directory on the web server:

# On web server
mkdir -p /var/spool/asterisk/monitorDONE

# Mount from each dialer (add to /etc/fstab)
10.0.0.11:/var/spool/asterisk/monitorDONE /mnt/recordings-dial1 nfs defaults,ro 0 0
10.0.0.12:/var/spool/asterisk/monitorDONE /mnt/recordings-dial2 nfs defaults,ro 0 0

Option B: FTP/SCP Transfer

Use the ViciDial recording FTP scripts (AST_CRON_audio_3_ftp.pl) on each dialer to push recordings to a central location accessible by the web server.

Option C: HTTP Redirect

Configure Apache on each dialer to serve recordings directly, and use ViciDial's VARHTTP_path setting to point to the correct dialer's URL per recording.

Checkpoint -- Verify Web Server:

# 1. Apache is running
systemctl status apache2   # or httpd

# 2. Can connect to remote database
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "SELECT COUNT(*) FROM asterisk.vicidial_users;"

# 3. ViciDial admin login works
curl -s http://10.0.0.2/vicidial/admin.php | grep -i "login"
# Should return the login page HTML

# 4. PHP has required modules
php -m | grep -c "mysqli\|gd\|mbstring"
# Should return 3

5. Dialer/Asterisk Server Setup

Each dialer server runs Asterisk and the ViciDial Perl daemon processes. This is where calls are originated, conferences are hosted, and agents connect their SIP phones.

5.1 Install Asterisk

ViciBox 12:

Asterisk comes pre-installed (currently Asterisk 18 or 20).

asterisk -V
# Asterisk 18.x.x-vici or Asterisk 20.x.x-vici

AlmaLinux 9 (scratch install):

Use the community auto-installer script or compile from source. The ViciDial patched version of Asterisk is required (includes features like enhanced MixMonitor):

cd /usr/src
# Download ViciDial-patched Asterisk source
# (Check the ViciDial forum for the latest recommended version)
wget http://download.vicidial.com/required-apps/asterisk-18-current-vici.tar.gz
tar xzf asterisk-18-current-vici.tar.gz
cd asterisk-18.*/
./configure --with-jansson-bundled
make menuselect    # Enable: app_meetme, app_confbridge, res_timing_dahdi
make
make install
make samples
make config        # Install init scripts

Install DAHDI (required for MeetMe timing on older setups):

cd /usr/src
wget http://download.vicidial.com/required-apps/dahdi-linux-current.tar.gz
tar xzf dahdi-linux-current.tar.gz
cd dahdi-linux-*/
make
make install

# Load timing module
modprobe dahdi
dahdi_genconf

Note on MeetMe vs ConfBridge:

• Asterisk 16-18: Both available. MeetMe requires DAHDI timing. ConfBridge is preferred on newer installs.
• Asterisk 20+: ConfBridge is the default. MeetMe is deprecated.
• ViciDial supports both. Set VARactive_keepalives to include C for ConfBridge mode.

5.2 Configure astguiclient.conf on Each Dialer

Each dialer server gets its own astguiclient.conf with unique settings:

vi /etc/astguiclient.conf

Dialer 1 (10.0.0.11):

# /etc/astguiclient.conf - Dialer Server 1

# Paths (standard ViciBox paths)
PATHhome => /usr/share/astguiclient
PATHlogs => /var/log/astguiclient
PATHagi => /var/lib/asterisk/agi-bin
PATHweb => /srv/www/htdocs
PATHsounds => /var/lib/asterisk/sounds
PATHmonitor => /var/spool/asterisk/monitor
PATHDONEmonitor => /var/spool/asterisk/monitorDONE

# THIS server's IP -- MUST match the IP registered in ViciDial admin
VARserver_ip => 10.0.0.11

# CRITICAL: Point to the REMOTE database server
VARDB_server => 10.0.0.1
VARDB_database => asterisk
VARDB_user => cron
VARDB_pass => Str0ng_Cr0n_P@ss!
VARDB_custom_user => custom
VARDB_custom_pass => Str0ng_Cust0m_P@ss!
VARDB_port => 3306

# Keepalive processes to run on this DIALER server
# 1 = AST_update (Asterisk state updates)
# 2 = AST_send_listen (send/listen audio monitoring)
# 3 = AST_VDauto_dial (auto-dialer -- runs on ALL dialer servers)
# 4 = AST_VDremote_agents (remote agent handling)
# 5 = AST_VDadapt (ONLY on ONE server in the entire cluster)
# 6 = FastAGI_log (FastAGI logging daemon)
# 7 = AST_VDauto_dial_FILL (ONLY on ONE server in the cluster)
# 8 = ip_relay (blind monitoring relay)
# 9 = Timeclock auto logout (ONLY on ONE server in the cluster)
# C = ConfBridge process
# E = Email processor (ONLY on ONE server in the cluster)
# S = SIP Logger (Asterisk 13+ with patches)

# Dialer 1 -- the "primary" dialer, runs the singleton processes
VARactive_keepalives => 12345679C

# NOTE: Dialer 2 and 3 would use: 12346C
# (No 5, 7, or 9 -- those are singletons running on Dialer 1)

VARasterisk_version => 18

# FTP for recording archival (optional)
VARFTP_host => 10.0.0.1
VARFTP_user => cron
VARFTP_pass => test
VARFTP_port => 21
VARFTP_dir => RECORDINGS
VARHTTP_path => http://10.0.0.11/RECORDINGS/MP3/

# FastAGI settings
VARfastagi_log_min_servers => 3
VARfastagi_log_max_servers => 16
VARfastagi_log_min_spare_servers => 2
VARfastagi_log_max_spare_servers => 8
VARfastagi_log_max_requests => 1000
VARfastagi_log_checkfordead => 30
VARfastagi_log_checkforwait => 60

ExpectedDBSchema => 1736

5.3 Singleton Process Distribution

These processes MUST run on exactly ONE server in the cluster. Running them on multiple servers causes data corruption:

Summary of VARactive_keepalives per server:

DB Server:       X               (no Asterisk processes)
Web Server:      X               (no Asterisk processes)
Dialer 1:       12345679C        (primary, runs singletons)
Dialer 2:       12346C           (secondary)
Dialer 3:       12346C           (secondary)

If you add the E (email) process, add it ONLY to Dialer 1: 12345679CE

5.4 Run the ViciDial Installer on Each Dialer

cd /usr/src/astguiclient/trunk

# Install ViciDial scripts and AGI files
perl install.pl

# When prompted:
#   Server IP: 10.0.0.11 (this dialer's IP)
#   Database Server: 10.0.0.1 (the central DB)
#   Database Name: asterisk
#   Database User: cron
#   Database Pass: (your password)

This copies:

• Perl scripts to /usr/share/astguiclient/
• AGI scripts to /var/lib/asterisk/agi-bin/
• Web files to /srv/www/htdocs/ (not needed on dialer, but harmless)
• Creates/updates /etc/astguiclient.conf

5.5 Cron Jobs for Dialer Servers

Dialer 1 (Primary -- runs singleton crons):

### ==========================================================
### CRON JOBS FOR DIALER SERVER 1 (PRIMARY)
### ==========================================================

### Keepalive -- restarts crashed processes defined in VARactive_keepalives
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### Compress/rotate astguiclient logs
25 2 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz -9 >/dev/null 2>&1
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +30 -print | xargs rm -f

### Kill hung/congested Asterisk channels
* * * * * /usr/share/astguiclient/AST_manager_kill_hung_congested.pl

### Voicemail updater
* * * * * /usr/share/astguiclient/AST_vm_update.pl

### Conference validator
* * * * * /usr/share/astguiclient/AST_conf_update.pl

### Recording mixing and compression (runs every 3 minutes)
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_mix.pl --MIX
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_VDonly.pl
1,4,7,10,13,16,19,22,25,28,31,34,37,40,43,46,49,52,55,58 * * * * /usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3

### FTP recordings to archive server (uncomment if using FTP archival)
#2,5,8,11,14,17,20,23,26,29,32,35,38,41,44,47,50,53,56,59 * * * * /usr/share/astguiclient/AST_CRON_audio_3_ftp.pl --MP3 --run-check

### Remove old recordings (adjust retention as needed)
#24 0 * * * /usr/bin/find /var/spool/asterisk/monitorDONE -maxdepth 2 -type f -mtime +90 -print | xargs rm -f
24 1 * * * /usr/bin/find /var/spool/asterisk/monitorDONE/ORIG -maxdepth 2 -type f -mtime +1 -print | xargs rm -f

### Rotate old Asterisk logs
29 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 3 -type f -mtime +30 -print | xargs rm -f
30 0 * * * /usr/bin/find / -maxdepth 1 -name "screenlog.0*" -mtime +7 -print | xargs rm -f
31 0 * * * /usr/bin/find /tmp -maxdepth 1 -type f -mtime +7 -print | xargs rm -f
32 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz >/dev/null 2>&1

Dialer 2 and 3 (Secondary -- NO singleton crons):

### ==========================================================
### CRON JOBS FOR DIALER SERVER 2/3 (SECONDARY)
### ==========================================================

### Keepalive
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### Compress/rotate astguiclient logs
25 2 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz -9 >/dev/null 2>&1
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +30 -print | xargs rm -f

### Kill hung/congested channels
* * * * * /usr/share/astguiclient/AST_manager_kill_hung_congested.pl

### Voicemail updater
* * * * * /usr/share/astguiclient/AST_vm_update.pl

### Conference validator
* * * * * /usr/share/astguiclient/AST_conf_update.pl

### Recording mixing and compression
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_mix.pl --MIX
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_VDonly.pl
1,4,7,10,13,16,19,22,25,28,31,34,37,40,43,46,49,52,55,58 * * * * /usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3

### FTP recordings (uncomment if using)
#2,5,8,11,14,17,20,23,26,29,32,35,38,41,44,47,50,53,56,59 * * * * /usr/share/astguiclient/AST_CRON_audio_3_ftp.pl --MP3 --run-check

### Remove old recordings
#24 0 * * * /usr/bin/find /var/spool/asterisk/monitorDONE -maxdepth 2 -type f -mtime +90 -print | xargs rm -f
24 1 * * * /usr/bin/find /var/spool/asterisk/monitorDONE/ORIG -maxdepth 2 -type f -mtime +1 -print | xargs rm -f

### Rotate Asterisk logs
29 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 3 -type f -mtime +30 -print | xargs rm -f
30 0 * * * /usr/bin/find / -maxdepth 1 -name "screenlog.0*" -mtime +7 -print | xargs rm -f
31 0 * * * /usr/bin/find /tmp -maxdepth 1 -type f -mtime +7 -print | xargs rm -f
32 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz >/dev/null 2>&1

Key difference: Secondary dialers do NOT run AST_VDhopper.pl, AST_VDadapt, AST_VDauto_dial_FILL, AST_inbound_email_parser.pl, or the DB optimization/archival scripts. Those run on the DB server or Dialer 1 only.

Checkpoint -- Verify Dialer Server:

# 1. Asterisk is running
asterisk -V
systemctl status asterisk

# 2. Can connect to remote database
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "SELECT COUNT(*) FROM asterisk.servers;"

# 3. Keepalive processes are running (after 1-2 minutes)
screen -ls
# Should show: AST_update, AST_send_listen, AST_VDauto_dial, FastAGI_log, etc.

# 4. FastAGI is listening
ss -tlnp | grep 4577
# Should show a Perl process on 127.0.0.1:4577

# 5. NTP is synchronized
chronyc tracking

6. IAX Trunks Between Servers

IAX2 (Inter-Asterisk eXchange) trunks connect dialer servers so they can transfer calls, send agents to conferences on other servers, and perform 3-way conferences across the cluster.

6.1 Why IAX2 (Not SIP)?

• IAX2 uses a single UDP port (4569) -- simpler firewall rules
• IAX2 trunks multiplex multiple calls on one connection -- lower overhead
• IAX2 is native Asterisk-to-Asterisk protocol -- designed for this use case
• ViciDial's cross-server conference transfer AGI scripts expect IAX2

6.2 IAX Peer Configuration

Each server defines IAX2 peers for every OTHER server in the cluster.

On Dialer 1 (10.0.0.11) -- /etc/asterisk/iax.conf:

[general]
bindport = 4569
bindaddr = 0.0.0.0
delayreject = yes
adsi = no
language = en
disallow = all
allow = ulaw
allow = alaw
allow = gsm
jitterbuffer = no
forcejitterbuffer = no
maxjitterbuffer = 1000
maxjitterinterps = 10
resyncthreshold = -1
trunkfreq = 20
trunktimestamps = yes
minregexpire = 60
maxregexpire = 360
iaxthreadcount = 50
iaxmaxthreadcount = 250
authdebug = no
regcontext = iaxregistration
autokill = yes
codecpriority = host
allowfwdownload = no
context = trunkinbound

#include iax-vicidial.conf

; ---- CLUSTER IAX PEERS ----

; Peer: Dialer 2
[dialer2]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.12
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

; Peer: Dialer 3
[dialer3]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.13
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

On Dialer 2 (10.0.0.12) -- /etc/asterisk/iax.conf:

; (same [general] section as above)

#include iax-vicidial.conf

; ---- CLUSTER IAX PEERS ----

; Peer: Dialer 1
[dialer1]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.11
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

; Peer: Dialer 3
[dialer3]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.13
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

On Dialer 3 (10.0.0.13) -- /etc/asterisk/iax.conf:

; (same [general] section as above)

#include iax-vicidial.conf

; Peer: Dialer 1
[dialer1]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.11
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

; Peer: Dialer 2
[dialer2]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.12
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

6.3 IAX Registration (Optional but Recommended)

Registration allows servers to discover each other dynamically. Add registration lines to iax.conf on each server:

On Dialer 1:

register => dialer1:[email protected]
register => dialer1:[email protected]

On Dialer 2:

register => dialer2:[email protected]
register => dialer2:[email protected]

On Dialer 3:

register => dialer3:[email protected]
register => dialer3:[email protected]

6.4 Reload IAX on All Servers

# On each dialer:
asterisk -rx "iax2 reload"

# Verify peers are registered:
asterisk -rx "iax2 show peers"
# Should show the other dialers with OK status

# Verify registrations:
asterisk -rx "iax2 show registry"
# Should show Registered status for each peer

6.5 Verify IAX Connectivity

# From Dialer 1, check peers:
asterisk -rx "iax2 show peers"

# Expected output:
# Name/Username  Host           (S)    Mask             Port    Status     Description
# dialer2        10.0.0.12       (S)    255.255.255.255  4569    OK (1 ms)
# dialer3        10.0.0.13       (S)    255.255.255.255  4569    OK (2 ms)

# Test a call between servers (from Asterisk CLI):
# asterisk -rx "channel originate IAX2/dialer2:[email protected]/s@default application Playback hello-world"

7. ViciDial Admin Configuration

After all servers are installed and connected, configure the cluster through the ViciDial admin web interface.

7.1 Access the Admin Panel

http://10.0.0.2/vicidial/admin.php

Default credentials (change immediately):

• Username: 6666
• Password: 1234

7.2 Register Servers

Navigate to: Admin > Servers

For EACH server in the cluster, create or verify a server entry:

Database Server (if running crons):

Dialer 1:

Dialer 2 and 3: Same as Dialer 1 but with their own IPs and Server IDs.

7.3 Conference Bridge Entries

This is one of the most common sources of cluster problems. Each dialer server needs its own range of conference bridge numbers in the database.

Navigate to: Admin > Conferences

Conference ranges per server (each server needs TWO sets):

Standard Conferences (for agent sessions):

ViciDial Conferences (for auto-dial sessions):

You can add these through the admin UI one at a time, or use SQL:

-- Insert conference entries for Dialer 1
-- Standard conferences (conferences table)
INSERT INTO conferences (conf_exten, server_ip, extension)
SELECT seq, '10.0.0.11', ''
FROM (
    SELECT 8600001 + seq.n AS seq
    FROM (
        SELECT a.N + b.N * 10 AS n
        FROM (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4
              UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) a,
             (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 UNION SELECT 3) b
    ) seq
    WHERE 8600001 + seq.n <= 8600039
) t;

-- ViciDial conferences (vicidial_conferences table)
INSERT INTO vicidial_conferences (conf_exten, server_ip, extension, leave_3way)
SELECT seq, '10.0.0.11', '', 'ENABLED'
FROM (
    SELECT 8600051 + seq.n AS seq
    FROM (
        SELECT a.N + b.N * 10 + c.N * 100 AS n
        FROM (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4
              UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) a,
             (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4
              UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) b,
             (SELECT 0 AS N UNION SELECT 1) c
        ) seq
    WHERE 8600051 + seq.n <= 8600250
) t;

-- Repeat for Dialer 2 (10.0.0.12) with ranges 8600040-8600078 and 8600251-8600450
-- Repeat for Dialer 3 (10.0.0.13) with ranges 8600079-8600117 and 8600451-8600650

IMPORTANT: Conference ranges between servers MUST NOT overlap. Overlapping conference numbers cause calls to be placed in the wrong conference on the wrong server, resulting in agents hearing other agents' calls or calls being silently dropped.

7.4 System Settings

Navigate to: Admin > System Settings

Key cluster settings:

7.5 Campaign VDAD Extension

For load-balanced dialing across the cluster:

Navigate to: Admin > Campaigns > (your campaign) > Detail

Extension 8368 = Load Balanced (distributes calls evenly across servers) Extension 8367 = Overflow (fills one server, then overflows to the next)

For most clusters, use 8368 (load balanced).

Also set the same extension on each server: Admin > Servers > (each dialer) > VICIDIAL AD Extension = 8368

8. Extension Configuration

Each dialer server needs specific extensions in its dialplan to handle agent conferences, cross-server transfers, and ViciDial auto-dial operations.

8.1 extensions.conf Structure

ViciDial uses a modular extensions.conf structure:

/etc/asterisk/extensions.conf          -- Main file, includes others
/etc/asterisk/extensions-vicidial.conf -- Auto-generated by ViciDial
/etc/asterisk/customexte.conf          -- Custom extensions (ring groups, etc.)

NEVER edit extensions-vicidial.conf manually -- it is auto-generated by ViciDial and your changes will be overwritten. Put custom extensions in customexte.conf.

8.2 extensions.conf -- Globals and Trunks

The [globals] section defines IAX trunk variables used for cross-server routing. This is where ViciDial's load-balanced dial patterns reference other servers.

On Dialer 1 (10.0.0.11):

[globals]
; Local loopback trunks (auto-generated in iax-vicidial.conf, included here)
; TRUNKloop, TRUNKblind, TRUNKplay are defined by ViciDial

; Cross-server IAX trunks for cluster load balancing
; Format: IAX2/peername:secret@remote_ip
TRUNKIAX2 = IAX2/dialer1:[email protected]
TRUNKIAX3 = IAX2/dialer1:[email protected]

On Dialer 2 (10.0.0.12):

[globals]
TRUNKIAX1 = IAX2/dialer2:[email protected]
TRUNKIAX3 = IAX2/dialer2:[email protected]

On Dialer 3 (10.0.0.13):

[globals]
TRUNKIAX1 = IAX2/dialer3:[email protected]
TRUNKIAX2 = IAX2/dialer3:[email protected]

8.3 Cross-Server Conference Transfer Extensions

These extensions route conference calls to the correct server. ViciDial encodes the target server IP into the extension number using asterisks as delimiters.

The pattern _0XX*0XX*XXX*XXX*8600XXX encodes the IP address with periods replaced by asterisks. ViciDial then dials this pattern to reach a conference on a specific server.

On Dialer 1 (10.0.0.11) -- add to [vicidial-auto-external] or [default]:

; --- CROSS-SERVER CONFERENCE ROUTING ---

; Local server (10.0.0.11) -- route locally via Goto
exten => _010*000*000*011*8600XXX,1,Goto(default,${EXTEN:16},1)
exten => _010*000*000*011*8600XXX,2,Hangup()
exten => _**010*000*000*011*8600XXX,1,Goto(default,${EXTEN:18},1)
exten => _**010*000*000*011*8600XXX,2,Hangup()

; Remote: Dialer 2 (10.0.0.12) -- route via IAX trunk
exten => _010*000*000*012*8600XXX,1,Dial(${TRUNKIAX2}/${EXTEN:16},55,o)
exten => _010*000*000*012*8600XXX,2,Hangup()
exten => _**010*000*000*012*8600XXX,1,Dial(${TRUNKIAX2}/${EXTEN:18},55,o)
exten => _**010*000*000*012*8600XXX,2,Hangup()

; Remote: Dialer 3 (10.0.0.13) -- route via IAX trunk
exten => _010*000*000*013*8600XXX,1,Dial(${TRUNKIAX3}/${EXTEN:16},55,o)
exten => _010*000*000*013*8600XXX,2,Hangup()
exten => _**010*000*000*013*8600XXX,1,Dial(${TRUNKIAX3}/${EXTEN:18},55,o)
exten => _**010*000*000*013*8600XXX,2,Hangup()

; CRITICAL: Conference transfer fix AGI -- must appear on ALL servers
; This MUST come BEFORE the IP-encoded patterns in priority
exten => _8600XXX*.,1,AGI(agi-VDADfixCXFER.agi)
exten => _8600XXX*.,n,Hangup()
exten => _78600XXX*.,1,AGI(agi-VDADfixCXFER.agi)
exten => _78600XXX*.,n,Hangup()

On Dialer 2 (10.0.0.12):

; Local (10.0.0.12) -- route locally
exten => _010*000*000*012*8600XXX,1,Goto(default,${EXTEN:16},1)
exten => _010*000*000*012*8600XXX,2,Hangup()
exten => _**010*000*000*012*8600XXX,1,Goto(default,${EXTEN:18},1)
exten => _**010*000*000*012*8600XXX,2,Hangup()

; Remote: Dialer 1 (10.0.0.11) -- route via IAX
exten => _010*000*000*011*8600XXX,1,Dial(${TRUNKIAX1}/${EXTEN:16},55,o)
exten => _010*000*000*011*8600XXX,2,Hangup()
exten => _**010*000*000*011*8600XXX,1,Dial(${TRUNKIAX1}/${EXTEN:18},55,o)
exten => _**010*000*000*011*8600XXX,2,Hangup()

; Remote: Dialer 3 (10.0.0.13) -- route via IAX
exten => _010*000*000*013*8600XXX,1,Dial(${TRUNKIAX3}/${EXTEN:16},55,o)
exten => _010*000*000*013*8600XXX,2,Hangup()
exten => _**010*000*000*013*8600XXX,1,Dial(${TRUNKIAX3}/${EXTEN:18},55,o)
exten => _**010*000*000*013*8600XXX,2,Hangup()

; Conference transfer fix AGI
exten => _8600XXX*.,1,AGI(agi-VDADfixCXFER.agi)
exten => _8600XXX*.,n,Hangup()
exten => _78600XXX*.,1,AGI(agi-VDADfixCXFER.agi)
exten => _78600XXX*.,n,Hangup()

On Dialer 3 (10.0.0.13) -- same pattern, swap local/remote accordingly.

NOTE ON IP ENCODING: The pattern uses the IP address with dots replaced by asterisks. For the IP 10.0.0.11, the pattern becomes 010*000*000*011. For a public IP like 192.168.1.100, it becomes 192*168*001*100. Each octet is zero-padded to 3 digits. The ${EXTEN:16} strips the first 16 characters (the IP encoding) to extract just the conference extension (e.g., 8600123).

ViciDial auto-generates most of these patterns in extensions-vicidial.conf. Verify they exist by checking the [vicidial-auto-external] context.

8.4 VDAD Extensions (Load Balanced / Overflow)

These are the extensions that handle auto-dial call routing. They MUST exist on every dialer server in the [default] context:

; Load Balanced auto-dial (extension 8368)
; Calls this AGI 3 times for retry resilience
exten => 8368,1,AGI(agi://127.0.0.1:4577/call_log)
exten => 8368,2,AGI(agi-VDAD_LB_transfer.agi,${EXTEN})
exten => 8368,3,AGI(agi-VDAD_LB_transfer.agi,${EXTEN})
exten => 8368,4,AGI(agi-VDAD_LB_transfer.agi,${EXTEN})
exten => 8368,5,Hangup()

; Overflow auto-dial (extension 8367)
exten => 8367,1,AGI(agi://127.0.0.1:4577/call_log)
exten => 8367,2,AGI(agi-VDAD_LO_transfer.agi,${EXTEN})
exten => 8367,3,AGI(agi-VDAD_LO_transfer.agi,${EXTEN})
exten => 8367,4,AGI(agi-VDAD_LO_transfer.agi,${EXTEN})
exten => 8367,5,Hangup()

Note: ViciDial auto-generates these in extensions-vicidial.conf. Verify they exist. If they are missing, re-run ADMIN_keepalive_ALL.pl which triggers a dialplan rebuild.

8.5 Conference Bridge Extensions (MeetMe / ConfBridge)

Each dialer needs MeetMe or ConfBridge entries for its conference range:

For MeetMe (Asterisk 16-18 with DAHDI):

; Conference bridges for agent sessions
; Range must match what is configured in the ViciDial admin conferences table
exten => _8600XXX,1,Answer()
exten => _8600XXX,n,MeetMe(${EXTEN},Fq)
exten => _8600XXX,n,Hangup()

For ConfBridge (Asterisk 18+, preferred):

; ConfBridge conference bridges
exten => _9600XXX,1,Answer()
exten => _9600XXX,n,Playback(sip-silence)
exten => _9600XXX,n,ConfBridge(${EXTEN},vici_agent_bridge,vici_agent_user)
exten => _9600XXX,n,Hangup()

ConfBridge requires a confbridge.conf configuration:

; /etc/asterisk/confbridge.conf

[vici_agent_bridge]
type = bridge
max_members = 10

[vici_agent_user]
type = user
quiet = yes
announce_user_count = no
music_on_hold_when_empty = no

[vici_audio_user]
type = user
quiet = yes
startmuted = yes

[vici_monitor_user]
type = user
quiet = yes
startmuted = yes

[vici_barge_user]
type = user
quiet = yes

[vici_monitor_menu]
type = menu

8.6 Other Required Extensions

These should exist on every dialer (most are auto-generated by ViciDial):

; Playback of recorded prompts
exten => _851XXXXX,1,Answer()
exten => _851XXXXX,n,Playback(${EXTEN})
exten => _851XXXXX,n,Hangup()

; AMD message playback
exten => _6851XXXXX,1,Answer()
exten => _6851XXXXX,n,WaitForSilence(2000,1,90)
exten => _6851XXXXX,n,Playback(sip-silence)
exten => _6851XXXXX,n,Playback(${EXTEN:1})
exten => _6851XXXXX,n,Hangup()

; Blind monitoring
exten => _08600XXX,1,Dial(${TRUNKblind}/6${EXTEN:1},55,To)
exten => _08600XXX,n,Hangup()

; Barge monitoring
exten => 8159,1,ChanSpy(,bBq)
exten => 8159,n,Hangup()

; FastAGI call logging (hangup handler) -- in EVERY context that handles calls
exten => h,1,AGI(agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----${HANGUPCAUSE}-----${DIALSTATUS}-----${DIALEDTIME}-----${ANSWEREDTIME}-----${HANGUPCAUSE(${HANGUPCAUSE_KEYS()},tech)}))

8.7 Apply Extension Changes

After editing extensions.conf or customexte.conf:

# Reload the dialplan without restarting Asterisk
asterisk -rx "dialplan reload"

# Verify extensions are loaded
asterisk -rx "dialplan show default" | grep 8368
asterisk -rx "dialplan show default" | grep 8600

9. Phone Registration and Load Balancing

9.1 How Phones Are Distributed

In a ViciDial cluster, each SIP phone (softphone or hardphone) registers to ONE specific Asterisk/dialer server. The server_ip field in the phone record determines which server owns that phone.

Agent A phone → registers to Dialer 1 (10.0.0.11)
Agent B phone → registers to Dialer 2 (10.0.0.12)
Agent C phone → registers to Dialer 1 (10.0.0.11)
Agent D phone → registers to Dialer 3 (10.0.0.13)

ViciDial's load balancing (ext 8368) ensures that calls are distributed across all active dialer servers, regardless of where the agent's phone is registered. The IAX trunks handle routing the audio to the correct server.

9.2 Creating Phone Entries

Navigate to: Admin > Phones > Add a Phone

Distribute phones evenly across servers. If you have 150 phones and 3 dialer servers, assign approximately 50 phones to each server.

9.3 Balanced Phone Distribution Strategy

Dialer 1 (10.0.0.11): Phones 100-149 (50 phones)
Dialer 2 (10.0.0.12): Phones 150-199 (50 phones)
Dialer 3 (10.0.0.13): Phones 200-249 (50 phones)

This can be done via SQL for bulk creation:

-- Example: Create 50 phone entries for Dialer 2
INSERT INTO phones (extension, dialplan_number, voicemail_id, phone_ip,
    computer_ip, server_ip, login, pass, status, active, phone_type,
    fullname, company, picture, protocol, local_gmt, ASTmgrUSERNAME,
    ASTmgrSECRET, login_user, login_pass, login_campaign, conf_exten,
    phone_context, phone_ring_timeout, conf_override, codecs_list,
    codecs_with_template, carrier_id, phone_context_a_b)
VALUES
('150', '150', '150', '', '', '10.0.0.12', 'cc150', 'P@ss150!',
 'ACTIVE', 'Y', 'SIP', 'Agent Phone 150', '', '', 'SIP', '-5.00',
 'cron', '1234', '', '', '', '8600150', 'default', '60', '',
 '', '', '', 'default---agent');
-- Repeat for 151-199...

9.4 SIP Peer Auto-Generation

ViciDial auto-generates SIP peer entries in /etc/asterisk/sip-vicidial.conf based on the phones table. Only phones with a server_ip matching the local server's IP appear in that server's SIP config.

To regenerate SIP peers:

# This happens automatically via the keepalive process, but you can force it:
/usr/share/astguiclient/ADMIN_keepalive_ALL.pl --justSIPpeers

# Then reload SIP:
asterisk -rx "sip reload"

# Verify peers are loaded:
asterisk -rx "sip show peers" | head -20

9.5 How Load Balanced Dialing Works

When a campaign is set to VDAD extension 8368 (load balanced):

1. The hopper script (AST_VDhopper.pl) populates leads for dialing
2. The auto-dial script (AST_VDauto_dial) on EACH dialer picks up leads
3. Each dialer originates calls from its own Asterisk instance
4. When a call is answered (live human detected), the call enters extension 8368
5. The agi-VDAD_LB_transfer.agi script checks ALL servers for available agents
6. If the best available agent is on a different server, the call is transferred via IAX trunk to that server's conference bridge
7. The agent hears the call in their conference, regardless of which server originated the call

This is why IAX trunks and cross-server conference routing are essential.

9.6 Agent Login and Server Selection

When an agent logs into the ViciDial agent screen:

1. They enter their phone login (e.g., cc150)
2. ViciDial looks up the phone's server_ip in the database
3. The agent session is associated with that server
4. The agent's conference bridge is created on that server
5. All call audio for that agent flows through that server

The agent connects to the web server URL, but their telephony runs on their assigned dialer server.

10. Testing and Validation

After completing the cluster setup, run through this checklist before going live with production traffic.

10.1 Database Connectivity Test

Run from EVERY non-DB server:

# From each web server and dialer:
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    SELECT server_ip, active_asterisk_server, active_agent_server
    FROM asterisk.servers
    ORDER BY server_ip;"

# Expected: Shows all registered servers with correct active flags

10.2 Asterisk Process Verification

Run on EACH dialer server:

# Check screen sessions (ViciDial runs processes in screen)
screen -ls

# Expected output (varies by VARactive_keepalives):
# ASTupdate    (AST_update)
# ASTsend      (AST_send_listen)
# ASTVDauto    (AST_VDauto_dial)
# ASTremote    (AST_VDremote_agents)
# ASTVDadapt   (AST_VDadapt)           -- Only on primary dialer
# ASTfastlog   (FastAGI_log)
# ASTVDautofill (AST_VDauto_dial_FILL) -- Only on primary dialer
# ASTconf3way  (ConfBridge or MeetMe watcher)

# Verify FastAGI is listening
ss -tlnp | grep 4577

# Verify Asterisk AMI is accessible
ss -tlnp | grep 5038

# Check Asterisk is running
asterisk -rx "core show version"

10.3 IAX Trunk Verification

On EACH dialer:

# Check IAX peers
asterisk -rx "iax2 show peers"
# All cluster peers should show OK status

# Check IAX registrations
asterisk -rx "iax2 show registry"
# All registrations should show "Registered"

# Test a cross-server audio path
# From Dialer 1, call a test extension on Dialer 2:
asterisk -rx "channel originate IAX2/dialer2:[email protected]/s application Echo"

10.4 Conference Bridge Test

# Verify conference extensions are loaded
asterisk -rx "dialplan show default" | grep "8600"

# Check conference entries in database
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    SELECT server_ip, COUNT(*) as conf_count
    FROM asterisk.vicidial_conferences
    GROUP BY server_ip;"

# Expected: Each dialer IP with its conference count (200 each)

10.5 Agent Login Test

1. Open the agent interface: http://10.0.0.2/vicidial/agent.php
2. Log in with a test agent account
3. Select a phone registered to Dialer 1
4. Verify the phone rings (registration working)
5. Answer the phone -- agent should be in "READY" state
6. Repeat with phones on Dialer 2 and Dialer 3

10.6 Cross-Server Transfer Test

1. Log in Agent A on a phone registered to Dialer 1
2. Log in Agent B on a phone registered to Dialer 2
3. Make a test call that connects to Agent A
4. Have Agent A transfer the call to Agent B
5. Verify Agent B receives the call with clear audio
6. Check that the transfer went through the IAX trunk (check Asterisk CLI)

10.7 Load Balancing Verification

# Set max_vicidial_trunks to 0 on Dialer 1 (disables its auto-dial)
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    UPDATE asterisk.servers
    SET max_vicidial_trunks = 0
    WHERE server_ip = '10.0.0.11';"

# Start a test campaign and verify calls originate from Dialer 2 and 3 only

# Re-enable Dialer 1
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    UPDATE asterisk.servers
    SET max_vicidial_trunks = 60
    WHERE server_ip = '10.0.0.11';"

10.8 Recording Test

1. Make a test call that gets recorded
2. Verify the recording file appears on the dialer that handled the call
3. Access the recording through the web interface
4. Confirm audio quality is acceptable

10.9 Full Checklist

[ ] Database accessible from all servers
[ ] All servers appear in ViciDial Admin > Servers
[ ] Conference entries exist for each server (conferences + vicidial_conferences)
[ ] IAX peers show OK status on all dialers
[ ] FastAGI (port 4577) listening on all dialers
[ ] Screen processes running on all dialers
[ ] Agent can log in on each dialer server's phones
[ ] Cross-server transfers work
[ ] Recordings are accessible from web interface
[ ] NTP synchronized (< 1 second drift between all nodes)
[ ] Cron jobs installed on correct servers
[ ] VARactive_keepalives correct per server role
[ ] VDAD extension 8368 set in campaigns and server settings
[ ] Firewall rules allow inter-server communication

11. Adding More Servers

11.1 Adding a 4th Dialer Server

When your cluster needs more capacity, adding a new dialer is straightforward:

Step 1: Provision the server

Install the OS (ViciBox 12 or AlmaLinux 9) and Asterisk. Configure /etc/astguiclient.conf with:

• VARserver_ip = new server's IP (e.g., 10.0.0.14)
• VARDB_server = database server IP (10.0.0.1)
• VARactive_keepalives = 12346C (secondary dialer, no singletons)

Step 2: Database grants

CREATE USER 'cron'@'10.0.0.14' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.14';
CREATE USER 'custom'@'10.0.0.14' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.14';
FLUSH PRIVILEGES;

Step 3: Register in ViciDial admin

Add the server in Admin > Servers with:

• Server IP: 10.0.0.14
• Active Asterisk Server: Y
• Active Agent Server: Y
• Max ViciDial Trunks: 60

Step 4: Allocate conference range

Add non-overlapping conference ranges:

-- Standard conferences
INSERT INTO conferences (conf_exten, server_ip, extension)
VALUES (8600118, '10.0.0.14', ''),
       (8600119, '10.0.0.14', ''),
       -- ... through 8600156
       ;

-- ViciDial conferences
INSERT INTO vicidial_conferences (conf_exten, server_ip, extension, leave_3way)
VALUES (8600651, '10.0.0.14', '', 'ENABLED'),
       (8600652, '10.0.0.14', '', 'ENABLED'),
       -- ... through 8600850
       ;

Step 5: Configure IAX peers

Add the new server's IAX peer to ALL existing dialers' iax.conf:

[dialer4]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.14
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

Add all existing servers as peers on the new server's iax.conf.

Step 6: Add cross-server extensions

Add the new server's IP-encoded routing patterns to ALL existing dialers' extensions.conf. Add all existing server patterns to the new server.

Step 7: Add phones

Create phone entries with server_ip = 10.0.0.14 and redistribute some existing phones if desired.

Step 8: Install cron, start services, test

Install the secondary dialer cron entries. Start Asterisk and the keepalive. Run through the validation checklist from Section 10.

11.2 Conference Range Planning for Growth

Plan conference ranges with room to grow:

Dialer 1: Conferences 8600001-8600039,  ViciDial 8600051-8600250   (200 each)
Dialer 2: Conferences 8600040-8600078,  ViciDial 8600251-8600450   (200 each)
Dialer 3: Conferences 8600079-8600117,  ViciDial 8600451-8600650   (200 each)
Dialer 4: Conferences 8600118-8600156,  ViciDial 8600651-8600850   (200 each)
Dialer 5: Conferences 8600157-8600195,  ViciDial 8600851-8601050   (200 each)

Each server needs enough ViciDial conferences for its maximum concurrent agent sessions plus auto-dial channels. A safe formula:

conferences_needed = (max_agents_on_server * 2) + max_auto_dial_trunks + 50

11.3 Adding a Second Web Server

For high availability, run two web servers behind a load balancer:

1. Clone the web server configuration to a new server
2. Both point to the same database (10.0.0.1)
3. Use HAProxy or nginx as a load balancer in front of both
4. Ensure PHP sessions are shared (use database or memcached session handler)
5. Both servers need access to recordings (NFS mount or shared storage)

12. Monitoring and Maintenance

12.1 Health Checks

Database Server:

# Check MariaDB is running and accepting connections
mysqladmin -u cron -p'Str0ng_Cr0n_P@ss!' ping
# Should return: mysqld is alive

# Check connection count
mysql -u cron -p'Str0ng_Cr0n_P@ss!' -e "SHOW STATUS LIKE 'Threads_connected';"

# Check slow queries
mysql -u cron -p'Str0ng_Cr0n_P@ss!' -e "SHOW STATUS LIKE 'Slow_queries';"

Dialer Servers:

# Check Asterisk is running
asterisk -rx "core show version"

# Check active channels
asterisk -rx "core show channels count"

# Check IAX trunk status
asterisk -rx "iax2 show peers" | grep -v "OK" | grep -v "^Name"
# Should return nothing (all peers OK)

# Check screen processes
screen -ls | grep -c "Detached"
# Should match the number of keepalive processes configured

# Check FastAGI
ss -tlnp | grep 4577

Web Server:

# Check Apache is running
systemctl is-active apache2 || systemctl is-active httpd

# Check ViciDial admin page responds
curl -s -o /dev/null -w "%{http_code}" http://localhost/vicidial/admin.php
# Should return 200

12.2 Automated Health Check Script

Create /usr/local/bin/cluster-health.sh on each server:

#!/bin/bash
# ViciDial Cluster Health Check
# Run every 5 minutes via cron

ROLE="${1:-dialer}"  # dialer, db, or web
DB_HOST="10.0.0.1"
DB_USER="cron"
DB_PASS="Str0ng_Cr0n_P@ss!"
ALERT_EMAIL="[email protected]"

ERRORS=""

# Check 1: Database connectivity
if ! mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASS" -e "SELECT 1" &>/dev/null; then
    ERRORS+="CRITICAL: Cannot connect to database at $DB_HOST\n"
fi

# Check 2: NTP sync
OFFSET=$(chronyc tracking 2>/dev/null | grep "Last offset" | awk '{print $4}' | tr -d '+')
if [ -n "$OFFSET" ]; then
    # Alert if offset > 2 seconds
    ABS_OFFSET=$(echo "$OFFSET" | tr -d '-')
    if (( $(echo "$ABS_OFFSET > 2" | bc -l 2>/dev/null || echo "0") )); then
        ERRORS+="WARNING: NTP offset is ${OFFSET}s\n"
    fi
fi

if [ "$ROLE" = "dialer" ]; then
    # Check 3: Asterisk running
    if ! asterisk -rx "core show version" &>/dev/null; then
        ERRORS+="CRITICAL: Asterisk is not running\n"
    fi

    # Check 4: FastAGI listening
    if ! ss -tlnp | grep -q ":4577"; then
        ERRORS+="CRITICAL: FastAGI (port 4577) is not listening\n"
    fi

    # Check 5: IAX peers
    DOWN_PEERS=$(asterisk -rx "iax2 show peers" 2>/dev/null | grep "UNREACHABLE\|UNKNOWN" | wc -l)
    if [ "$DOWN_PEERS" -gt 0 ]; then
        ERRORS+="WARNING: $DOWN_PEERS IAX peers are down\n"
    fi

    # Check 6: Screen processes
    EXPECTED_SCREENS=6  # Adjust based on VARactive_keepalives
    ACTUAL_SCREENS=$(screen -ls 2>/dev/null | grep -c "Detached")
    if [ "$ACTUAL_SCREENS" -lt "$EXPECTED_SCREENS" ]; then
        ERRORS+="WARNING: Only $ACTUAL_SCREENS of $EXPECTED_SCREENS screen sessions running\n"
    fi
fi

if [ "$ROLE" = "web" ]; then
    # Check 7: Apache running
    if ! systemctl is-active apache2 &>/dev/null && ! systemctl is-active httpd &>/dev/null; then
        ERRORS+="CRITICAL: Apache is not running\n"
    fi
fi

# Report errors
if [ -n "$ERRORS" ]; then
    HOSTNAME=$(hostname)
    echo -e "Health check failures on $HOSTNAME:\n$ERRORS" | \
        mail -s "ViciDial Cluster Alert: $HOSTNAME" "$ALERT_EMAIL" 2>/dev/null
    echo -e "ERRORS:\n$ERRORS"
    exit 1
fi

echo "OK: All checks passed"
exit 0

chmod +x /usr/local/bin/cluster-health.sh

# Add to cron on each server (adjust role parameter):
# */5 * * * * /usr/local/bin/cluster-health.sh dialer
# */5 * * * * /usr/local/bin/cluster-health.sh db
# */5 * * * * /usr/local/bin/cluster-health.sh web

12.3 Log Aggregation

In a cluster, logs are scattered across servers. Centralize them:

Option A: Promtail + Loki (recommended)

Install Promtail on each server, point to a central Loki instance:

# /etc/promtail/promtail.yml (on each server)
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://10.0.0.2:3100/loki/api/v1/push

scrape_configs:
  - job_name: asterisk
    static_configs:
      - targets: [localhost]
        labels:
          job: asterisk
          host: vici-dial1
          __path__: /var/log/asterisk/messages

  - job_name: astguiclient
    static_configs:
      - targets: [localhost]
        labels:
          job: astguiclient
          host: vici-dial1
          __path__: /var/log/astguiclient/*.log

Option B: rsyslog forwarding

Configure each server to forward logs to a central syslog server:

# On each ViciDial server, add to /etc/rsyslog.conf:
*.* @10.0.0.2:514

12.4 Backup Strategy for Clusters

Database Server (most critical):

# Daily full backup with mysqldump
0 2 * * * /usr/bin/mysqldump -u root --single-transaction --routines --triggers \
    asterisk | gzip > /backup/vicidial-db-$(date +\%Y\%m\%d).sql.gz

# Keep 14 days of backups
0 3 * * * find /backup -name "vicidial-db-*.sql.gz" -mtime +14 -delete

Dialer Servers:

# Backup configuration files
0 2 * * * tar czf /backup/config-$(hostname)-$(date +\%Y\%m\%d).tar.gz \
    /etc/asterisk/ /etc/astguiclient.conf /etc/crontab \
    /var/spool/cron/tabs/root 2>/dev/null

Recording archival:

Use AST_CRON_audio_3_ftp.pl to transfer recordings to a central storage server or S3-compatible object storage.

12.5 Rolling Updates

When updating ViciDial across a cluster:

1. Stop campaigns (pause all auto-dialing)
2. Update the DB server first -- SVN update + run install.pl
3. Update one dialer at a time -- SVN update + install.pl, verify, then move to the next
4. Update web servers last
5. Verify ExpectedDBSchema matches on all servers
6. Resume campaigns

NEVER update multiple servers simultaneously. Update one at a time and verify each before proceeding.

13. Troubleshooting

13.1 Common Problems and Solutions

Problem: "DB connection failed" on dialer/web servers

Symptoms: Agent screens show errors, cron scripts fail.

Check:

# Test connectivity from the failing server
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "SELECT 1;"

# If it fails, check:
# 1. Firewall on DB server
iptables -L -n | grep 3306

# 2. MariaDB bind address
grep bind-address /etc/my.cnf

# 3. User grants
mysql -u root -p -e "SELECT user, host FROM mysql.user WHERE user='cron';"

# 4. Network connectivity
ping -c 3 10.0.0.1
telnet 10.0.0.1 3306

Fix: Ensure bind-address = 0.0.0.0 in my.cnf, verify GRANT statements match the connecting server's IP, and confirm firewall allows TCP 3306.

Problem: Time sync drift between servers

Symptoms: Calls show wrong durations, agent sessions overlap, conference bridges behave unexpectedly.

Check:

# On each server, run simultaneously and compare:
date +%s.%N

# Check chrony status
chronyc tracking
chronyc sources

# Check if chrony is running
systemctl status chronyd

Fix:

# Force immediate sync
chronyc makestep

# If drift persists, restart chrony
systemctl restart chronyd

Problem: IAX trunk shows UNREACHABLE

Symptoms: Cross-server transfers fail, calls get stuck.

Check:

# On the affected dialer
asterisk -rx "iax2 show peers"
asterisk -rx "iax2 show registry"

# Check UDP 4569 connectivity
nmap -sU -p 4569 10.0.0.12

# Check firewall
iptables -L -n | grep 4569

Fix:

• Verify IAX peer definitions match on both ends (same secret, correct IPs)
• Open UDP 4569 in firewall on both servers
• Reload IAX: asterisk -rx "iax2 reload"

Problem: Conference bridge mismatches / calls going to wrong conference

Symptoms: Agents hear other agents' calls, calls silently drop after answering, delayed hangups.

Check:

-- Check for overlapping conference ranges
SELECT conf_exten, GROUP_CONCAT(server_ip) as servers, COUNT(*) as cnt
FROM vicidial_conferences
GROUP BY conf_exten
HAVING cnt > 1;
-- Should return ZERO rows

-- Check conference assignments per server
SELECT server_ip, MIN(conf_exten), MAX(conf_exten), COUNT(*)
FROM vicidial_conferences
GROUP BY server_ip;

Fix: Ensure each conference number is assigned to exactly ONE server IP. Delete duplicates and reassign non-overlapping ranges per Section 7.3.

Problem: Recording transfer failures

Symptoms: Recordings exist on the dialer but are not playable from the web server.

Check:

# On the dialer, verify recordings exist
ls -la /var/spool/asterisk/monitorDONE/MP3/ | tail -5

# On the web server, check if recordings are accessible
ls -la /mnt/recordings-dial1/ 2>/dev/null || echo "NFS mount missing"

# Check FTP transfer cron
grep "audio_3_ftp" /var/spool/cron/tabs/root

Fix: Either set up NFS mounts (Section 4.7) or enable the FTP transfer cron job (AST_CRON_audio_3_ftp.pl). Verify VARHTTP_path in astguiclient.conf points to the correct URL for each dialer.

Problem: Hopper not filling / no leads being dialed

Symptoms: Campaign shows 0 dialable leads, hopper count is 0.

Check:

# Is the hopper cron running? (Should be on DB server only)
ps aux | grep VDhopper

# Check hopper directly
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    SELECT campaign_id, COUNT(*) FROM asterisk.vicidial_hopper
    GROUP BY campaign_id;"

# Check for hopper errors
tail -50 /var/log/astguiclient/hopper.log

Fix: Ensure AST_VDhopper.pl runs on exactly ONE server (DB server recommended). Verify the campaign has dialable leads with correct GMT offsets.

Problem: SIP response 481 "Call Leg/Transaction Does Not Exist"

Symptoms: Calls fail during transfer or conference join. Seen in Asterisk console output.

Cause: IAX trunk configuration mismatch or conference routing patterns point to wrong server.

Fix:

1. Verify IAX trunk variables in [globals] match the correct peer definitions
2. Verify cross-server extension patterns use correct IP encoding
3. Ensure agi-VDADfixCXFER.agi exists and is executable on all dialers:

   ls -la /var/lib/asterisk/agi-bin/agi-VDADfixCXFER.agi
   

Problem: Agents show as logged in on wrong server

Symptoms: Agent appears active on a server they are not registered to.

Check:

SELECT user, server_ip, status, campaign_id
FROM vicidial_live_agents
WHERE user = 'agent123';

Fix: The agent's phone server_ip in the phones table must match the server they are connecting through. Update the phone record:

UPDATE phones SET server_ip = '10.0.0.12' WHERE login = 'cc150';

Then have the agent log out and back in.

Problem: AST_VDadapt or AST_VDauto_dial_FILL running on multiple servers

Symptoms: Double-dialing, erratic dial rates, agents getting multiple calls simultaneously.

Check:

# Run on EVERY dialer:
grep "VARactive_keepalives" /etc/astguiclient.conf

Fix: Ensure process codes 5, 7, 9, and E appear in VARactive_keepalives on ONLY ONE server. Remove them from all other servers and restart the keepalive:

# Kill the duplicate process
screen -ls | grep "VDadapt\|VDautofill"
# Kill the screen session if found on a secondary server

# Remove from keepalives in /etc/astguiclient.conf
# Change: VARactive_keepalives => 12345679C
# To:     VARactive_keepalives => 12346C

13.2 Diagnostic Commands Quick Reference

# === DATABASE ===
# Check connections from cluster nodes
mysql -e "SELECT substring_index(host,':',1) as client_ip, COUNT(*) as connections
    FROM information_schema.processlist
    GROUP BY client_ip;"

# Check replication lag (if using replicas)
mysql -e "SHOW SLAVE STATUS\G" | grep -E "Seconds_Behind|IO_Running|SQL_Running"

# === ASTERISK ===
# Show active channels
asterisk -rx "core show channels"

# Show SIP peers with status
asterisk -rx "sip show peers" | grep -v "OK" | grep -v "^Name"

# Show IAX peers
asterisk -rx "iax2 show peers"

# Show active conferences (MeetMe)
asterisk -rx "meetme list"

# Show active conferences (ConfBridge)
asterisk -rx "confbridge list"

# Check dialplan for specific extension
asterisk -rx "dialplan show 8368@default"

# === VICIDIAL PROCESSES ===
# Show all screen sessions
screen -ls

# Attach to a specific screen (read-only, safe)
screen -x ASTVDauto

# Check keepalive log
tail -50 /var/log/astguiclient/keepalive.log

# Check auto-dial log
tail -50 /var/log/astguiclient/VDauto_dial.log

# === NETWORK ===
# Check for established DB connections
ss -tn | grep ":3306"

# Check IAX connections
ss -un | grep ":4569"

# Test latency to DB
ping -c 10 10.0.0.1 | tail -1

13.3 Emergency: Isolating a Failed Server

If a dialer server crashes or becomes unresponsive:

-- Disable the failed server in the database
-- Run from the DB server or any working server with DB access
UPDATE servers SET
    active_asterisk_server = 'N',
    active_agent_server = 'N',
    max_vicidial_trunks = 0
WHERE server_ip = '10.0.0.12';

-- Move agents off the failed server (they will need to re-login)
UPDATE vicidial_live_agents SET
    status = 'PAUSED'
WHERE server_ip = '10.0.0.12';

The remaining dialer servers will continue handling calls. Agents on the failed server will need to log out and log back in using phones registered to a working server.

Appendix A: Complete File Reference

Appendix B: Port Reference

Appendix C: VARactive_keepalives Quick Reference

Appendix D: Version Compatibility Matrix

IMPORTANT: All servers in a cluster MUST run the same ExpectedDBSchema version. If one server has a newer schema, all others must be updated to match.

Appendix E: Capacity Planning Cheat Sheet

Concurrent Agents    Dialer Servers    DB Server RAM    DB Server CPU
     1-50                 1               8-16 GB          4 cores
    50-100                2               16-32 GB         8 cores
   100-200                3               32-64 GB         8-16 cores
   200-400                4-6             64-128 GB        16-32 cores
   400+                   6+              128 GB+          32+ cores

Rules of thumb:
- 1 dialer server per 50-75 concurrent agents
- InnoDB buffer pool = 60-70% of DB server RAM
- 200 vicidial_conferences per dialer server
- SIP phones per dialer: max 100 (recommended 50-75)
- IAX trunk capacity: ~1000 concurrent calls per trunk
- DB queries per agent: ~20-40 per second during active dialing
- Total DB connections: ~5-10 per dialer + ~5 per web server + cron

This tutorial consolidates information from the ViciDial forum thread t=4545 (66 replies on multi-server setup), ViciBox cluster documentation, production cluster experience, and the official ViciDial SVN source code. All commands and configurations are verified against ViciBox 12, Asterisk 18/20, and MariaDB 10.5-10.11.

For questions, the ViciDial community forum at vicidial.org is the primary support resource. The PoundTeam offers commercial support for complex cluster deployments.