ViciDial Safe Upgrade Playbook

The step-by-step guide to upgrading ViciDial without killing your call center

Table of Contents

1. Why Upgrade
2. Understanding ViciDial Versions
3. Pre-Upgrade Checklist
4. SVN Code Upgrade
5. Database Schema Upgrade
6. Perl/PHP Script Installation
7. Post-Upgrade Verification
8. ViciBox Major Version Upgrades
9. CentOS 7 Migration
10. Multi-Server Cluster Upgrades
11. Rollback Procedures
12. Common Upgrade Failures and Fixes
13. Automation Script
14. Quick Reference Card
15. Sources and Further Reading

1. Why Upgrade

The Uncomfortable Truth

The ViciDial forum thread on SVN upgrades (viewtopic.php?t=16326) has 88+ replies of users who encountered blank admin pages, missing settings, and dead call flows after upgrading. The fear is justified -- but NOT upgrading is also dangerous.

Security Vulnerabilities

ViciDial has 13 documented CVEs as of 2025, including critical RCE and SQL injection:

Critical Severity

High Severity

Moderate Severity

If your ViciDial has not been updated since September 2024, CVE-2024-8503 means any attacker can dump your entire database without a password. Exploit code is publicly available.

Operating System End-of-Life

Running ViciDial on a dead OS means unpatched kernel, OpenSSL, and MariaDB vulnerabilities.

New Features in Recent Versions

• DNC compliance improvements (TCPA/Ofcom)
• WebRTC agent support (browser-based softphones)
• Enhanced reporting and API methods
• PHP 8 compatibility
• Hundreds of bug fixes to call handling, recording, and reporting

Risk Assessment Matrix

Bottom line: The risk of NOT upgrading almost always exceeds the risk of upgrading, provided you follow this playbook.

2. Understanding ViciDial Versions

ViciDial has multiple version numbers that move independently.

Version Components

VERSION: 2.14-871a    BUILD: 221230-2231    SVN Version: 3703    DB Schema Version: 1729

SVN Revision Milestones

DB Schema Thresholds

When code expects schema 1729 but your database is at 1359, you get "schema mismatch" warnings and broken functionality.

ViciBox Versions

ViciBox is the official appliance/distribution bundling OS + Asterisk + ViciDial:

Types of Upgrades

Most routine upgrades are Type 1 or Type 2.

3. Pre-Upgrade Checklist

Complete every item before starting. Skipping any step is how upgrades go wrong.

3.1 Document Current State

# Run on EVERY server in your cluster
echo "=== Server: $(hostname) ==="
echo "=== Date: $(date) ==="

# ViciDial version
echo "--- ViciDial Version ---"
cd /usr/src/astguiclient/trunk
svn info 2>/dev/null || echo "No SVN info available"

# DB Schema version (run on DB server only)
echo "--- DB Schema ---"
mysql -u root -p -e "SELECT db_schema_version, db_schema_update_date FROM asterisk.system_settings;"

# Asterisk version
echo "--- Asterisk ---"
asterisk -V

# OS version
echo "--- OS ---"
cat /etc/os-release 2>/dev/null || cat /etc/centos-release 2>/dev/null

# PHP version
echo "--- PHP ---"
php -v | head -1

# MariaDB/MySQL version
echo "--- MariaDB ---"
mysql -V

# Kernel
echo "--- Kernel ---"
uname -r

# Installed RPMs (ViciDial-related)
echo "--- ViciDial RPMs ---"
rpm -qa | grep -i -E "vici|asterisk|dahdi" 2>/dev/null

# Disk space
echo "--- Disk Space ---"
df -h /

# Database size
echo "--- Database Size ---"
mysql -u root -p -e "SELECT table_schema, ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS 'Size_MB' FROM information_schema.TABLES GROUP BY table_schema ORDER BY Size_MB DESC;"

3.2 Full Database Backup

DANGER: This is your lifeline. If the backup fails or is incomplete, DO NOT proceed.

mkdir -p /root/upgrade-backup-$(date +%Y%m%d)
cd /root/upgrade-backup-$(date +%Y%m%d)

# Full dump (no lock, consistent snapshot)
mysqldump -u root -p --single-transaction --routines --triggers --events \
  --all-databases --flush-logs | gzip > all-databases-$(date +%Y%m%d-%H%M).sql.gz

# Asterisk database specifically
mysqldump -u root -p --single-transaction --routines --triggers \
  asterisk | gzip > asterisk-db-$(date +%Y%m%d-%H%M).sql.gz

# Verify integrity
gunzip -t all-databases-*.sql.gz && echo "GZIP OK" || echo "GZIP CORRUPT!"
gunzip -t asterisk-db-*.sql.gz && echo "GZIP OK" || echo "GZIP CORRUPT!"

CRITICAL: Test the Backup

The forum thread (t=16326) repeatedly emphasizes: restore your backup to a test database to verify it works.

# On the same server or a test server
mysql -u root -p -e "CREATE DATABASE asterisk_backup_test;"
zcat asterisk-db-*.sql.gz | mysql -u root -p asterisk_backup_test

# Verify recent data exists
mysql -u root -p -e "SELECT call_date, COUNT(*) FROM asterisk_backup_test.vicidial_closer_log WHERE call_date >= CURDATE() - INTERVAL 1 DAY GROUP BY DATE(call_date);"

# Clean up test database
mysql -u root -p -e "DROP DATABASE asterisk_backup_test;"

If the restore fails or today's data is missing, STOP. Fix your backup before proceeding.

3.3 Configuration File Backup

BACKUP_DIR="/root/upgrade-backup-$(date +%Y%m%d)/configs"
mkdir -p "$BACKUP_DIR"
cp -a /etc/asterisk/ "$BACKUP_DIR/asterisk-etc/"
cp /etc/astguiclient.conf "$BACKUP_DIR/"
cp -a /var/lib/asterisk/agi-bin/ "$BACKUP_DIR/agi-bin/"
crontab -l > "$BACKUP_DIR/crontab-root.txt"
cp -a /etc/apache2/ "$BACKUP_DIR/apache2/" 2>/dev/null
cp -a /etc/httpd/ "$BACKUP_DIR/httpd/" 2>/dev/null

3.4 Disk Space Check

df -h / /var /tmp
mysql -u root -p -e "SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024 / 1024, 2) AS 'DB_Size_GB' FROM information_schema.TABLES WHERE table_schema = 'asterisk';"

DANGER: You need at least 3x your database size free (1x backup, 1x temp tables during ALTER, 1x safety margin). Running out of disk mid-ALTER TABLE can corrupt your database.

3.5 Pre-Upgrade Checklist Summary

[ ] Current versions documented (ViciDial, Asterisk, OS, PHP, MariaDB, SVN rev, DB schema)
[ ] Database backup completed, verified, and test-restored
[ ] Config files backed up (/etc/asterisk/, astguiclient.conf, crontab)
[ ] Disk space sufficient (3x database size minimum)
[ ] Maintenance window scheduled (2-4 hours minimum)
[ ] Agents and managers notified
[ ] SVN accessible (test: svn info svn://svn.eflo.net/agc_2-X/trunk)
[ ] Rollback plan understood
[ ] Test clone available (strongly recommended for major upgrades)

3.6 Setting Up a Test Clone

For major upgrades, test on a clone first (VM, 4 CPU / 8 GB RAM minimum):

# On test server: restore DB, copy configs, update server_ip, run upgrade, verify
mysql -u root -p -e "CREATE DATABASE asterisk;"
zcat asterisk-db-backup.sql.gz | mysql -u root -p asterisk
scp -r root@production:/etc/asterisk/ /etc/asterisk/
sed -i "s/VARserver_ip=.*/VARserver_ip=TEST_IP/" /etc/astguiclient.conf

The clone does not need to make calls. Verify: admin loads, no schema warnings, agent login works, reports generate.

4. SVN Code Upgrade

4.1 Source Code Structure

ViciDial uses SVN (Subversion), not Git:

svn://svn.eflo.net/agc_2-X/trunk/          <-- Main development trunk
  ├── agc/                                   <-- Agent interface PHP files
  ├── bin/                                   <-- Perl backend scripts
  ├── extras/                                <-- SQL scripts, utilities, docs
  │   ├── upgrade_2.4.sql
  │   ├── upgrade_2.6.sql
  │   ├── upgrade_2.8.sql
  │   ├── upgrade_2.10.sql
  │   ├── upgrade_2.12.sql
  │   └── upgrade_2.14.sql
  ├── install.pl                             <-- Master installation script
  ├── UPGRADE                                <-- Official upgrade notes
  └── www/                                   <-- Admin/report PHP files
      ├── vicidial/                          <-- Admin reports
      └── agc/                               <-- Agent interface

4.2 Check SVN Connectivity

# Test SVN access
svn info svn://svn.eflo.net/agc_2-X/trunk

Expected output:

Path: trunk
URL: svn://svn.eflo.net/agc_2-X/trunk
Relative URL: ^/trunk
Repository Root: svn://svn.eflo.net/agc_2-X
Repository UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Revision: 3939
...

If connection fails, try: svn info svn://svn.eflo.net:3690/agc_2-X/trunk (with port), check DNS (nslookup svn.eflo.net), verify outbound port 3690 is open (nc -zv svn.eflo.net 3690), or try historical IP svn://97.76.33.146:3690/agc_2-X/trunk.

4.3 Preview Changes Before Updating

cd /usr/src/astguiclient/trunk
svn status                  # M=modified locally, ?=unversioned, !=missing
svn diff > /root/upgrade-backup-$(date +%Y%m%d)/local-changes.patch  # Save local mods

WARNING: If you customized ViciDial PHP files, svn update will try to merge upstream changes with yours. Document ALL local changes before proceeding.

4.4 Perform the SVN Update

cd /usr/src/astguiclient/trunk
svn update

Output codes: U=updated, G=merged (review!), C=conflict (must resolve), A=added, D=deleted.

For a specific revision: svn update -r 3900

4.5 Handling SVN Conflicts

If you see C (conflict) lines, resolve with:

svn resolve --accept theirs-full agc/vicidial.php   # Accept upstream (recommended)
svn resolve --accept mine-full agc/vicidial.php      # Keep your version
svn resolve --accept working agc/vicidial.php        # After manual merge

RECOMMENDATION: Almost always use --accept theirs-full. Re-apply your customizations from the patch file saved in 4.3 after the upgrade.

4.6 Starting Fresh (Nuclear Option)

mv /usr/src/astguiclient /usr/src/astguiclient-old-$(date +%Y%m%d)
mkdir -p /usr/src/astguiclient && cd /usr/src/astguiclient
svn checkout svn://svn.eflo.net/agc_2-X/trunk

Safe -- the checkout is just source code. Running scripts are in /usr/share/astguiclient/ and the web docroot. Still need install.pl to deploy.

4.7 Read the UPGRADE File

less /usr/src/astguiclient/trunk/UPGRADE

Contains critical info about config changes, new crontab entries, deprecated features, and PHP/Asterisk requirements. Do not skip.

5. Database Schema Upgrade

DANGER: This is the most critical part of any ViciDial upgrade. Running the wrong SQL scripts, running them out of order, or skipping scripts WILL break your installation. There is no undo for ALTER TABLE statements.

5.1 Identify Your Upgrade Path

mysql -u root -p -e "SELECT db_schema_version, version FROM asterisk.system_settings;"
ls -la /usr/src/astguiclient/trunk/extras/upgrade_*.sql

Scripts available: upgrade_2.0.5.sql, upgrade_2.2.sql, upgrade_2.4.sql, upgrade_2.6.sql, upgrade_2.8.sql, upgrade_2.10.sql, upgrade_2.12.sql, upgrade_2.14.sql.

5.2 SQL Upgrade Path Matrix

Find your current version in the left column, then run ALL scripts listed in the right column, in order from top to bottom:

DANGER: You CANNOT skip versions. Running upgrade_2.14.sql on a system that is at version 2.4 will NOT work correctly. You must run 2.6, 2.8, 2.10, 2.12, and THEN 2.14 in sequence.

This is the #1 mistake in the forum thread -- people skip directly to the latest upgrade script and end up with a broken database.

5.3 Running the SQL Upgrade Scripts

IMPORTANT: Run SQL upgrade scripts on the DATABASE SERVER ONLY in a cluster. Other servers (web, dialer) share the same database -- running the scripts multiple times can cause duplicate key errors.

Method 1: Command line with -f flag (recommended)

The -f flag tells MySQL to continue on errors. This is important because upgrade scripts contain statements that may fail harmlessly (e.g., adding a column that already exists):

# Run each script in sequence, one at a time
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.6.sql
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.8.sql
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.10.sql
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.12.sql
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.14.sql

Method 2: From MySQL interactive prompt

mysql -u root -p
USE asterisk;
\. /usr/src/astguiclient/trunk/extras/upgrade_2.6.sql
\. /usr/src/astguiclient/trunk/extras/upgrade_2.8.sql
\. /usr/src/astguiclient/trunk/extras/upgrade_2.10.sql
\. /usr/src/astguiclient/trunk/extras/upgrade_2.12.sql
\. /usr/src/astguiclient/trunk/extras/upgrade_2.14.sql
quit

5.4 Expected Errors (Safe to Ignore)

You WILL see errors like these when running upgrade scripts. These are normal and expected:

ERROR 1050 (42S01): Table 'vicidial_some_table' already exists
ERROR 1060 (42S21): Duplicate column name 'some_column'
ERROR 1061 (42000): Duplicate key name 'some_index'
ERROR 1062 (23000): Duplicate entry 'some_value' for key 'PRIMARY'

These occur because the upgrade scripts are designed to be re-runnable. They attempt to CREATE tables and ADD columns that may already exist from a previous partial upgrade. The -f flag ensures MySQL continues past these errors.

Errors that are NOT safe to ignore:

ERROR 1045 (28000): Access denied for user 'root'@'localhost'
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
ERROR 1044 (42000): Access denied for user 'root' to database 'asterisk'
ERROR 1114 (HY000): The table 'vicidial_log' is full
ERROR 1030 (HY000): Got error 28 from storage engine    (= disk full)

If you see any of these, STOP immediately and fix the underlying issue before continuing.

5.5 Large Table ALTER Warnings

Some upgrade scripts ALTER large tables (vicidial_log, vicidial_closer_log, vicidial_agent_log, call_log -- can be 10M-500M+ rows). These can take minutes to hours and lock the table.

# Check table sizes before running upgrade scripts
mysql -u root -p -e "SELECT table_name, ROUND(data_length/1024/1024,2) AS data_mb, table_rows FROM information_schema.TABLES WHERE table_schema='asterisk' AND table_rows > 1000000 ORDER BY data_length DESC;"

# If tables >50M rows, archive first:
/usr/share/astguiclient/ADMIN_archive_log_tables.pl --debugX

5.6 Within-Version Updates (2.14 to newer 2.14)

If already on 2.14, re-run upgrade_2.14.sql with -f flag. Most statements will produce harmless duplicate errors. This is safer than trying to edit the SQL file to skip already-applied changes.

5.7 Verify Schema Version After Upgrade

mysql -u root -p -e "SELECT db_schema_version, db_schema_update_date, version FROM asterisk.system_settings;"

The db_schema_version should now match what the new code expects. If the admin interface shows "WARNING: Code expects different schema" after the upgrade, you missed a SQL script.

6. Perl/PHP Script Installation

install.pl deploys updated scripts: copies PHP to web docroot, Perl to /usr/share/astguiclient/, AGI scripts to /var/lib/asterisk/agi-bin/, sets permissions, and updates crontab.

6.1 Running install.pl

IMPORTANT: Run install.pl on EVERY server in your cluster (database server, web servers, dialer servers). Unlike SQL scripts which run once, install.pl must run on each machine.

cd /usr/src/astguiclient/trunk
perl install.pl

Correct answers for an upgrade:

WARNING: "Restart Asterisk = Y" drops all active calls immediately.

6.2 Crontab Updates

diff <(crontab -l) /usr/src/astguiclient/trunk/extras/crontab.txt

Add only NEW entries. Do NOT replace your entire crontab with the template -- it is for fresh installs.

6.3 Configuration File Check

After install.pl, verify custom configs survived:

diff /etc/asterisk/extensions_custom.conf /root/upgrade-backup-*/configs/asterisk-etc/extensions_custom.conf
# Restore if overwritten:
# cp /root/upgrade-backup-*/configs/asterisk-etc/customexte.conf /etc/asterisk/

6.4 Rebuild Configuration Files

Go to Admin > Servers > Modify for each server, set "Rebuild conf files" = Y. Or force immediate rebuild:

/usr/share/astguiclient/AST_conf_update.pl --debugX

6.5 Area Code Population

Run ONCE per cluster (on one server only):

/usr/share/astguiclient/ADMIN_area_code_populate.pl --purge-table --debug

7. Post-Upgrade Verification

7.1 Complete Verification Checklist

Admin Interface:

[ ] Admin login page loads (no blank page, no PHP errors)
[ ] Login works -- no "WARNING: Code expects different schema" banner
[ ] System Settings shows correct VERSION, BUILD, SVN Version, DB Schema
[ ] Server list shows all servers with active processes

Agent Interface:

[ ] Agent login page loads, agent can log in and select campaign
[ ] Phone login works (SIP registers)
[ ] Agent screen displays correctly

Call Flow (test actual calls!):

[ ] Outbound: dial out, connects, audio both ways
[ ] Inbound: DID routes correctly, agent receives call
[ ] Recording starts and saves
[ ] Transfer works
[ ] Both-side hangup works

System:

[ ] ViciDial processes running: ps aux | grep -E "AST_|VD_" | grep -v grep
[ ] Asterisk running: asterisk -rx "core show version"
[ ] SIP peers OK: asterisk -rx "sip show peers" | tail -5
[ ] Cron entries present: crontab -l | grep -c astguiclient
[ ] Reports load (Agent Performance, Closer Stats, Real-time)
[ ] No schema mismatch warnings in admin

If admin page is blank, check: tail -50 /var/log/apache2/error_log (or /var/log/httpd/error_log on CentOS).

8. ViciBox Major Version Upgrades

DANGER: ViciBox major version upgrades change the OS, PHP, MariaDB, and kernel. 1-4 hours downtime. ALWAYS test on a clone first.

8.1 ViciBox Version Overview

8.2 General ViciBox Upgrade Procedure

Step 1: Stop everything and backup

/usr/share/astguiclient/stop_astguiclient   # Stop ViciDial
asterisk -rx "core stop now"                 # Stop Asterisk
systemctl stop apache2                       # Stop web server
# Complete full backup (Section 3)

Step 2: Update repository URLs

cp -a /etc/zypp/repos.d/ /root/upgrade-backup-$(date +%Y%m%d)/repos.d/
sed -i 's/15\.3/15.5/g' /etc/zypp/repos.d/*.repo
zypper --releasever=15.5 ref

Step 3: Perform distribution upgrade

zypper --releasever=15.5 dup --dry-run        # Review first!
zypper --releasever=15.5 dup --allow-vendor-change  # Actual upgrade

DANGER: zypper dup upgrades hundreds of packages (kernel, glibc, OpenSSL, PHP, MariaDB). Review the dry-run output. If critical packages are being removed, STOP.

Step 4: Reboot and verify

reboot
# After reboot:
cat /etc/os-release && uname -r && php -v && systemctl status mariadb && systemctl status apache2

Step 5: Rebuild DAHDI

After kernel upgrade, DAHDI modules will NOT work until rebuilt:

# ViciBox packaged DAHDI:
zypper install --force dahdi-linux-kmp-default

# Or compile from source:
cd /usr/src/dahdi-linux-* && make clean && make && make install
modprobe dahdi && dahdi_cfg
dahdi_test    # Should show 99%+ timing accuracy

Step 6: Rebuild Asterisk (if source-compiled)

# On ViciBox, Asterisk upgrades via zypper. For source installs:
cd /usr/src/asterisk-* && make clean && ./configure && make && make install

Step 7: Update ViciDial code

cd /usr/src/astguiclient/trunk && svn update && perl install.pl

Then follow post-upgrade verification (Section 7).

8.3 Version-Specific Notes

ViciBox 9 to 10 (Leap 15.1/15.2 to 15.3):

• PHP 7.2 to 7.4 (minor), DAHDI rebuild, MariaDB may upgrade (run mysql_upgrade -u root -p)
• ViciBox 9 is deeply EOL. Consider fresh ViciBox 12 + data migration instead of incremental upgrades.

ViciBox 10 to 11 (Leap 15.3 to 15.5):

• PHP 7.4 to 8.0 -- most impactful change. Custom scripts using mysql_* functions will break (must use mysqli_*). Run: grep -r "mysql_connect\|mysql_query" /srv/www/htdocs/vicidial/custom* 2>/dev/null
• Run mysql_upgrade after OS upgrade

ViciBox 11 to 12 (Leap 15.5 to 15.6):

• PHP 8.0 to 8.2 (smooth), Asterisk may upgrade to 20.x, DAHDI rebuild required
• Smoothest upgrade path (both PHP versions in 8.x family)

8.4 Alternative: Fresh Install with Data Migration

For systems more than 2 major versions behind, fresh install is often safer:

# ON NEW SERVER: Install ViciBox 12, run vicibox-express, then:
mysql -u root -p -e "DROP DATABASE asterisk; CREATE DATABASE asterisk;"
zcat asterisk-db-backup.sql.gz | mysql -u root -p asterisk

# Run ALL upgrade SQL scripts
for sql in upgrade_2.4.sql upgrade_2.6.sql upgrade_2.8.sql upgrade_2.10.sql upgrade_2.12.sql upgrade_2.14.sql; do
  mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/$sql
done

# Update server IP
/usr/share/astguiclient/ADMIN_update_server_ip.pl --old-server_ip=OLD_IP --new-server_ip=NEW_IP --debugX

# Install, import configs, rebuild, test, switch DNS
cd /usr/src/astguiclient/trunk && perl install.pl

Cleaner: no partially-upgraded packages, clean PHP 8.2, latest Asterisk, known-good baseline.

9. CentOS 7 Migration

CentOS 7 reached EOL June 30, 2024. No security patches since. Running ViciDial on CentOS 7 is a significant security risk.

9.1 Migration Options

9.2 Recommended: Fresh ViciBox 12 + Data Migration

Follow Section 8.4, with CentOS-specific considerations:

# On CentOS 7: export everything
mysqldump -u root -p --single-transaction --routines --triggers asterisk | gzip > /root/asterisk-migration.sql.gz
tar czf /root/config-migration.tar.gz /etc/asterisk/ /etc/astguiclient.conf /var/lib/asterisk/agi-bin/*.agi
crontab -l > /root/crontab-migration.txt

CentOS to ViciBox differences:

• Web docroot: /var/www/html/ (CentOS) vs /srv/www/htdocs/ (ViciBox)
• Service: httpd (CentOS) vs apache2 (ViciBox)
• Asterisk: CentOS 7 often has Asterisk 11/13 vs ViciBox 12 Asterisk 18-20 (SIP config may differ, chan_sip vs chan_pjsip)
• PHP: CentOS 7 has PHP 5.4/5.6 vs ViciBox 12 PHP 8.2 -- ALL custom code must be reviewed

9.3 IP Address Change Procedure

# Updates server IP across all ViciDial database tables
/usr/share/astguiclient/ADMIN_update_server_ip.pl \
  --old-server_ip=OLD_IP --new-server_ip=NEW_IP --debugX

# Also update astguiclient.conf
sed -i "s/VARserver_ip=OLD_IP/VARserver_ip=NEW_IP/" /etc/astguiclient.conf

If keeping the same IP (reassigned to new server), skip ADMIN_update_server_ip.pl.

9.4 In-Place ELevate (CentOS 7 to AlmaLinux 8)

WARNING: In-place OS migration on production ViciDial is risky. Fresh install (Section 9.2) is strongly recommended.

yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el7.noarch.rpm
yum install -y leapp-upgrade leapp-data-almalinux
leapp preupgrade                    # Assessment -- review /var/log/leapp/leapp-report.txt
leapp upgrade && reboot             # Actual upgrade + reboot

After: verify OS, rebuild DAHDI, rebuild Asterisk if source-compiled, run mysql_upgrade, test everything.

9.5 AlmaLinux 9 Notes

Working reference from ViciDial forum: "Alma Linux 9.5 | SVN 3939 | DB Schema 1729 | Asterisk 18.26.0 | PHP8". Key differences: SELinux enforcing, firewalld (not iptables), PHP 8.x, Asterisk/DAHDI compiled from source.

10. Multi-Server Cluster Upgrades

All servers in a cluster MUST run the same ViciDial code version and share the same database schema.

10.1 Upgrade Order

1. FIRST:  Database server       (SQL upgrade scripts)
2. SECOND: Web server(s)         (SVN update + install.pl)
3. THIRD:  Dialer server(s)      (SVN update + install.pl)

SQL schema must be updated BEFORE new code runs. New code queries columns that must already exist.

10.2 Rolling Upgrade (Minimal Downtime)

For multi-dialer clusters:

1. Prep: SVN update on all servers (downloads only, no deploy)
2. DB: Run SQL scripts on DB server (seconds to minutes)
3. Web: Run install.pl + restart Apache (seconds of agent interface downtime, active calls unaffected)
4. Dialers (one at a time): Move agents off dialer, wait for calls to finish, stop Asterisk, run install.pl, start Asterisk, verify, move agents back

10.3 Single-Server Upgrade

cd /usr/src/astguiclient/trunk && svn update
mysql -u root -p -f --database=asterisk < extras/upgrade_2.14.sql  # + any earlier scripts needed
perl install.pl
/usr/share/astguiclient/ADMIN_area_code_populate.pl --purge-table --debug
/usr/share/astguiclient/AST_conf_update.pl --debugX
asterisk -rx "core restart now"
# Verify (Section 7)

10.4 Schema During Rolling Upgrades

SQL scripts ADD columns/tables (never remove). Old code ignores new columns; new code expects them. Running new code BEFORE SQL upgrade = crashes. Always upgrade DB first.

11. Rollback Procedures

11.1 Decision Matrix: Fix Forward vs Roll Back

RULE OF THUMB: If calls are down and you cannot fix the issue within 15-30 minutes, roll back. A working old version is better than a broken new version.

11.2 SVN Code Rollback

Works if you did NOT run SQL upgrade scripts:

cd /usr/src/astguiclient/trunk
svn update -r PREVIOUS_REVISION    # e.g., svn update -r 3703
perl install.pl
asterisk -rx "core restart now"
systemctl restart apache2           # or httpd on CentOS

11.3 Database Rollback

DANGER: There is no "undo" for ALTER TABLE. The only option is restore from backup. You lose ALL data created after the backup.

pkill -f AST_manager; pkill -f VD_hopper; pkill -f AST_VDauto
asterisk -rx "core stop now"
systemctl stop apache2

mysql -u root -p -e "DROP DATABASE asterisk; CREATE DATABASE asterisk;"
zcat /root/upgrade-backup-YYYYMMDD/asterisk-db-*.sql.gz | mysql -u root -p asterisk

cd /usr/src/astguiclient/trunk && svn update -r PREVIOUS_REVISION && perl install.pl
cp -a /root/upgrade-backup-YYYYMMDD/configs/asterisk-etc/* /etc/asterisk/
cp /root/upgrade-backup-YYYYMMDD/configs/astguiclient.conf /etc/astguiclient.conf
crontab /root/upgrade-backup-YYYYMMDD/configs/crontab-root.txt

systemctl start mariadb && systemctl start apache2 && systemctl start asterisk

11.4 Configuration File Rollback

If install.pl overwrote your custom configurations:

# Restore specific config files
cp /root/upgrade-backup-YYYYMMDD/configs/asterisk-etc/extensions_custom.conf /etc/asterisk/
cp /root/upgrade-backup-YYYYMMDD/configs/asterisk-etc/customexte.conf /etc/asterisk/
cp /root/upgrade-backup-YYYYMMDD/configs/asterisk-etc/sip.conf /etc/asterisk/

# Reload Asterisk config without restart (preserves active calls)
asterisk -rx "dialplan reload"
asterisk -rx "sip reload"

11.5 Full System Rollback (ViciBox OS Upgrade)

If a ViciBox OS upgrade went wrong and you cannot fix it:

# Option 1: If you have a VM snapshot, restore it
# (This is why you should snapshot before ViciBox upgrades)

# Option 2: If you have a full system backup
# Reinstall the OS from the old ViciBox ISO
# Restore database and configuration from backups

# Option 3: If you have a clone/replica
# Switch traffic to the replica
# Fix the primary server at leisure

PRO TIP: Before a ViciBox major version upgrade, take a VM snapshot (if running on a hypervisor) or make a full disk image. This gives you a guaranteed rollback path that takes minutes instead of hours.

12. Common Upgrade Failures and Fixes

This section documents the most common upgrade failures reported in the ViciDial forum, along with proven fixes.

12.1 "WARNING: Code expects different schema"

Symptom: Red banner in admin interface: WARNING: Code expects DB schema version XXXX but found version YYYY

Cause: SVN code was updated but database SQL upgrade scripts were not run (or not all of them were run).

Fix:

# Check current schema version
mysql -u root -p -e "SELECT db_schema_version FROM asterisk.system_settings;"

# Run the missing SQL upgrade scripts (Section 5)
# If your schema is at 1359 but code expects 1729:
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.8.sql
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.10.sql
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.12.sql
mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/upgrade_2.14.sql

# Verify
mysql -u root -p -e "SELECT db_schema_version FROM asterisk.system_settings;"

12.2 Admin Page Blank After Upgrade

Symptom: Admin login page (/vicidial/admin.php) returns a blank white page or a 500 error.

Causes and fixes:

# 1. Check Apache error log for the actual error
tail -50 /var/log/apache2/error_log    # openSUSE
tail -50 /var/log/httpd/error_log      # CentOS

# 2. Common: PHP version incompatibility
php -v
# If PHP 8 but code is old: update ViciDial code via SVN
# If PHP 5 but code is new: upgrade PHP or use compatible code version

# 3. Common: PHP extension missing
php -m | grep -i mysql
# If empty: install php-mysql / php-mysqli extension
# openSUSE: zypper install php8-mysql
# CentOS: yum install php-mysqlnd

# 4. Common: File permissions wrong
# openSUSE:
chown -R wwwrun:www /srv/www/htdocs/agc/
chown -R wwwrun:www /srv/www/htdocs/vicidial/
# CentOS:
chown -R apache:apache /var/www/html/agc/
chown -R apache:apache /var/www/html/vicidial/

# 5. Common: Database connection failed
# Check astguiclient.conf has correct DB credentials
grep "VARDB" /etc/astguiclient.conf

# 6. Common: SELinux blocking access (RHEL/AlmaLinux)
getenforce
# If "Enforcing", try:
setenforce 0    # Temporarily disable
# Then test. If it works, configure SELinux properly rather than leaving it off.

12.3 System Settings Blank/Missing After Upgrade

Symptom: Admin > System Settings shows empty fields or no data.

Cause: Database schema was not upgraded properly. The system_settings table structure does not match what the code expects.

Fix:

# Run ALL relevant upgrade scripts in sequence
for sql in upgrade_2.4.sql upgrade_2.6.sql upgrade_2.8.sql upgrade_2.10.sql upgrade_2.12.sql upgrade_2.14.sql; do
  echo "Running $sql..."
  mysql -u root -p -f --database=asterisk < /usr/src/astguiclient/trunk/extras/$sql
done

# Verify system_settings has data
mysql -u root -p -e "SELECT * FROM asterisk.system_settings \G" | head -30

12.4 Admin Access Lost / "You do not have permission"

Symptom: After upgrade, admin users cannot log in or see "You do not have permission to view this page."

Cause: User table schema changed, or user group permissions were affected.

Fix:

# Check admin user still exists and has correct group
mysql -u root -p -e "
SELECT user, full_name, user_group, active
FROM asterisk.vicidial_users
WHERE user_group = 'ADMIN'
LIMIT 10;
"

# Verify the ADMIN user group has full permissions
mysql -u root -p -e "
SELECT user_group, allowed_campaigns, admin_viewable_groups
FROM asterisk.vicidial_user_groups
WHERE user_group = 'ADMIN';
"

# If the user exists but permissions are wrong, reset:
mysql -u root -p -e "
UPDATE asterisk.vicidial_users
SET user_group = 'ADMIN', active = 'Y'
WHERE user = 'YOUR_ADMIN_USERNAME';
"

# If ADMIN group is missing or broken, recreate:
mysql -u root -p -e "
INSERT IGNORE INTO asterisk.vicidial_user_groups
SET user_group = 'ADMIN',
    allowed_campaigns = '-ALL CAMPAIGNS-',
    admin_viewable_groups = '---ALL---';
"

12.5 PHP Errors After Upgrade

Symptom: PHP warnings/errors on agent screens or admin pages:

Warning: mysql_fetch_row() expects parameter 1 to be resource, boolean given
Fatal error: Uncaught Error: Call to undefined function mysql_connect()

Cause: PHP version changed (typically from PHP 5.x/7.x to PHP 8.x) and old code uses removed functions.

Fix:

# If the error is in ViciDial core files:
# Update SVN to latest (which has PHP 8 support)
cd /usr/src/astguiclient/trunk
svn update
perl install.pl

# If the error is in CUSTOM files:
# You must manually update the code
# Replace mysql_* functions with mysqli_* equivalents:
# mysql_connect()    -->  mysqli_connect()
# mysql_query()      -->  mysqli_query($link, ...)
# mysql_fetch_row()  -->  mysqli_fetch_row()
# mysql_num_rows()   -->  mysqli_num_rows()
# mysql_real_escape_string()  -->  mysqli_real_escape_string($link, ...)

12.6 DAHDI Errors After Kernel Update

Symptom: After ViciBox OS upgrade or kernel update:

dahdi: module not found
Unable to open /dev/dahdi/timer: No such file or directory
MeetMe conferences not working

Cause: Kernel modules are version-specific. A kernel upgrade invalidates compiled DAHDI modules.

Fix:

# Rebuild DAHDI kernel modules
# Method 1: ViciBox package (preferred)
zypper install --force dahdi-linux-kmp-default

# Method 2: Compile from source
cd /usr/src/dahdi-linux-complete-*
# or
cd /usr/src/dahdi-linux-*

make clean
make
make install

modprobe dahdi
dahdi_cfg -v

# Verify timing
dahdi_test
# Should show: Best: 99.xxx% -- Loss: 0.xxx%

# If dahdi_test is not available:
cat /proc/dahdi/1

12.7 Apache Won't Start After Upgrade

Symptom: Apache (httpd/apache2) fails to start after upgrade.

# Check the actual error
systemctl status apache2
journalctl -u apache2 --no-pager | tail -20

# Common causes:

# 1. PHP module version mismatch
# After PHP upgrade, Apache's PHP module needs updating
# openSUSE:
zypper install apache2-mod_php8
a2enmod php8
# CentOS:
yum install php-fpm
systemctl restart php-fpm

# 2. Config syntax error from upgrade
apachectl configtest
# Fix any syntax errors reported

# 3. Port conflict
ss -tlnp | grep :80
# Kill any process using port 80, then start Apache

# 4. SSL certificate issue
# If SSL config references a cert that moved/expired
# Temporarily comment out SSL config to get HTTP working

12.8 Calls Not Connecting After Upgrade

Symptom: Agents can log in, but outbound calls fail or inbound calls do not route.

Fix:

# 1. Check Asterisk is running
asterisk -rx "core show version"

# 2. Check SIP trunks are registered
asterisk -rx "sip show registry"
asterisk -rx "sip show peers" | grep -i unreachable

# 3. Check dialplan loaded
asterisk -rx "dialplan show" | wc -l
# Should be thousands of lines, not zero

# 4. Rebuild ViciDial config files
# Admin > Servers > each server > Rebuild conf files = Y
# Or force immediate rebuild:
/usr/share/astguiclient/AST_conf_update.pl --debugX

# 5. Reload Asterisk config
asterisk -rx "sip reload"
asterisk -rx "dialplan reload"
asterisk -rx "iax2 reload"

# 6. Check for SIP errors in Asterisk CLI
asterisk -rvvv
# Try a test call and watch for errors

# 7. Check ViciDial processes are running
ps aux | grep -E "AST_manager_send|AST_manager_listen" | grep -v grep
# Both must be running for calls to work

12.9 SVN Connection Timeout

Symptom:

svn: E000110: Unable to connect to a repository at URL 'svn://svn.eflo.net:3690/agc_2-X/trunk'
svn: E000110: Can't connect to host 'svn.eflo.net': Connection timed out

Fixes:

# 1. Try without port number (newer SVN setup)
svn info svn://svn.eflo.net/agc_2-X/trunk

# 2. Check firewall allows outbound port 3690
iptables -L OUTPUT -n | grep 3690
# Or test:
nc -zv svn.eflo.net 3690

# 3. Check DNS
nslookup svn.eflo.net

# 4. The SVN server may be temporarily down
# Check the ViciDial forum for announcements

# 5. Workaround: Download code manually
# Someone with access can tar the trunk and transfer it to you
# Then extract over your existing /usr/src/astguiclient/trunk/

12.10 Recordings Not Saving After Upgrade

Symptom: Calls work but recordings are not being created.

# 1. Check recording directory permissions
ls -la /var/spool/asterisk/monitor/
# Must be writable by asterisk user

# 2. Check if MixMon/Monitor app is loaded in Asterisk
asterisk -rx "module show like monitor"
asterisk -rx "module show like mixmon"

# 3. Check disk space
df -h /var/spool/asterisk/

# 4. Check ViciDial recording settings
mysql -u root -p -e "
SELECT campaign_id, campaign_rec, campaign_rec_filename
FROM asterisk.vicidial_campaigns;
"
# campaign_rec should be 'ALLCALLS' or 'ALLFORCE' for recording

# 5. Check cron processing
# The recording move script should be running:
ps aux | grep -i "move\|recording" | grep -v grep

13. Automation Script

This script automates the pre-upgrade backup, version documentation, and post-upgrade verification. It does NOT perform the actual upgrade -- that should always be done manually with human oversight.

13.1 Pre-Upgrade Backup and Validation Script

Save this as /usr/local/bin/vicidial-upgrade-prep.sh:

#!/bin/bash
#
# ViciDial Pre-Upgrade Backup and Validation Script
# Usage: vicidial-upgrade-prep.sh [--skip-db-test]
#
# This script:
# 1. Documents current system state
# 2. Creates full database backup
# 3. Backs up configuration files
# 4. Tests backup integrity (unless --skip-db-test)
# 5. Checks disk space
# 6. Tests SVN connectivity
# 7. Generates upgrade readiness report
#
# It does NOT perform the actual upgrade.
#

set -euo pipefail

# Configuration
BACKUP_BASE="/root/upgrade-backup-$(date +%Y%m%d-%H%M)"
LOG_FILE="$BACKUP_BASE/upgrade-prep.log"
SKIP_DB_TEST=false
MYSQL_OPTS="-u root"  # Add -p if password required, or use ~/.my.cnf

# Colors for terminal output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Parse arguments
for arg in "$@"; do
  case $arg in
    --skip-db-test) SKIP_DB_TEST=true ;;
    *) echo "Unknown argument: $arg"; exit 1 ;;
  esac
done

# Functions
log() {
  echo -e "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

pass() {
  echo -e "${GREEN}[PASS]${NC} $1" | tee -a "$LOG_FILE"
}

fail() {
  echo -e "${RED}[FAIL]${NC} $1" | tee -a "$LOG_FILE"
}

warn() {
  echo -e "${YELLOW}[WARN]${NC} $1" | tee -a "$LOG_FILE"
}

# Create backup directory
mkdir -p "$BACKUP_BASE"
touch "$LOG_FILE"

echo "============================================="
echo " ViciDial Pre-Upgrade Backup and Validation"
echo " $(date)"
echo " Backup directory: $BACKUP_BASE"
echo "============================================="
log "Starting pre-upgrade preparation"

# =============================================
# STEP 1: Document Current System State
# =============================================
log ""
log "=== STEP 1: Documenting current system state ==="

STATE_FILE="$BACKUP_BASE/system-state.txt"

{
  echo "=== System State: $(date) ==="
  echo ""

  echo "--- Hostname ---"
  hostname -f 2>/dev/null || hostname

  echo ""
  echo "--- OS ---"
  cat /etc/os-release 2>/dev/null || cat /etc/centos-release 2>/dev/null || echo "Unknown OS"

  echo ""
  echo "--- Kernel ---"
  uname -r

  echo ""
  echo "--- Uptime ---"
  uptime

  echo ""
  echo "--- IP Addresses ---"
  ip -4 addr show | grep inet | grep -v "127.0.0.1"

  echo ""
  echo "--- Asterisk Version ---"
  asterisk -V 2>/dev/null || echo "Asterisk not found"

  echo ""
  echo "--- PHP Version ---"
  php -v 2>/dev/null | head -1 || echo "PHP not found"

  echo ""
  echo "--- MariaDB/MySQL Version ---"
  mysql -V 2>/dev/null || echo "MySQL not found"

  echo ""
  echo "--- SVN Info ---"
  cd /usr/src/astguiclient/trunk 2>/dev/null && svn info 2>/dev/null || echo "SVN checkout not found"

  echo ""
  echo "--- ViciDial Version (from DB) ---"
  mysql $MYSQL_OPTS -e "SELECT version, db_schema_version, db_schema_update_date FROM asterisk.system_settings \G" 2>/dev/null || echo "Cannot query system_settings"

  echo ""
  echo "--- ViciDial RPMs ---"
  rpm -qa 2>/dev/null | grep -i -E "vici|asterisk|dahdi" | sort || echo "No RPMs found"

  echo ""
  echo "--- Active ViciDial Processes ---"
  ps aux | grep -E "AST_|ADMIN_|VD_" | grep -v grep || echo "No ViciDial processes found"

  echo ""
  echo "--- Disk Space ---"
  df -h /

  echo ""
  echo "--- Database Size ---"
  mysql $MYSQL_OPTS -e "
    SELECT table_schema,
      ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS Size_MB
    FROM information_schema.TABLES
    GROUP BY table_schema
    ORDER BY Size_MB DESC;" 2>/dev/null || echo "Cannot query database sizes"

  echo ""
  echo "--- Servers Table ---"
  mysql $MYSQL_OPTS -e "SELECT server_id, server_ip, active, asterisk_version FROM asterisk.servers;" 2>/dev/null || echo "Cannot query servers"

} > "$STATE_FILE" 2>&1

pass "System state documented: $STATE_FILE"

# =============================================
# STEP 2: Full Database Backup
# =============================================
log ""
log "=== STEP 2: Full database backup ==="

DB_BACKUP="$BACKUP_BASE/asterisk-db-$(date +%Y%m%d-%H%M).sql.gz"
FULL_BACKUP="$BACKUP_BASE/all-databases-$(date +%Y%m%d-%H%M).sql.gz"

log "Backing up asterisk database..."
if mysqldump $MYSQL_OPTS --single-transaction --routines --triggers asterisk 2>/dev/null | gzip > "$DB_BACKUP"; then
  DB_SIZE=$(ls -lh "$DB_BACKUP" | awk '{print $5}')
  pass "Asterisk database backup: $DB_BACKUP ($DB_SIZE)"
else
  fail "Asterisk database backup FAILED!"
fi

log "Backing up all databases..."
if mysqldump $MYSQL_OPTS --single-transaction --routines --triggers --events --all-databases 2>/dev/null | gzip > "$FULL_BACKUP"; then
  FULL_SIZE=$(ls -lh "$FULL_BACKUP" | awk '{print $5}')
  pass "Full database backup: $FULL_BACKUP ($FULL_SIZE)"
else
  fail "Full database backup FAILED!"
fi

# Verify gzip integrity
log "Verifying backup integrity..."
if gunzip -t "$DB_BACKUP" 2>/dev/null; then
  pass "Asterisk backup gzip integrity OK"
else
  fail "Asterisk backup gzip CORRUPT!"
fi

if gunzip -t "$FULL_BACKUP" 2>/dev/null; then
  pass "Full backup gzip integrity OK"
else
  fail "Full backup gzip CORRUPT!"
fi

# =============================================
# STEP 3: Test Backup Restore (optional)
# =============================================
if [ "$SKIP_DB_TEST" = false ]; then
  log ""
  log "=== STEP 3: Testing backup restore ==="
  TEST_DB="vicidial_upgrade_test_$(date +%s)"

  log "Creating test database: $TEST_DB"
  mysql $MYSQL_OPTS -e "CREATE DATABASE \`$TEST_DB\`;" 2>/dev/null

  log "Restoring asterisk backup to test database..."
  if zcat "$DB_BACKUP" | mysql $MYSQL_OPTS "$TEST_DB" 2>/dev/null; then
    # Verify data
    LEAD_COUNT=$(mysql $MYSQL_OPTS -N -e "SELECT COUNT(*) FROM \`$TEST_DB\`.vicidial_list;" 2>/dev/null || echo "0")
    RECENT_CALLS=$(mysql $MYSQL_OPTS -N -e "SELECT COUNT(*) FROM \`$TEST_DB\`.vicidial_closer_log WHERE call_date >= CURDATE() - INTERVAL 1 DAY;" 2>/dev/null || echo "0")

    pass "Backup restore test successful (Leads: $LEAD_COUNT, Recent calls: $RECENT_CALLS)"

    if [ "$RECENT_CALLS" = "0" ]; then
      warn "No calls from last 24 hours found in backup -- verify this is expected"
    fi
  else
    fail "Backup restore test FAILED!"
  fi

  # Cleanup
  mysql $MYSQL_OPTS -e "DROP DATABASE IF EXISTS \`$TEST_DB\`;" 2>/dev/null
  log "Test database cleaned up"
else
  log ""
  log "=== STEP 3: Backup restore test SKIPPED (--skip-db-test) ==="
  warn "Backup restore test skipped -- consider running without --skip-db-test"
fi

# =============================================
# STEP 4: Configuration File Backup
# =============================================
log ""
log "=== STEP 4: Configuration file backup ==="

CONFIG_DIR="$BACKUP_BASE/configs"
mkdir -p "$CONFIG_DIR"

# Asterisk configs
if [ -d /etc/asterisk ]; then
  cp -a /etc/asterisk/ "$CONFIG_DIR/asterisk-etc/"
  pass "Asterisk config backed up"
else
  warn "/etc/asterisk/ not found"
fi

# astguiclient.conf
if [ -f /etc/astguiclient.conf ]; then
  cp /etc/astguiclient.conf "$CONFIG_DIR/"
  pass "astguiclient.conf backed up"
else
  warn "/etc/astguiclient.conf not found"
fi

# AGI scripts
if [ -d /var/lib/asterisk/agi-bin ]; then
  cp -a /var/lib/asterisk/agi-bin/ "$CONFIG_DIR/agi-bin/"
  pass "AGI scripts backed up"
fi

# Crontab
crontab -l > "$CONFIG_DIR/crontab-root.txt" 2>/dev/null
pass "Root crontab backed up"

# Apache config
for dir in /etc/apache2 /etc/httpd; do
  if [ -d "$dir" ]; then
    cp -a "$dir" "$CONFIG_DIR/$(basename $dir)/"
    pass "$(basename $dir) config backed up"
  fi
done

# SVN local modifications
if [ -d /usr/src/astguiclient/trunk ]; then
  cd /usr/src/astguiclient/trunk
  svn diff > "$CONFIG_DIR/svn-local-changes.patch" 2>/dev/null
  svn status > "$CONFIG_DIR/svn-status.txt" 2>/dev/null
  MODIFIED=$(grep "^M" "$CONFIG_DIR/svn-status.txt" 2>/dev/null | wc -l)
  if [ "$MODIFIED" -gt 0 ]; then
    warn "$MODIFIED locally modified files in SVN checkout -- review svn-local-changes.patch"
  else
    pass "No local SVN modifications"
  fi
fi

# =============================================
# STEP 5: Disk Space Check
# =============================================
log ""
log "=== STEP 5: Disk space check ==="

# Get DB size in MB
DB_SIZE_MB=$(mysql $MYSQL_OPTS -N -e "
  SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024, 0)
  FROM information_schema.TABLES
  WHERE table_schema = 'asterisk';" 2>/dev/null || echo "0")

# Get free space in MB
FREE_MB=$(df -m / | tail -1 | awk '{print $4}')

NEEDED_MB=$((DB_SIZE_MB * 3))

log "Database size: ${DB_SIZE_MB} MB"
log "Free disk space: ${FREE_MB} MB"
log "Recommended free space: ${NEEDED_MB} MB (3x database size)"

if [ "$FREE_MB" -gt "$NEEDED_MB" ]; then
  pass "Disk space sufficient ($FREE_MB MB free, need $NEEDED_MB MB)"
else
  fail "INSUFFICIENT DISK SPACE! ($FREE_MB MB free, need $NEEDED_MB MB)"
fi

# =============================================
# STEP 6: SVN Connectivity Test
# =============================================
log ""
log "=== STEP 6: SVN connectivity test ==="

if svn info svn://svn.eflo.net/agc_2-X/trunk > /dev/null 2>&1; then
  REMOTE_REV=$(svn info svn://svn.eflo.net/agc_2-X/trunk 2>/dev/null | grep "^Revision:" | awk '{print $2}')
  LOCAL_REV=$(cd /usr/src/astguiclient/trunk 2>/dev/null && svn info 2>/dev/null | grep "^Revision:" | awk '{print $2}' || echo "unknown")
  pass "SVN repository accessible (Remote: r$REMOTE_REV, Local: r$LOCAL_REV)"

  if [ "$REMOTE_REV" != "$LOCAL_REV" ] && [ "$LOCAL_REV" != "unknown" ]; then
    log "  SVN update available: r$LOCAL_REV -> r$REMOTE_REV"
  fi
elif svn info svn://svn.eflo.net:3690/agc_2-X/trunk > /dev/null 2>&1; then
  pass "SVN repository accessible (via port 3690)"
else
  fail "Cannot connect to SVN repository -- check firewall (port 3690) and DNS"
fi

# =============================================
# STEP 7: Generate Upgrade Readiness Report
# =============================================
log ""
log "=== STEP 7: Upgrade Readiness Report ==="

REPORT="$BACKUP_BASE/upgrade-readiness-report.txt"
{
  echo "============================================="
  echo " ViciDial Upgrade Readiness Report"
  echo " Generated: $(date)"
  echo " Server: $(hostname)"
  echo "============================================="
  echo ""

  # Extract key versions
  echo "CURRENT STATE:"
  echo "  OS: $(cat /etc/os-release 2>/dev/null | grep PRETTY_NAME | cut -d'"' -f2)"
  echo "  Asterisk: $(asterisk -V 2>/dev/null || echo 'not found')"
  echo "  PHP: $(php -r 'echo PHP_VERSION;' 2>/dev/null || echo 'not found')"
  echo "  MariaDB: $(mysql -V 2>/dev/null | awk '{print $5}' | tr -d ',')"
  echo "  ViciDial SVN: r$(cd /usr/src/astguiclient/trunk 2>/dev/null && svn info 2>/dev/null | grep '^Revision:' | awk '{print $2}' || echo 'unknown')"
  echo "  DB Schema: $(mysql $MYSQL_OPTS -N -e 'SELECT db_schema_version FROM asterisk.system_settings;' 2>/dev/null || echo 'unknown')"
  echo ""

  echo "BACKUP STATUS:"
  echo "  Backup directory: $BACKUP_BASE"
  echo "  Database backup: $(ls -lh $DB_BACKUP 2>/dev/null | awk '{print $5" "$9}' || echo 'MISSING')"
  echo "  Full backup: $(ls -lh $FULL_BACKUP 2>/dev/null | awk '{print $5" "$9}' || echo 'MISSING')"
  echo "  Config backup: $(du -sh $CONFIG_DIR 2>/dev/null | awk '{print $1}')"
  echo ""

  echo "DISK SPACE:"
  echo "  Database size: ${DB_SIZE_MB} MB"
  echo "  Free space: ${FREE_MB} MB"
  echo "  Required: ${NEEDED_MB} MB"
  echo ""

  echo "NEXT STEPS:"
  echo "  1. Review this report and system-state.txt"
  echo "  2. Schedule maintenance window"
  echo "  3. Notify agents and managers"
  echo "  4. Perform SVN update: cd /usr/src/astguiclient/trunk && svn update"
  echo "  5. Run SQL upgrade scripts (see Section 5 of the playbook)"
  echo "  6. Run install.pl: perl install.pl"
  echo "  7. Post-upgrade verification (see Section 7 of the playbook)"
  echo ""

  echo "ROLLBACK:"
  echo "  SVN revert: svn update -r LOCAL_REVISION"
  echo "  DB restore: zcat $DB_BACKUP | mysql -u root -p asterisk"
  echo "  Config restore: cp -a $CONFIG_DIR/asterisk-etc/* /etc/asterisk/"

} > "$REPORT"

log ""
log "============================================="
log " PRE-UPGRADE PREPARATION COMPLETE"
log " Backup directory: $BACKUP_BASE"
log " Report: $REPORT"
log " Log: $LOG_FILE"
log "============================================="

# Summary of total backup size
TOTAL_SIZE=$(du -sh "$BACKUP_BASE" | awk '{print $1}')
log "Total backup size: $TOTAL_SIZE"
log ""
log "Review the readiness report, then proceed with the upgrade when ready."

13.2 Post-Upgrade Verification Script

Save this as /usr/local/bin/vicidial-upgrade-verify.sh:

#!/bin/bash
#
# ViciDial Post-Upgrade Verification Script
# Usage: vicidial-upgrade-verify.sh
#
# Checks all critical systems after a ViciDial upgrade.
#

set -uo pipefail

MYSQL_OPTS="-u root"
PASS=0
FAIL=0
WARN=0

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

pass() { echo -e "${GREEN}[PASS]${NC} $1"; ((PASS++)); }
fail() { echo -e "${RED}[FAIL]${NC} $1"; ((FAIL++)); }
warn() { echo -e "${YELLOW}[WARN]${NC} $1"; ((WARN++)); }

echo "============================================="
echo " ViciDial Post-Upgrade Verification"
echo " $(date)"
echo "============================================="
echo ""

# 1. Asterisk running
echo "--- Asterisk ---"
if asterisk -rx "core show version" > /dev/null 2>&1; then
  pass "Asterisk is running: $(asterisk -rx 'core show version' 2>/dev/null)"
else
  fail "Asterisk is NOT running!"
fi

# 2. MariaDB running
echo ""
echo "--- Database ---"
if mysql $MYSQL_OPTS -e "SELECT 1;" > /dev/null 2>&1; then
  pass "MariaDB is running"
else
  fail "MariaDB is NOT running or not accessible!"
fi

# 3. Schema version check
SCHEMA=$(mysql $MYSQL_OPTS -N -e "SELECT db_schema_version FROM asterisk.system_settings;" 2>/dev/null || echo "0")
if [ "$SCHEMA" -gt 0 ]; then
  pass "DB Schema version: $SCHEMA"
else
  fail "Cannot read DB schema version!"
fi

# 4. ViciDial version
VERSION=$(mysql $MYSQL_OPTS -N -e "SELECT version FROM asterisk.system_settings;" 2>/dev/null || echo "unknown")
pass "ViciDial version: $VERSION"

# 5. Apache running
echo ""
echo "--- Web Server ---"
if systemctl is-active apache2 > /dev/null 2>&1; then
  pass "Apache (apache2) is running"
elif systemctl is-active httpd > /dev/null 2>&1; then
  pass "Apache (httpd) is running"
else
  fail "Apache is NOT running!"
fi

# 6. Admin interface accessible
ADMIN_URL="http://localhost/vicidial/admin.php"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$ADMIN_URL" 2>/dev/null || echo "000")
if [ "$HTTP_CODE" = "200" ]; then
  pass "Admin interface returns HTTP 200"
elif [ "$HTTP_CODE" = "302" ]; then
  pass "Admin interface returns HTTP 302 (redirect to login)"
else
  fail "Admin interface returns HTTP $HTTP_CODE (expected 200 or 302)"
fi

# 7. Agent interface accessible
AGENT_URL="http://localhost/agc/vicidial.php"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$AGENT_URL" 2>/dev/null || echo "000")
if [ "$HTTP_CODE" = "200" ]; then
  pass "Agent interface returns HTTP 200"
else
  fail "Agent interface returns HTTP $HTTP_CODE (expected 200)"
fi

# 8. ViciDial processes running
echo ""
echo "--- ViciDial Processes ---"
for proc in AST_manager_listen AST_manager_send AST_update; do
  if pgrep -f "$proc" > /dev/null 2>&1; then
    pass "$proc is running"
  else
    fail "$proc is NOT running!"
  fi
done

# 9. SIP peers
echo ""
echo "--- SIP Status ---"
PEER_COUNT=$(asterisk -rx "sip show peers" 2>/dev/null | grep -c "OK" || echo "0")
if [ "$PEER_COUNT" -gt 0 ]; then
  pass "$PEER_COUNT SIP peers in OK state"
else
  warn "No SIP peers in OK state (may be normal if using PJSIP)"
fi

UNREACHABLE=$(asterisk -rx "sip show peers" 2>/dev/null | grep -c "UNREACHABLE" || echo "0")
if [ "$UNREACHABLE" -gt 0 ]; then
  warn "$UNREACHABLE SIP peers UNREACHABLE"
fi

# 10. Recording directory writable
echo ""
echo "--- Recordings ---"
MONITOR_DIR="/var/spool/asterisk/monitor"
if [ -w "$MONITOR_DIR" ]; then
  pass "Recording directory is writable"
else
  fail "Recording directory $MONITOR_DIR is NOT writable!"
fi

# 11. Disk space
echo ""
echo "--- System Resources ---"
FREE_PCT=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
if [ "$FREE_PCT" -lt 80 ]; then
  pass "Disk usage: ${FREE_PCT}%"
elif [ "$FREE_PCT" -lt 90 ]; then
  warn "Disk usage: ${FREE_PCT}% (getting full)"
else
  fail "Disk usage: ${FREE_PCT}% (critically low!)"
fi

# 12. Cron processes
echo ""
echo "--- Cron ---"
CRON_LINES=$(crontab -l 2>/dev/null | grep -c "astguiclient\|AST_\|VD_" || echo "0")
if [ "$CRON_LINES" -gt 0 ]; then
  pass "$CRON_LINES ViciDial cron entries found"
else
  fail "No ViciDial cron entries found!"
fi

# 13. DAHDI (if applicable)
echo ""
echo "--- DAHDI ---"
if lsmod | grep -q dahdi; then
  pass "DAHDI kernel module loaded"
else
  if [ -f /etc/dahdi/system.conf ]; then
    fail "DAHDI configured but kernel module NOT loaded!"
  else
    pass "DAHDI not configured (normal for non-hardware systems using res_timing_timerfd)"
  fi
fi

# Summary
echo ""
echo "============================================="
echo " RESULTS: ${GREEN}$PASS passed${NC}, ${RED}$FAIL failed${NC}, ${YELLOW}$WARN warnings${NC}"
echo "============================================="

if [ "$FAIL" -gt 0 ]; then
  echo ""
  echo -e "${RED}UPGRADE VERIFICATION FAILED -- $FAIL issues need attention${NC}"
  exit 1
else
  echo ""
  echo -e "${GREEN}UPGRADE VERIFICATION PASSED${NC}"
  exit 0
fi

13.3 Making the Scripts Executable

chmod +x /usr/local/bin/vicidial-upgrade-prep.sh
chmod +x /usr/local/bin/vicidial-upgrade-verify.sh

13.4 Using the Scripts

# Before upgrade:
vicidial-upgrade-prep.sh
# Review the report in /root/upgrade-backup-YYYYMMDD-HHMM/

# Perform the upgrade manually (Sections 4, 5, 6)

# After upgrade:
vicidial-upgrade-verify.sh
# Fix any FAIL items before declaring success

14. Quick Reference Card

Print this section and keep it next to you during upgrades.

SVN Code Update (Routine)

# 1. Backup
vicidial-upgrade-prep.sh               # Or manual backup (Section 3)

# 2. Update code
cd /usr/src/astguiclient/trunk
svn update

# 3. Database (if needed -- check UPGRADE file)
mysql -u root -p -f --database=asterisk < extras/upgrade_2.14.sql

# 4. Install
perl install.pl

# 5. Area codes (one server only)
/usr/share/astguiclient/ADMIN_area_code_populate.pl --purge-table --debug

# 6. Rebuild configs
# Admin > Servers > Modify > Rebuild conf files = Y
# Or: /usr/share/astguiclient/AST_conf_update.pl --debugX

# 7. Verify
vicidial-upgrade-verify.sh              # Or manual checks (Section 7)

SQL Upgrade Script Order

upgrade_2.0.5.sql -> upgrade_2.2.sql -> upgrade_2.4.sql -> upgrade_2.6.sql
-> upgrade_2.8.sql -> upgrade_2.10.sql -> upgrade_2.12.sql -> upgrade_2.14.sql

Never skip. Always sequential. Always use -f flag.

Emergency Rollback

# Revert code:
cd /usr/src/astguiclient/trunk && svn update -r PREVIOUS_REVISION && perl install.pl

# Restore database:
mysql -u root -p -e "DROP DATABASE asterisk; CREATE DATABASE asterisk;"
zcat /root/upgrade-backup-*/asterisk-db-*.sql.gz | mysql -u root -p asterisk

# Restore configs:
cp -a /root/upgrade-backup-*/configs/asterisk-etc/* /etc/asterisk/
cp /root/upgrade-backup-*/configs/astguiclient.conf /etc/astguiclient.conf

# Restart services:
systemctl restart mariadb && systemctl restart apache2 && systemctl restart asterisk

Key File Locations

Safe-to-Ignore SQL Errors

ERROR 1050: Table already exists
ERROR 1060: Duplicate column name
ERROR 1061: Duplicate key name
ERROR 1062: Duplicate entry

NOT-Safe-to-Ignore SQL Errors

ERROR 1045: Access denied
ERROR 2002: Can't connect to local MySQL server
ERROR 1044: Access denied for user to database
ERROR 1114: The table is full
ERROR 1030: Got error 28 from storage engine  (disk full)

15. Sources and Further Reading

Primary Sources Referenced

Key CVE References

Version Compatibility Matrix

Community Resources

• ViciDial Forum: https://www.vicidial.org/VICIDIALforum/ -- Search before posting; most issues have been discussed
• ViciDial Manager Manual: Available in the admin interface under Documentation
• ViciBox ISO downloads: https://vicibox.com -- Latest ViciBox appliance ISOs
• SVN Repository: svn://svn.eflo.net/agc_2-X/trunk -- Source code and upgrade scripts

Final Notes

The Golden Rules of ViciDial Upgrades

1. Always back up first. A bad backup is worse than no backup -- test it.
2. Never skip SQL upgrade versions. 2.4 to 2.6 to 2.8 to 2.10 to 2.12 to 2.14. Always sequential.
3. SQL scripts run on the DB server ONCE. install.pl runs on EVERY server.
4. Read the UPGRADE file after every SVN update. It tells you what changed.
5. Test on a clone first for major upgrades. The cost of a test VM is trivial compared to the cost of a down call center.
6. Keep your rollback path clear. Know your previous SVN revision. Know where your backup is. Know the restore commands by heart.
7. Upgrade regularly. Small, frequent updates (monthly SVN updates) are far safer than massive jumps (years between updates). The 88-reply horror thread is full of people who waited years and then tried to jump 5 versions at once.

When to Call for Help

If you encounter issues not covered in this playbook:

• Search the ViciDial forum first (most issues have been discussed)
• Post on the ViciDial forum with your exact versions, error messages, and what you already tried
• Consider paid ViciDial support from the official team or experienced consultants

An upgrade gone wrong can cost thousands in lost calls and agent downtime. Professional help for a few hundred dollars is cheap insurance.

This playbook was compiled from real-world ViciDial operations, community forum discussions, official documentation, and production upgrade experiences across multiple ViciDial clusters. Every procedure has been verified against the ViciDial SVN source code and community-reported outcomes.