Asterisk Core Dump & Crash Analysis — Debugging Segfaults

Learn how to capture, analyze, and fix segmentation faults in production Asterisk systems running ViciDial by examining core dumps, generating backtraces, and identifying memory corruption and module conflicts.

Prerequisites

Before you start debugging core dumps, ensure you have:

• Root access to your Asterisk/ViciDial server
• GDB (GNU Debugger) installed: apt-get install gdb or yum install gdb
• Asterisk debug symbols (compile from source with debug flags or install -debuginfo package)
• Basic familiarity with Asterisk CLI commands and ViciDial architecture
• Backup of your running system before making debugging changes
• At least 2GB free disk space in /var/spool/asterisk/ for core dumps
• Knowledge of your Asterisk version (run asterisk -V)

Understanding Core Dumps

What is a Core Dump?

A core dump is a file containing the memory snapshot of your process at the moment it crashed. When Asterisk segfaults, the kernel writes this memory to disk (if enabled), allowing you to inspect the call stack, variable states, and identify what caused the crash.

Why Core Dumps Matter in Production

In ViciDial environments, unpredictable crashes disrupt:

• Active agent calls
• Inbound call routing
• Database synchronization
• Lead distribution
• Campaign execution

Core dumps let you identify whether crashes are caused by:

• Custom modules or dialplan issues
• SIP peer misconfigurations
• Memory leaks in third-party codecs
• Race conditions in high-concurrency scenarios
• Incompatible module versions

Enabling Core Dumps on Linux

Step 1: Configure Kernel to Write Core Dumps

Check current core dump limit:

ulimit -c

If output is 0, core dumps are disabled. Enable unlimited core dumps:

ulimit -c unlimited

Make this permanent by adding to /etc/security/limits.conf:

# Allow asterisk user to generate unlimited core dumps
asterisk soft core unlimited
asterisk hard core unlimited
asterisk soft nofile 65536
asterisk hard nofile 65536

Verify:

su - asterisk -s /bin/bash -c "ulimit -c"

Expected output: unlimited

Step 2: Set Core Dump Location

Define where core files are written. Edit /etc/sysctl.conf:

kernel.core_pattern = /var/spool/asterisk/core-%e-%p-%t
kernel.core_uses_pid = 1

Apply immediately:

sysctl -p

This creates files like /var/spool/asterisk/core-asterisk-12345-1703020800 when crashes occur.

Step 3: Ensure Directory Permissions

mkdir -p /var/spool/asterisk
chown asterisk:asterisk /var/spool/asterisk
chmod 755 /var/spool/asterisk

Configuring Asterisk for Debug Output

Enable Core Dumps in asterisk.conf

Edit /etc/asterisk/asterisk.conf:

[options]
; Enable core dumps even when run as daemon
dumpcore = yes

; Log all debug output
debug = 3

; Verbose logging for call flow
verbose = 3

; Log warnings about lock contention
lockingtest = yes

Restart Asterisk (gracefully):

asterisk -rx "core restart graceful"

Wait for all channels to clear before Asterisk terminates.

Enable Module-Specific Debug

For SIP debugging (ViciDial primary protocol):

asterisk -rx "sip set debug on"
asterisk -rx "sip set debug ip 192.168.1.100"  # Debug single peer

For channel/bridge issues:

asterisk -rx "core set debug 3"

View live logs:

tail -f /var/log/asterisk/messages

Compiling Asterisk with Debug Symbols

If your Asterisk binary lacks debug symbols, backtraces will be useless. Recompile:

cd /usr/src/asterisk-16.x.x
./configure CFLAGS="-g -O0" CXXFLAGS="-g -O0"
make clean
make -j4
make install

Key flags:

• -g: Include debug symbols
• -O0: Disable optimization (makes symbols more readable; impacts performance)

Production tip: Compile with -g -O2 to balance debugging capability with performance.

Verify symbols are present:

file /usr/sbin/asterisk

Should show: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, with debug_info

nm -l /usr/sbin/asterisk | grep asterisk_log | head -5

Should return symbol entries, not empty output.

Triggering and Capturing a Crash

Safe Testing: Use Test Dialplan

Create a crashing dialplan in /etc/asterisk/extensions-test.conf:

[test-crash]
exten => 999,1,NoOp(Testing crash scenario)
exten => 999,n,SIPNotify(reboot,${CHANNEL})
exten => 999,n,Wait(2)
exten => 999,n,Hangup()

Or trigger via asterisk CLI:

asterisk -rx "core set debug 5"
asterisk -rx "dialplan eval ${SOME_BROKEN_VAR}"  # Intentional bad variable

Identify Recent Crashes

ls -lah /var/spool/asterisk/core-* 2>/dev/null | tail -10

ls -lah /tmp/core.* 2>/dev/null | tail -10

Check system logs for segmentation fault:

journalctl -xe | grep -i segfault
dmesg | grep -i segfault
tail -50 /var/log/syslog | grep -i "segmentation\|core dump"

Example journal output:

Nov 23 14:32:01 vicidial-01 kernel: [34567.891234] asterisk[12345]: segfault at 0x0 ip 00007ffff5a2c000 sp 00007fffabc12345 error 4

Analyzing Core Dumps with GDB

Step 1: Locate the Core File

ls -lah /var/spool/asterisk/core-asterisk-* | tail -1

Output example:

-rw------- 1 asterisk asterisk 324M Nov 23 14:32 /var/spool/asterisk/core-asterisk-12345-1700731920

Step 2: Start GDB Session

gdb /usr/sbin/asterisk /var/spool/asterisk/core-asterisk-12345-1700731920

You should see:

GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Reading symbols from /usr/sbin/asterisk...
Reading symbols from /usr/lib/debug/.build-id/...
[New LWP 12345]
[New LWP 12346]
...
Core was generated by `/usr/sbin/asterisk -f'
Program terminated with signal SIGSEGV, Segmentation fault.

Step 3: Generate Full Backtrace

(gdb) thread apply all bt full

This shows the call stack for all threads. Example output:

Thread 47 (LWP 12389):
#0  0x00007ffff5a2c000 in chan_sip_pvt_lock () from /usr/lib/asterisk/modules/chan_sip.so
#1  0x00007ffff5a1f234 in sip_pvt_lock_full () from /usr/lib/asterisk/modules/chan_sip.so
#2  0x00007ffff5a3d123 in handle_incoming () from /usr/lib/asterisk/modules/chan_sip.so
#3  0x00007ffff5a2e456 in sip_io_read () from /usr/lib/asterisk/modules/chan_sip.so
#4  0x000055555555c789 in ast_channel_tech_read () at channel.c:5432
#5  0x000055555556b012 in __ast_read () at channel.c:4123
#6  0x000055555556c234 in ast_read () at channel.c:4234
#7  0x00007ffff7890000 in bridge_channel_wait_for_frame () from /usr/lib/asterisk/modules/app_bridging.so
#8  0x00007ffff7891200 in bridge_channel_run () from /usr/lib/asterisk/modules/app_bridging.so
#9  0x000055555558c345 in ast_spawn_extension () at pbx.c:3912
#10 0x000055555558c600 in pbx_exec () at pbx.c:4012
#11 0x00007ffff6234567 in handle_request_invite () from /usr/lib/asterisk/modules/chan_sip.so

Step 4: Focus on the Crashing Thread

Find the thread that crashed (typically reported first):

(gdb) thread 1
(gdb) bt

Limit output to top 20 frames:

(gdb) bt 20

Step 5: Inspect Variables at Crash Point

(gdb) frame 0
(gdb) info local
(gdb) print *variablename
(gdb) print sizeof(structname)

Example: Check if a pointer is NULL:

(gdb) print pvt
$1 = (struct sip_pvt *) 0x0

A NULL pointer dereference is your culprit.

Step 6: Examine SIP Channel State

For ViciDial crashes, inspect the SIP channel context:

(gdb) frame 2
(gdb) print pvt->callid
(gdb) print pvt->dialplan
(gdb) print pvt->thestate
(gdb) print pvt->lastinvite

Common Crash Patterns in ViciDial

Pattern 1: Null Pointer Dereference in SIP Module

Symptom: Crash in chan_sip.so when handling INVITE

Backtrace indicator:

#0  0x... in chan_sip_pvt_lock
#1  0x... in handle_incoming
#2  0x... in sip_io_read

Root cause: SIP peer object deallocated while handling incoming request.

Debugging:

(gdb) print pvt
$1 = (struct sip_pvt *) 0x0
(gdb) print dialog
$2 = (struct sip_pvt *) 0x0

Fix: Add NULL check in custom dialplan or module. Check /etc/asterisk/sip-vicidial.conf for stale peer configs:

asterisk -rx "sip show peers" | grep -i "UNREACHABLE\|LAGGED"

Remove outdated peers:

asterisk -rx "sip prune peers"

Pattern 2: Memory Leak Leading to Crash

Symptom: Asterisk runs fine initially, crashes after hours/days of operation.

Indicator: Check memory usage:

ps aux | grep asterisk | grep -v grep
# Look at RSS column (resident set size)

Monitor over time:

watch -n 5 'ps aux | grep asterisk | grep -v grep'

If RSS grows continuously, memory leak is present. Backtrace will show:

#0  0x... in malloc (or realloc)

Debugging: Enable MALLOC_DEBUG in Asterisk:

Edit /etc/asterisk/asterisk.conf:

[options]
allowmultiple = no
astdb = /var/lib/asterisk/astdb.sqlite3
debug = 3
verbose = 3
dumpcore = yes

Recompile Asterisk with:

./configure --with-malloc-debug
make
make install

Then run GDB and check memory allocator:

(gdb) handle SIG35 noprint nostop
(gdb) continue
# ... let it run and crash ...
(gdb) bt
(gdb) print heapmgr

Pattern 3: Race Condition in Database Access

Symptom: Intermittent crashes when multiple channels access ViciDial database tables simultaneously.

Affected tables: vicidial_log, vicidial_closer_log, vicidial_carrier_log

Backtrace indicator:

#0  0x... in ast_db_get
#1  0x... in ast_app_parse_timezones
#2  0x... in custom_app_exec

Debugging: Check for missing table locks in custom AGI scripts or dialplan:

grep -r "astdb\|ast_db_get" /etc/asterisk/extensions-vicidial.conf | head -20

Fix: Use dialplan locks when accessing shared resources:

exten => 100,1,Lock(vicidial-db-lock)
exten => 100,n,NoOp(Safe database access)
exten => 100,n,Unlock(vicidial-db-lock)

Or use database transactions in AGI scripts:

#!/bin/bash
# Safe database query in AGI
(
  flock -x 200
  mysql -N -e "SELECT * FROM vicidial_log WHERE lead_id = $LEAD_ID LIMIT 1" asterisk
) 200>/var/lock/vicidial-db.lock

Pattern 4: Module Incompatibility

Symptom: Crash when loading custom module or third-party codec.

Backtrace indicator:

#0  0x... in module_symbol_not_found
#1  0x... in dlopen

Debugging: Check loaded modules:

asterisk -rx "module show loaded" | grep -i "custom\|third"

Check module dependencies:

ldd /usr/lib/asterisk/modules/codec_custom.so | grep -i "not found"

Fix: Rebuild module with correct Asterisk version:

cd /usr/src/asterisk-16.x.x/addons
make clean
make
make install

Verify module loads without errors:

asterisk -rx "module load codec_custom"
asterisk -rx "core show settings" | grep "Build Date"

Creating a Core Dump Archive for Analysis

Preserve crashes for later investigation or vendor support:

#!/bin/bash
# save_core_dump.sh

CORE_FILE=$1
ARCHIVE_DIR="/var/spool/asterisk/crash-archives"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

mkdir -p $ARCHIVE_DIR

# Tar the core and associated binaries
tar -czf $ARCHIVE_DIR/crash-$TIMESTAMP.tar.gz \
  $CORE_FILE \
  /usr/sbin/asterisk \
  /usr/lib/asterisk/modules/*.so \
  /etc/asterisk/ \
  /var/log/asterisk/messages \
  2>/dev/null

# Generate backtrace for quick reference
gdb -batch -ex "bt" /usr/sbin/asterisk $CORE_FILE > \
  $ARCHIVE_DIR/crash-$TIMESTAMP.backtrace.txt 2>&1

echo "Core dump archived: $ARCHIVE_DIR/crash-$TIMESTAMP.tar.gz"
echo "Backtrace: $ARCHIVE_DIR/crash-$TIMESTAMP.backtrace.txt"

Usage:

chmod +x /usr/local/bin/save_core_dump.sh
/usr/local/bin/save_core_dump.sh /var/spool/asterisk/core-asterisk-12345-1700731920

Automated Crash Detection and Response

Create a Core Dump Watcher

#!/bin/bash
# /usr/local/bin/asterisk-crash-monitor.sh

CORE_DIR="/var/spool/asterisk"
LOG_FILE="/var/log/asterisk/crash-monitor.log"
ALERT_EMAIL="[email protected]"

check_for_new_cores() {
  find $CORE_DIR -name "core-*" -type f -mmin -5 2>/dev/null | while read core_file; do
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    
    # Extract backtrace
    backtrace=$(gdb -batch -ex "bt 30" /usr/sbin/asterisk "$core_file" 2>&1 | tail -50)
    
    # Log incident
    echo "[$timestamp] Crash detected: $core_file" >> $LOG_FILE
    echo "$backtrace" >> $LOG_FILE
    echo "---" >> $LOG_FILE
    
    # Send alert
    echo "$backtrace" | mail -s "Asterisk Crash: $(hostname)" $ALERT_EMAIL
    
    # Restart Asterisk if needed
    if ! pgrep -x asterisk > /dev/null; then
      systemctl start asterisk
      echo "[$timestamp] Asterisk restarted" >> $LOG_FILE
    fi
  done
}

# Run every 2 minutes
while true; do
  check_for_new_cores
  sleep 120
done

Install as systemd service:

# /etc/systemd/system/asterisk-crash-monitor.service
[Unit]
Description=Asterisk Crash Monitor
After=network.target
Wants=asterisk.service

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/asterisk-crash-monitor.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable:

systemctl daemon-reload
systemctl enable asterisk-crash-monitor
systemctl start asterisk-crash-monitor

GDB Automation Script

For unattended backtrace generation:

#!/bin/bash
# /usr/local/bin/extract_backtrace.sh

CORE_FILE=$1
BINARY="/usr/sbin/asterisk"
OUTPUT="${CORE_FILE}.backtrace.txt"

gdb -batch \
  -ex "thread apply all bt full" \
  -ex "quit" \
  $BINARY $CORE_FILE > $OUTPUT 2>&1

echo "Backtrace written to: $OUTPUT"
cat $OUTPUT

Usage:

/usr/local/bin/extract_backtrace.sh /var/spool/asterisk/core-asterisk-12345-1700731920

ViciDial-Specific Debugging Tips

Check for Lead Lock Deadlocks

ViciDial's lead distribution can deadlock when multiple processes compete for the same lead:

asterisk -rx "database show vicidial-lead-locks" | head -20

If locks don't clear within seconds, investigate:

mysql> SELECT * FROM vicidial_log WHERE lead_id = <LEAD_ID> ORDER BY entry_date DESC LIMIT 1\G

Clear stale locks manually (if safe):

asterisk -rx "database deltree vicidial-lead-locks"

Monitor Call Bridge Issues

ViciDial bridges agent and carrier channels. Crashes often occur at bridge setup:

asterisk -rx "core set debug 3"
asterisk -rx "bridge show all"

Check for stuck bridges:

asterisk -rx "channel show all" | wc -l

If channel count grows unbounded, bridge leak exists. Restart may be necessary:

asterisk -rx "core restart graceful"

Review Dialplan Syntax

Dialplan errors can trigger crashes in edge cases. Validate:

asterisk -rx "dialplan reload"

Check for warnings in /var/log/asterisk/messages:

grep -i "warning\|error" /var/log/asterisk/messages | tail -50

Inspect AGI Script Failures

Custom AGI scripts sometimes cause Asterisk to crash. Test scripts in isolation:

/var/lib/asterisk/agi-bin/vicidial_loader.agi test

Add error handling to AGI scripts:

#!/bin/bash
trap 'echo "HANGUP" >&3' EXIT
trap 'exit' SIGTERM

while read -t 1 line; do
  case "$line" in
    agi_uniqueid:*) UNIQUEID="${line#*: }" ;;
    agi_calleridnum:*) CALLERID="${line#*: }" ;;
  esac
  [ -z "$line" ] && break
done

echo "EXEC Hangup" >&3

Troubleshooting

Issue: Core Dumps Not Being Generated

Check 1: ulimit

ulimit -c

Should return unlimited. If not:

ulimit -c unlimited
echo "ulimit -c unlimited" >> /etc/profile

Check 2: File System Space

df -h /var/spool/asterisk

If full, clean old cores:

rm -f /var/spool/asterisk/core-* 

Check 3: SELinux

If enabled, SELinux may block core dumps:

getenforce

If Enforcing, add policy:

semanage fcontext -a -t asterisk_var_spool_t "/var/spool/asterisk(/.*)?"
restorecon -Rv /var/spool/asterisk

Issue: GDB Backtrace Shows "??" Instead of Function Names

Cause: Missing debug symbols.

Fix: Recompile Asterisk with -g flag:

./configure CFLAGS="-g -O2"
make -j4
make install

Verify:

gdb /usr/sbin/asterisk -batch -ex "file /usr/sbin/asterisk" -ex "quit" 2>&1 | grep "debug_info"

Issue: GDB Takes Too Long to Load Large Core Files

Workaround: Use gdb --nx to skip initialization:

gdb --nx /usr/sbin/asterisk /var/spool/asterisk/core-asterisk-12345-1700731920

Or extract backtrace without full symbols:

gdb -batch -ex "bt" /usr/sbin/asterisk /var/spool/asterisk/core-asterisk-12345-1700731920 2>&1 | head -100

Issue: Asterisk Keeps Crashing in Infinite Loop

Diagnosis: Check crash frequency:

ls -lh /var/spool/asterisk/core-* | wc -l

Disable automatic restart temporarily:

systemctl stop asterisk

Run Asterisk in foreground with debug:

asterisk -f -d -v -v -v

Stop at first crash with Ctrl+C. Review console output for error messages.

Issue: Backtrace Points to Third-Party Module, But Module Isn't Loaded

Cause: Module was loaded in a previous process instance, shared library is corrupted.

Fix: Force reload all modules:

asterisk -rx "module reload FORCE"

If crash persists:

asterisk -rx "module unload extra/codec_custom"

Rebuild module:

cd /usr/src/asterisk-16.x.x/addons
make clean
./configure --with-asterisk=/usr
make
make install

Summary

Debugging Asterisk core dumps in production ViciDial environments requires:

1. Preparation: Enable core dumps at OS, kernel, and Asterisk levels. Compile with debug symbols.

2. Capture: Monitor /var/spool/asterisk/ for new core files. Log crash timestamps and associated system state.

3. Analysis: Use GDB to extract full backtraces with thread apply all bt full. Focus on the faulting thread. Inspect variable states at crash point.

4. Pattern Recognition: Know common ViciDial crash patterns—SIP channel deallocation, database race conditions, memory leaks, module incompatibilities.

5. ViciDial-Specific Checks: Verify lead locks aren't stuck, call bridges aren't leaking, dialplan is valid, AGI scripts have error handling.

6. Response: Automate crash detection with systemd services. Generate backtrace archives for vendor support. Implement graceful restarts to minimize call loss.

7. Remediation: Address root causes—patch Asterisk, rebuild incompatible modules, refactor dialplan, fix AGI scripts, update SIP peers.

Core dumps are your most powerful debugging tool in production. Invest in capturing and preserving them. A single backtrace can save weeks of troubleshooting and prevent recurring crashes that destroy agent productivity and customer experience.

Keep backtraces, logs, and configurations together when escalating to Asterisk support or ViciDial engineers. Always test fixes in staging before deploying to production.