ViciDial Hopper Not Loading Leads — Troubleshooting the Lead Hopper

Learn how to diagnose and fix lead hopper loading failures in ViciDial by checking database configurations, API endpoints, dial groups, and backend processes.

Prerequisites

Before troubleshooting your ViciDial lead hopper, ensure you have:

• SSH access to your ViciDial server with sudo privileges
• MySQL/MariaDB command-line access to the asterisk database
• Familiarity with basic ViciDial terminology (campaigns, dial groups, leads, hoppers)
• Access to ViciDial's web admin interface at /vicidial/admin.php
• Knowledge of basic Linux commands and log file inspection
• A test campaign and dial group already configured
• At least one lead loaded in the system to test with

This guide applies to ViciDial versions 2.13+ running on Asterisk 13 or later.

Understanding the ViciDial Lead Hopper Architecture

The ViciDial lead hopper is a queue management system that delivers leads to agents in a specific order based on campaign settings. When leads don't load, the failure typically occurs at one of four stages:

1. Lead retrieval from the database
2. Lead filtering based on dial group rules
3. Hopper population by the backend loader process
4. Agent delivery through the web interface

The hopper load process involves multiple components:

• The vicidial_hopper table (temporary storage)
• The loader.pl Perl script (backend lead loader)
• Campaign and dial group configurations
• The vicidial_user_log table (session tracking)

Section 1: Verify Basic Campaign and Dial Group Configuration

Check Campaign Settings

Start by confirming your campaign exists and has appropriate settings:

mysql -u cron -pPASSWORD asterisk -e "
SELECT campaign_id, campaign_name, dial_method, dial_filter_id, 
       leads_per_cycle, hopper_level, lead_order 
FROM vicidial_campaigns 
WHERE campaign_id = 'TEST_CAMP' \G"

Key fields to verify:

• dial_method: Should be "MANUAL" (agents select leads), "PREVIEW" (agents preview), "POWER" (auto-dial), or "INBOUND"
• hopper_level: The target number of leads to keep in hopper (default 5)
• lead_order: How leads are ordered ("RANDOM", "OLDEST", "NEWEST", etc.)
• leads_per_cycle: Number of leads loaded per cycle

If your campaign has dial_method = 'MANUAL', agents must explicitly request leads. Ensure this is intentional.

Verify Dial Group Configuration

Dial groups control which leads are eligible for dialing:

mysql -u cron -pPASSWORD asterisk -e "
SELECT dial_group_id, campaign_id, dial_method, lead_filter_id, 
       next_dial_time, dialable_leads 
FROM vicidial_dial_groupz 
WHERE dial_group_id = 'TEST_DG' \G"

Critical settings:

• lead_filter_id: If set to a non-zero value, only leads matching that filter are loaded
• dialable_leads: Should be > 0 (check next section if it's 0)
• next_dial_time: If this is in the future, the dial group is on cooldown

Check if the dial group's next_dial_time is preventing loads:

mysql -u cron -pPASSWORD asterisk -e "
SELECT dial_group_id, next_dial_time, NOW() as current_time,
       TIMESTAMPDIFF(SECOND, NOW(), next_dial_time) as seconds_until_ready
FROM vicidial_dial_groupz 
WHERE dial_group_id = 'TEST_DG';"

If the time difference is positive, the dial group is throttled. This is by design but often misconfigured.

Section 2: Verify Lead Availability

Count Available Leads

Check how many leads exist for your campaign:

SELECT COUNT(*) as total_leads, 
       SUM(CASE WHEN status = 'NEW' THEN 1 ELSE 0 END) as new_leads,
       SUM(CASE WHEN status = 'CALL' THEN 1 ELSE 0 END) as call_leads,
       SUM(CASE WHEN status = 'CALLBK' THEN 1 ELSE 0 END) as callback_leads
FROM vicidial_list 
WHERE campaign_id = 'TEST_CAMP';

The most common issue: all leads have been exhausted or filtered out. Check the status distribution:

SELECT status, COUNT(*) as count 
FROM vicidial_list 
WHERE campaign_id = 'TEST_CAMP' 
GROUP BY status 
ORDER BY count DESC;

Leads must have status in the campaign's dialable_status list to be eligible. Check your campaign's dialable status settings in the web admin, or query:

mysql -u cron -pPASSWORD asterisk -e "
SELECT campaign_id, dialable_status 
FROM vicidial_campaigns 
WHERE campaign_id = 'TEST_CAMP';"

The dialable_status field is a pipe-delimited list (e.g., NEW|CALL|CALLBK). Leads with any status not in this list are ineligible.

Check for Lead Filters

If a lead_filter_id is assigned, verify the filter hasn't excluded all leads:

SELECT * FROM vicidial_filters 
WHERE filter_id = 5;

Then test the filter manually to see which leads would match:

mysql -u cron -pPASSWORD asterisk -e "
SELECT COUNT(*) as matching_leads 
FROM vicidial_list 
WHERE campaign_id = 'TEST_CAMP' 
AND status IN ('NEW', 'CALL') 
AND phone_number REGEXP '^[0-9]{10}$';"

(Adjust the WHERE clause based on your actual filter criteria.)

Verify Dial Group Has Dialable Leads

Check the dialable_leads count for your dial group:

mysql -u cron -pPASSWORD asterisk -e "
SELECT dial_group_id, campaign_id, dialable_leads, active 
FROM vicidial_dial_groupz 
WHERE dial_group_id = 'TEST_DG';"

If dialable_leads = 0, no leads match the dial group's criteria. The dialable_leads count is updated by the loader.pl script. If it's stuck at 0, the loader may not be running correctly (see Section 4).

Section 3: Inspect the Hopper Table

Check Current Hopper Contents

The vicidial_hopper table holds leads temporarily assigned to the current session:

SELECT hopper_id, lead_id, campaign_id, user_id, phone_number, 
       status, load_id, called_count 
FROM vicidial_hopper 
WHERE campaign_id = 'TEST_CAMP' 
LIMIT 10;

If this returns no rows, the hopper is empty. Check if leads were ever populated:

SELECT COUNT(*) as hopper_count 
FROM vicidial_hopper 
WHERE campaign_id = 'TEST_CAMP';

Check Hopper Load History

The vicidial_hopper_log table (if available) tracks hopper loads:

SELECT * FROM vicidial_hopper_log 
WHERE campaign_id = 'TEST_CAMP' 
ORDER BY load_date DESC 
LIMIT 20;

This shows when leads were last loaded and how many. If timestamps are old, the loader isn't running.

Verify Session is Active

Leads are only loaded for active user sessions. Check your current session:

SELECT user_id, campaign_id, session_id, login_date, last_update, 
       status 
FROM vicidial_user_log 
WHERE user_id = 'AGENT001' 
AND logout_date IS NULL;

The last_update timestamp should be recent (within the last minute for an active agent). If it's old, the agent's session may have timed out.

Section 4: Verify the Loader Process is Running

Check if loader.pl is Active

The loader.pl script is responsible for populating hoppers:

ps aux | grep -i loader | grep -v grep

Output should show:

root      1234  0.5  2.1  85432 21456 ?  Ss  10:32  0:15  /usr/bin/perl /usr/share/astguiclient/loader.pl

If you see nothing, the loader is not running. Check why:

systemctl status vicidial-loader

Or check the startup script directly:

cat /etc/init.d/astguiclient | grep -A5 loader

Start the Loader Manually

If the loader crashed, restart it:

/etc/init.d/astguiclient restart

Or start just the loader:

nohup /usr/bin/perl /usr/share/astguiclient/loader.pl > /var/log/vicidial_loader.log 2>&1 &

Monitor Loader Output in Real-Time

Check the loader's actual execution:

tail -f /var/log/vicidial_loader.log

Look for errors like:

• "Cannot connect to database"
• "Permission denied"
• "Dial group X has no leads"

If you don't see these logs, the script isn't writing output. Check the log file path in /usr/share/astguiclient/loader.pl:

grep -n "log_file\|open.*LOG" /usr/share/astguiclient/loader.pl | head -5

Check Loader Configuration

Review the loader's database connection settings:

grep -A10 "^my \$" /usr/share/astguiclient/loader.pl | grep -i "mysql\|host\|pass"

Ensure the credentials match your MySQL setup:

mysql -u cron -pPASSWORD -h localhost -e "SELECT 'Connection OK';"

Section 5: Review ViciDial Log Files

Check Asterisk Messages Log

Asterisk logs related hopper and dial group activity:

tail -100 /var/log/asterisk/messages | grep -i "hopper\|loader\|dial_group"

Look for errors:

[2024-01-15 10:32:45] WARNING: hopper overflow on campaign TEST_CAMP
[2024-01-15 10:33:12] ERROR: cannot load leads for dial group TEST_DG

Check ViciDial-Specific Logs

If /var/log/vicidial/ exists, check those logs:

ls -lah /var/log/vicidial/
tail -50 /var/log/vicidial/*.log

Enable Debug Logging in loader.pl

If you suspect the loader is running but silently failing, enable debug output. Edit /usr/share/astguiclient/loader.pl and find the $debug variable:

my $debug = 0;  # Change to 1

Change it to:

my $debug = 1;

Then restart the loader and check output again.

Section 6: Test the Hopper Manually

Test as an Agent: Manual Dial Method

Log into the agent interface at /agc/vicidial.php and attempt to get leads:

1. Select the campaign with the hopper issue
2. Click "LOAD LEADS" button if your dial method is MANUAL
3. Check the browser console (F12) for JavaScript errors
4. Note any error messages displayed on-screen

Query Hopper Load Directly from CLI

Simulate a hopper load by running the load query manually:

SELECT lead_id, phone_number, status, called_count 
FROM vicidial_list 
WHERE campaign_id = 'TEST_CAMP' 
AND status IN ('NEW', 'CALL', 'CALLBK') 
AND (called_count < 3 OR called_count IS NULL)
ORDER BY RAND() 
LIMIT 5;

If this returns no rows, there are no eligible leads to load.

Test Hopper Insert Manually

Try inserting a test hopper record manually to verify the database isn't locked or corrupted:

INSERT INTO vicidial_hopper 
(lead_id, campaign_id, user_id, phone_number, status) 
VALUES (99999, 'TEST_CAMP', 'ADMIN', '5551234567', 'NEW');

If this fails with a permission error, check table permissions:

SHOW GRANTS FOR 'cron'@'localhost';

Ensure the cron user has INSERT privileges on vicidial_hopper.

Section 7: Check Agent-Specific Issues

Verify Agent Exists and is Active

An agent must have an active user account in the campaign:

SELECT user_id, full_name, user_level, campaign_id, active 
FROM vicidial_users 
WHERE user_id = 'AGENT001';

Check for campaign assignment:

SELECT user_id, campaign_id, allowed_campaignz 
FROM vicidial_user_campaignz 
WHERE user_id = 'AGENT001';

Check User Session Timeout

Long-idle agents may have their sessions terminate. ViciDial cleans up sessions with last_update older than 15 minutes:

mysql -u cron -pPASSWORD asterisk -e "
SELECT user_id, campaign_id, last_update, 
       TIMESTAMPDIFF(MINUTE, last_update, NOW()) as idle_minutes
FROM vicidial_user_log 
WHERE user_id = 'AGENT001' 
AND logout_date IS NULL;"

If idle_minutes > 15, the session may be invalidated. Have the agent log out and back in.

Check Agent's Dial Group Assignment

Agents don't directly request leads; dial groups do. Verify the agent's campaign has an active dial group:

SELECT dial_group_id, campaign_id, active, dialable_leads 
FROM vicidial_dial_groupz 
WHERE campaign_id = 'TEST_CAMP' 
AND active = 'Y';

Section 8: Database and Table Health

Check for Table Locks

If the hopper table is locked, no loads occur:

mysql -u cron -pPASSWORD asterisk -e "SHOW OPEN TABLES WHERE In_use > 0;"

If vicidial_hopper appears here, find the blocking process:

mysql -u cron -pPASSWORD asterisk -e "SHOW PROCESSLIST \G" | grep -A5 -B5 "vicidial_hopper"

Kill the blocking query if safe:

mysql -u cron -pPASSWORD asterisk -e "KILL 1234;"  # Replace 1234 with process ID

Repair Corrupted Tables

If tables are corrupted, MySQL reports errors. Repair:

mysql -u cron -pPASSWORD asterisk -e "REPAIR TABLE vicidial_hopper;"
mysql -u cron -pPASSWORD asterisk -e "CHECK TABLE vicidial_hopper;"

Verify Database Connectivity

Confirm the loader can connect to MySQL:

mysql -u cron -pPASSWORD -h 127.0.0.1 asterisk -e "SELECT COUNT(*) FROM vicidial_list;"

If this fails, update /usr/share/astguiclient/loader.pl with correct credentials.

Section 9: Dial Group and Campaign Timing Issues

Check Dial Group Cooldown

Dial groups have a next_dial_time field that throttles lead loading:

SELECT dial_group_id, campaign_id, next_dial_time, NOW() as current_time
FROM vicidial_dial_groupz 
WHERE campaign_id = 'TEST_CAMP';

If next_dial_time is in the future, manually reset it:

UPDATE vicidial_dial_groupz 
SET next_dial_time = NOW() 
WHERE dial_group_id = 'TEST_DG';

Check Campaign Inactivity Settings

Some campaigns auto-disable if inactive:

SELECT campaign_id, active, active_disable_after_minutes 
FROM vicidial_campaigns 
WHERE campaign_id = 'TEST_CAMP';

Reactivate if needed:

UPDATE vicidial_campaigns 
SET active = 'Y', last_load_date = NOW() 
WHERE campaign_id = 'TEST_CAMP';

Verify Dial Group is Active

UPDATE vicidial_dial_groupz 
SET active = 'Y' 
WHERE dial_group_id = 'TEST_DG';

Section 10: API and Integration Issues

Check if Hopper Load is Called via API

Some deployments load hoppers via an external API instead of the built-in loader. Check for API calls in logs:

grep -r "hopper\|load_leads" /var/log/apache2/ /var/log/nginx/ 2>/dev/null | head -20

Verify the API endpoint is receiving requests and returning data. Test manually:

curl -X GET "http://your-vicidial-server/api/hopper?campaign_id=TEST_CAMP&user_id=AGENT001"

Check Web Server Error Logs

If hoppers load via web interface, check web server logs:

tail -50 /var/log/apache2/error.log
tail -50 /var/log/nginx/error.log

Look for PHP errors in ViciDial's hopper-loading scripts.

Troubleshooting Checklist

Use this checklist to isolate the issue systematically:

# 1. Verify campaign exists and is active
mysql -u cron -pPASSWORD asterisk -e "SELECT campaign_id, active FROM vicidial_campaigns WHERE campaign_id = 'TEST_CAMP';"

# 2. Count available leads
mysql -u cron -pPASSWORD asterisk -e "SELECT COUNT(*) FROM vicidial_list WHERE campaign_id = 'TEST_CAMP' AND status IN ('NEW', 'CALL');"

# 3. Check loader is running
ps aux | grep -i loader | grep -v grep

# 4. Verify dial group exists and has dialable leads
mysql -u cron -pPASSWORD asterisk -e "SELECT dial_group_id, dialable_leads, active FROM vicidial_dial_groupz WHERE campaign_id = 'TEST_CAMP';"

# 5. Check hopper is empty
mysql -u cron -pPASSWORD asterisk -e "SELECT COUNT(*) FROM vicidial_hopper WHERE campaign_id = 'TEST_CAMP';"

# 6. Verify agent session is active
mysql -u cron -pPASSWORD asterisk -e "SELECT user_id, campaign_id, last_update FROM vicidial_user_log WHERE user_id = 'AGENT001' AND logout_date IS NULL;"

# 7. Check dial group next_dial_time isn't in the future
mysql -u cron -pPASSWORD asterisk -e "SELECT dial_group_id, next_dial_time FROM vicidial_dial_groupz WHERE campaign_id = 'TEST_CAMP';"

# 8. Review loader logs
tail -50 /var/log/vicidial_loader.log

# 9. Check Asterisk logs for hopper errors
tail -50 /var/log/asterisk/messages | grep -i hopper

Common Root Causes and Fixes

Summary

The ViciDial lead hopper relies on a chain of components: campaign configuration, lead availability, active loader process, database connectivity, dial group settings, and agent sessions. When leads don't load:

1. Start with basics: Confirm the campaign is active, leads exist, and the loader process is running.
2. Check lead eligibility: Verify leads have appropriate statuses and pass any active filters.
3. Inspect the hopper table: Query it directly to confirm it's empty or if leads stuck loading.
4. Review logs: Check /var/log/vicidial_loader.log and /var/log/asterisk/messages for clues.
5. Reset timing issues: Ensure dial groups aren't on cooldown with next_dial_time in the future.
6. Verify sessions: Confirm agents have active user sessions not older than 15 minutes.
7. Test manually: Insert leads into the hopper directly to isolate whether the issue is data or process-related.

In production, hopper failures are usually caused by exhausted leads, misconfigured filters, a crashed loader process, or session timeouts. By methodically working through the database queries and process checks above, you'll identify which component is failing and apply the appropriate fix.

Remember to always backup your database before making structural changes, and test fixes in a staging environment before applying to production campaigns.