Auto-Reload and Callbacks

Matchy supports automatic database reloading when files change, enabling zero-downtime updates in production systems. The auto-reload feature uses lock-free Arc swapping for minimal performance overhead.

Quick Start

Rust API

#![allow(unused)]
fn main() {
use matchy::Database;

// Enable auto-reload
let db = Database::from("threats.mxy")
    .watch()
    .open()?;

// Queries automatically use the latest database version
let result = db.lookup("192.168.1.1")?;
}

C API

#include <matchy/matchy.h>

// Configure auto-reload
matchy_open_options_t opts;
matchy_init_open_options(&opts);
opts.auto_reload = true;

matchy_t *db = matchy_open_with_options("threats.mxy", &opts);

// Queries automatically use latest version
matchy_result_t result = matchy_query(db, "192.168.1.1");
matchy_free_result(&result);

matchy_close(db);

How Auto-Reload Works

When auto-reload is enabled:

1. File watching - A background thread monitors the database file using OS notifications
2. Debouncing - File changes are debounced (200ms) to avoid rapid reload cycles
3. Background loading - New database is loaded in a background thread
4. Atomic swap - New database is atomically swapped using lock-free Arc pointer
5. Graceful handoff - Old database stays alive until all query threads finish with it

┌─────────────┐
│ Query Thread│
│  Thread 1   │──┐
└─────────────┘  │
                 │    ┌──────────────┐     ┌──────────────┐
┌─────────────┐  ├───→│  ArcSwap     │────→│  Database v1 │
│ Query Thread│  │    │ (atomic ptr) │     └──────────────┘
│  Thread 2   │──┤    └──────────────┘            │
└─────────────┘  │           ▲                     │
                 │           │                     │ (stays alive
┌─────────────┐  │    ┌──────────────┐            │  until all
│ Query Thread│  │    │    Watcher   │            │  refs drop)
│  Thread N   │──┘    │    Thread    │            │
└─────────────┘       └──────────────┘            ▼
                             │              ┌──────────────┐
                             │  (atomic     │  Database v2 │
                             └─  swap)      │  (new)       │
                                            └──────────────┘

Performance

Auto-reload uses a generation check and thread-local snapshot cache:

• Per-query path: Atomic generation check plus thread-local snapshot access
• After a reload: The first query on each thread refreshes its snapshot
• Synchronization: Query-side selection uses atomics rather than a global mutex
• Measurement: Include reload-enabled and static runs on the target workload

Performance Breakdown

#![allow(unused)]
fn main() {
// First query after reload refreshes generation-dependent state
let result = db.lookup("192.168.1.1")?;  // Check generation + cache Arc

// Subsequent queries reuse thread-local state but still perform normal checks
let result = db.lookup("192.168.1.2")?;  // Pure thread-local access
let result = db.lookup("192.168.1.3")?;  // Pure thread-local access
}

The generation check uses an atomic load. Measure its effect as part of the full query path on the target workload rather than assigning a fixed nanosecond cost.

Reload Callbacks

Get notified when database reloads occur:

Rust API

#![allow(unused)]
fn main() {
use matchy::{Database, ReloadEvent};

let db = Database::from("threats.mxy")
    .watch()
    .on_reload(|event: ReloadEvent| {
        if event.success {
            println!("✅ Database reloaded successfully");
            println!("   Path: {}", event.path.display());
            println!("   Generation: {}", event.generation);
        } else {
            eprintln!("❌ Database reload failed");
            eprintln!("   Path: {}", event.path.display());
            eprintln!("   Error: {}", event.error.unwrap());
        }
    })
    .open()?;
}

The ReloadEvent structure contains:

#![allow(unused)]
fn main() {
pub struct ReloadEvent {
    pub path: PathBuf,           // Database file path
    pub success: bool,            // Whether reload succeeded
    pub error: Option<String>,    // Error message (if failed)
    pub generation: u64,          // Generation counter
    pub source: ReloadSource,     // What triggered the reload
}
}

C API

#include <matchy/matchy.h>
#include <stdio.h>

// Callback function
void on_reload(const matchy_reload_event_t *event, void *user_data) {
    if (event->success) {
        printf("✅ Reloaded: %s (generation %lu)\n",
               event->path, event->generation);
    } else {
        fprintf(stderr, "❌ Reload failed: %s - %s\n",
                event->path, event->error);
    }
}

int main() {
    // Configure callback
    matchy_open_options_t opts;
    matchy_init_open_options(&opts);
    opts.auto_reload = true;
    opts.reload_callback = on_reload;
    opts.reload_callback_user_data = NULL;  // Optional context pointer
    
    matchy_t *db = matchy_open_with_options("threats.mxy", &opts);
    
    // ... use database ...
    
    matchy_close(db);
    return 0;
}

Callback Safety

Important considerations:

• Callbacks run on the watcher thread, not query threads
• Keep callbacks fast and non-blocking
• Do not call matchy query functions from callbacks (potential deadlock)
• Copy event.path and event.error if you need them after callback returns
• Callbacks must be thread-safe

Use Cases

Production Threat Intelligence

#![allow(unused)]
fn main() {
// Threat database updated hourly from feed
let db = Database::from("/data/threats.mxy")
    .watch()
    .on_reload(|event| {
        if event.success {
            // Log to monitoring system
            metrics::increment_counter!("db_reload_success");
            info!("Threat database updated: generation {}", event.generation);
        } else {
            // Alert on failure
            metrics::increment_counter!("db_reload_failure");
            error!("Failed to reload threats: {:?}", event.error);
        }
    })
    .open()?;

// Queries automatically use latest threat data
for log_entry in log_stream {
    if let Some(threat @ (QueryResult::Ip { .. } | QueryResult::Pattern { .. })) =
        db.lookup(&log_entry.ip)?
    {
        alert_security_team(log_entry, threat);
    }
}
}

GeoIP Database Updates

#![allow(unused)]
fn main() {
// GeoIP database refreshed weekly
let geoip = Database::from("/data/GeoLite2-City.mmdb")
    .watch()
    .on_reload(|event| {
        println!("GeoIP database updated: {}", event.path.display());
    })
    .open()?;

// No service restart needed for updates
let location = geoip.lookup("8.8.8.8")?;
}

Multi-Process Deployment

#![allow(unused)]
fn main() {
// Worker process
let db = Arc::new(
    Database::from("threats.mxy")
        .watch()
        .open()?
);

// Spawn multiple worker threads
for i in 0..num_cpus::get() {
    let db_clone = Arc::clone(&db);
    thread::spawn(move || {
        // Each thread automatically gets reloaded database
        loop {
            let work = get_work();
            let result = db_clone.lookup(&work.query)?;
            process_result(result);
        }
    });
}
}

HTTP Auto-Update

Matchy supports automatic updates for databases that include an embedded update URL. The database uses this internal metadata to periodically check for updates and download them if changed.

Rust API

#![allow(unused)]
fn main() {
// Database must have embedded update URL (from DatabaseBuilder::with_update_url())
let db = Database::from("threats.mxy")
    .auto_update() // No URL parameter - uses embedded metadata
    .update_interval(Duration::from_secs(3600))
    .cache_dir("/var/cache/myapp") // Optional: defaults to ~/.cache/matchy/
    .on_reload(|event| {
        match event.source {
            ReloadSource::FileChange => println!("Local file changed"),
            ReloadSource::NetworkUpdate => println!("Downloaded new version"),
        }
    })
    .open()?; // Returns error if database has no embedded URL
}

The auto-update feature:

• Self-describing: Uses URL embedded in the database file (set during build)
• Safe updates: Downloads to a cache directory (~/.cache/matchy/ by default), never overwriting the original file
• Composable: Can be combined with watch() to handle both local replacements and network updates
• Efficient: Uses ETag and Last-Modified headers to avoid unnecessary downloads
• Robust: Validates the database before swapping

C API

matchy_open_options_t opts;
matchy_init_open_options(&opts);

// Enable auto-update (requires embedded URL in database)
opts.auto_update = true;
// Optional: set custom download location (formerly update_url)
opts.cache_dir = "/var/cache/myapp"; 

matchy_t *db = matchy_open_with_options("threats.mxy", &opts);

Database Update Best Practices

Atomic File Replacement

Always use atomic rename for updates:

# Build new database
matchy build new-threats.csv --input-format csv --output threats.mxy.tmp

# Atomic rename (works on all platforms)
mv threats.mxy.tmp threats.mxy

This ensures:

• No partial database reads
• Auto-reload detects the change
• Zero query errors during update

Update Scripts

#!/bin/bash
# update-threats.sh - Safe database update script

set -e

DB_PATH="/data/threats.mxy"
TEMP_DB="${DB_PATH}.tmp"

# Download and build new database
curl -o threats.csv "https://threat-feed.example.com/latest"
matchy build threats.csv --input-format csv --output "$TEMP_DB"

# Validate before deploying
matchy validate "$TEMP_DB" --level strict

# Atomic replace
mv "$TEMP_DB" "$DB_PATH"

echo "✅ Database updated successfully"

Monitoring Reloads

#![allow(unused)]
fn main() {
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;

let reload_count = Arc::new(AtomicU64::new(0));
let reload_count_clone = Arc::clone(&reload_count);

let db = Database::from("threats.mxy")
    .watch()
    .on_reload(move |event| {
        if event.success {
            reload_count_clone.fetch_add(1, Ordering::Relaxed);
        }
    })
    .open()?;

// Later: check reload metrics
let reloads = reload_count.load(Ordering::Relaxed);
println!("Database has been reloaded {} times", reloads);
}

Limitations

File System Events

• Linux: Uses inotify (requires kernel support)
• macOS: Uses FSEvents (works with atomic renames)
• Windows: Uses ReadDirectoryChangesW
• Network filesystems: May have delayed notifications (NFS, CIFS, etc.)

Debouncing

File changes are debounced for 200ms to avoid rapid reload cycles. Debouncing does not make in-place multi-stage writes safe for an existing memory map. Write a complete new database to a temporary file, fsync as required by your durability policy, and atomically replace the watched path. Never truncate or rewrite the mapped inode in place.

Memory Usage

During reload, old and new mappings can coexist until query threads release the old snapshot. Peak resident memory depends on the pages touched in each mapping, private runtime state, and how long callers retain snapshots; it is not a fixed multiple of file size.

Troubleshooting

Reload Not Triggering

Check file watcher:

#![allow(unused)]
fn main() {
// Enable debug logging
RUST_LOG=matchy=debug cargo run
}

Verify file changes:

# Check file modification time
stat threats.mxy

# Force update
touch threats.mxy

Callbacks Not Firing

Ensure callback is set before database changes:

#![allow(unused)]
fn main() {
// ❌ Wrong: callback set after database loaded
let db = Database::from("threats.mxy").watch().open()?;
// Database changes here won't trigger callback yet

// ✅ Correct: callback set during open
let db = Database::from("threats.mxy")
    .watch()
    .on_reload(|e| println!("Reloaded!"))
    .open()?;
}

Performance Impact

If auto-reload overhead is too high:

#![allow(unused)]
fn main() {
// Measure overhead
let start = Instant::now();
for i in 0..1_000_000 {
    db.lookup("192.168.1.1")?;
}
println!("Time: {:?}", start.elapsed());
}

Compare this result with a static database under the same CPU affinity, cache state, query mix, and reload frequency. Investigate snapshot refreshes, NUMA placement, and excessive update frequency when the measured difference matters.

Next Steps

• Performance Considerations - Detailed performance analysis
• Query Result Caching - Combine with caching for maximum throughput
• Examples - Complete working examples