Database Concepts

This chapter covers the fundamental concepts of Matchy databases.

What is a Database?

A Matchy database is a binary file containing:

• Entries - IP addresses, CIDR ranges, patterns, or exact strings
• Data - Structured information associated with each entry
• Indexes - Optimized data structures for fast lookups

Databases use the .mxy extension by convention, though any extension works.

Immutability

Databases are read-only once built. You cannot add, remove, or modify entries in an existing database.

To update a database:

1. Create a new builder
2. Add all entries (old + new + modified)
3. Build the new database
4. Atomically replace the old file

This ensures readers always see consistent state and enables safe concurrent access.

Entry Types

Matchy automatically detects four types of entries:

You don’t need to specify the type - Matchy infers it from the format.

Auto-Detection

When you query a database, Matchy automatically:

1. Checks if the query is an IP address → searches IP tree
2. Checks for exact string match → searches hash table
3. Searches patterns → uses Aho-Corasick algorithm

This makes querying simple: db.lookup("anything") works for all types.

Memory Mapping

Databases use memory mapping (mmap) for efficient opening:

Traditional Database          Matchy Database
─────────────────────        ─────────────────
1. Open file                 1. Open file
2. Read into memory          2. Memory map
3. Parse full contents       3. Validate bounded structure
4. Build data structures
   (100-500ms for large DB)

Memory mapping has several benefits:

Efficient opening - Memory mapping avoids whole-file deserialization. Matchy still validates structural envelopes, and observed latency depends on storage, page-cache state, platform, optional sections, and legacy scanning.

Shared memory - The OS shares memory-mapped pages across processes automatically:

• 64 processes with a 100MB database = ~100MB RAM total
• Traditional approach = 64 × 100MB = 6,400MB RAM

Large databases - Work with databases larger than available RAM. The OS pages data in and out as needed.

Binary Format

Databases use a compact binary format based on MaxMind’s MMDB specification:

• IP tree - Binary trie for IP address lookups (MMDB compatible)
• Hash table - For exact string matches (Matchy extension)
• Aho-Corasick automaton - For pattern matching (Matchy extension)
• Data section - Standard MMDB values plus an optional Matchy timestamp type

This means:

• Standard MMDB readers can read IP values that use only standard MMDB types
• Matchy can read standard MMDB v2 files (like GeoIP databases) within its documented decoder resource limits
• Cross-platform compatible (same file works on Linux, macOS, Windows)

Matchy’s compact Timestamp is extended type 128 and is not understood by standard MMDB readers. Use standard string values when interoperability is required.

Building a Database

The general workflow is:

1. Create a builder - Specify match mode (case-sensitive or not)
2. Add entries - Add IP addresses, patterns, strings with associated data
3. Build - Generate optimized binary format
4. Save - Write to file

How to build:

• Using the CLI
• Using the Rust API
• Using the C API

Querying a Database

The query process:

1. Open database - Memory map the file
2. Query - Call lookup with any string
3. Get result - Receive match data or None

How to query:

• Using the CLI
• Using the Rust API
• Using the C API

Query Results

Queries return one of:

• IP match - IP address or CIDR range matched
• Pattern match - One or more patterns matched
• Exact match - Exact string matched
• No match - No entries matched

For pattern matches, Matchy returns all matching patterns and their associated data. This is useful when multiple patterns match (e.g., *.com and example.* both match example.com).

Database Size

Database size depends on:

• Number of entries
• Pattern complexity (more patterns = larger automaton)
• Data size (structured data per entry)

Typical sizes:

• 1,000 entries - ~50-100KB
• 10,000 entries - ~500KB-1MB
• 100,000 entries - ~5-10MB
• 1,000,000 entries - ~50-100MB

Pattern-heavy databases are larger due to the Aho-Corasick automaton.

Thread Safety

Databases are thread-safe for concurrent queries:

• Multiple threads can safely query the same database
• Memory-mapped data is read-only
• No locking required

Builders are NOT thread-safe:

• Don’t share a builder across threads
• Build databases sequentially

Compatibility

Databases are:

• ✅ Platform-independent - Same file on Linux, macOS, Windows
• ✅ Tool-independent - CLI-built databases work with APIs
• ✅ Language-independent - Rust-built databases work with C
• ✅ MMDB-aware - Reads standard MMDB v2 types within documented decoder limits

Next Steps

Now that you understand database concepts, dive into specific topics:

• Entry Types - Deep dive on IP, CIDR, patterns, strings
• Pattern Matching - Glob syntax and matching rules
• Data Types and Values - What data you can store
• Performance Considerations - Optimization strategies