Benchmarks¶

Performance comparisons: rusket vs mlxtend, scaling to 1 Billion rows, and FPMiner throughput.

rusket includes two mining algorithms (FP-Growth and Eclat), both implemented in Rust. These benchmarks compare rusket against mlxtend (pure Python) on synthetic and real-world datasets.

Benchmark environment: Apple Silicon MacBook Air (M-series, arm64, 8 GB RAM). All timings are single-run wall-clock measurements. Your results may vary depending on hardware, thermal throttling, and background load.

Scale Benchmarks: 1M → 1B Rows 🚀¶

Interactive Chart¶

Three Input Paths¶

rusket supports three ways to ingest data at scale:

1. from_transactions → sparse DataFrame — returns a pandas DataFrame, easy API
2. Direct CSR → Rust — pass scipy.sparse.csr_matrix directly to fpgrowth(), skips pandas entirely
3. FPMiner Streaming — memory-safe accumulator for 100M+ rows that don't fit in RAM

In-Memory Scale (fpgrowth)¶

Out-of-Core Scale (FPMiner Streaming)¶

For real-world retail datasets scaling to 1 Billion rows, FPMiner uses a memory-safe chunks approach (per-chunk sort + k-way merge).

Benchmark: 2,603 retail items, avg 4.4 items/basket, min_support = 0.1%

vs mlxtend¶

The "Auto" Routine Algorithm¶

rusket.mine(method="auto") dynamically selects the algorithm that performs best based on the dataset density (Borgelt 2003 heuristic).

• FP-Growth: Excellent for dense data.
• Eclat: Excellent for sparse data (like retail baskets). Eclat directly uses hardware SIMD array-intersections (popcnt) on the TID-lists, resulting in massive speedups (often 5× to 15× faster on sparse arrays).

Real-World Datasets¶

The Power-User Pipeline¶

import numpy as np
from scipy import sparse as sp
from rusket import FPGrowth

csr = sp.csr_matrix(
    (np.ones(len(txn_ids), dtype=np.int8), (txn_ids, item_ids)),
    shape=(n_transactions, n_items),
)

freq = FPGrowth(csr).mine(min_support=0.001, max_len=3, column_names=item_names)

At 100M rows, the mining step takes 1.2 seconds (not a typo).

🏆 Conquering the 1 Billion Row Challenge¶

Bottleneck 1: Memory Exhaustion during Ingestion¶

The Solution: The FPMiner class provides an out-of-core streaming API. It accepts chunks of (txn_id, item_id) pairs, performs a fast \(O(k \log k)\) sort in Rust, buffers them, and uses a k-way merge to stream directly into the final compressed CSR memory block.

Bottleneck 2: Algorithmic Memory Thrashing¶

The Solution: rusket employs a zero-allocation intersect_count_into() kernel. It pre-allocates a thread-local scratch buffer, intersected in-place, with an early-exit heuristic that aborts the memory scan the moment it proves the remaining bits cannot satisfy min_support.

Bottleneck 3: Sequential Seriality¶

The Solution: rusket merges tree construction into the parallel worker loop. Conditional trees are collected and mined concurrently inside the rayon thread pool.

Running the benchmarks¶

uv run maturin develop --release
uv run python benchmarks/bench_scale.py    # Scale benchmark + Plotly chart
uv run python benchmarks/bench_realworld.py
uv run python benchmarks/bench_vs_mlxtend.py
uv run pytest tests/test_benchmark.py -v -s

Why is rusket faster?¶

Recommender Benchmarks vs LibRecommender¶

Measured with pytest-benchmark (5 rounds, warmed up, GC disabled). MovieLens 100k dataset (943 users, 1,682 items, 100k ratings). Only model.fit() is timed — no startup or data loading overhead.

Benchmark vs PySpark¶

PySpark is the industry standard for large-scale recommendation workloads. While rusket runs strictly single-machine (utilizing Rust Rayon threading), it relies on native memory and avoids the JVM serialization overhead, resulting in massive speedups over PySpark local mode.

Benchmark environment: Apple Silicon M-Series, local[*] Spark master, 1 million ALS interactions (20k Users × 10k Items), 50k FPGrowth transactions.