Benchmarks & Comparison

oCAS ships a criterion-based benchmark suite in ocas-tests/benches/, plus cross-language comparison harnesses for SymPy, SageMath, and Symbolica. This chapter explains how to run each and what they measure.

Running oCAS benchmarks

# All benchmarks
cargo bench --workspace

# A specific benchmark
cargo bench --bench poly_gcd
cargo bench --bench poly_factor
cargo bench --bench groebner
cargo bench --bench integrate
cargo bench --bench ntt_poly_mul

# Faster, less precise runs
cargo bench --bench poly_gcd -- --warm-up-time 0.5 --measurement-time 1

SymPy comparison (automated)

The sympy_comparison benchmark drives SymPy through a uv-managed Python subprocess (scripts/compare_sympy.py) and feeds the elapsed nanoseconds into criterion via iter_custom, so oCAS and SymPy appear side-by-side in the same report.

# Requires `uv` on PATH; the Python env is provisioned automatically.
cargo bench --bench sympy_comparison

Supported tasks: parse, diff, expand, factor, gcd, series.

SageMath comparison (local, manual)

SageMath is too heavy to install in CI, so the comparison runs locally via the sage interpreter. The harness mirrors bench_sympy.py's output contract.

# From the ocas-tests directory (requires SageMath installed)
sage scripts/bench_sage.py factor "x^30 - 1" 100

Tasks: parse, diff, expand, factor, gcd. Note that SageMath uses ^ for exponentiation (same as oCAS), so no syntax translation is needed.

JIT & evaluation

The eval_jit benchmark compares the Cranelift JIT against the stack-based interpreter on single- and multi-output workloads (1000 calls each, criterion iter_custom timing):

The figures above are from the 0.15.0 measurements (criterion iter_custom, 1000 calls each) and may drift slightly with later releases.

cargo bench --bench eval_jit --features jit

The multi-output case compiles three expressions (sin(x), cos(x), sin(x)/cos(x)) into one evaluator via compile_multi, sharing the sin(x) subexpression across outputs; call_into writes results into a stack-allocated buffer so each call performs zero heap allocation.

Streaming evaluation

StreamingEvaluator reuses internal buffers across rows, so processing a million-row dataset uses constant memory regardless of stream length:

cargo bench --bench eval_streaming

f32 mixed precision

compile_jit_f32 / compile_vector_evaluator_f32 generate single-precision code. On the same hardware the SIMD evaluator doubles its lane count (16 lanes vs 8 for f64 on AVX-512). Use when f32 accuracy is sufficient.

Symbolica comparison (local, manual)

Symbolica is the primary performance reference for oCAS. Because Symbolica uses an AGPL-style license and ships as a separate Cargo workspace, it is not linked into the oCAS build. Instead, run Symbolica's own example binaries from the source checkout and compare timings manually.

Recommended comparison matrix (Symbolica examples → oCAS benchmarks):

# Run a Symbolica example (from the symbolica source root)
cd ../symbolica
cargo run --release --example polynomial_gcd

# Then run the corresponding oCAS benchmark
cd ../ocas
cargo bench --bench poly_gcd -- --warm-up-time 0.5 --measurement-time 1

Correctness comparison

Beyond performance, oCAS ships a correctness cross-validation framework in ocas-tests/tests/correctness/. It runs 201 automated tests across 19 mathematical modules (as of 0.24.0), comparing oCAS results against SymPy, SageMath, and Symbolica. See the Correctness chapter for details.

Reporting results

criterion writes HTML reports to target/criterion/. Open target/criterion/poly_gcd/index.html in a browser to inspect distributions, regressions, and side-by-side comparisons.