Performance controls

Two optional backends expose CUDA: ONNX Runtime and the native laya.cpp/ggml adapter. CPU remains the default. These controls target different costs, and enabling every switch does not necessarily improve latency. Read the measurements first. Options named laya_options in the table apply to the ONNX backend; native execution has its own provider and precision options described below.

Priority	Change	Expected benefit and status
1	CUDA execution provider	laya_options.provider = laya_provider::cuda; errors on unavailable CUDA. TF32 off by default.
2	Session reuse and warmup	One session per backend; warmup(requests, iterations) validates representative inputs through the same path.
3	Focus context and select fields	brain.select<"category", "is_urgent">() preserves typed results; context_token_limit optionally caps state tokens. Serializer chooses relevant object properties. No automatic lossy pruning.
4	Tune threads and concurrency	Intra/inter-op threads, explicit parallel execution and spinning controls; jevt_laya_load measures client concurrency together with ORT threads. No universal auto-tuned default.
5	Bounded execution	Optional batching_backend: fixed workers, bounded waiting rows, explicit overload rejection, graceful drain.
6	Cross-request microbatching	Configurable row count, accumulation delay and byte-length buckets; batching contract.
7	Schema token cache	Per-instance bounded FIFO cache of question/option heads. Shared state is encoded once within a batch, not retained between calls.
8	Reusable buffers and I/O binding	Isolated host buffer leases, capped retention; optional ORT I/O binding with host inputs and output. Still performs device copies; no CUDA Graphs or persistent device buffers. Not enabled by default.
9	Precision experiments	Opt-in FP16/INT8 conversion and quality/parity gates. Conversion success alone does not qualify a candidate for deployment.
10	Diagnostics sampling	recent_sample_every controls detailed traces without sampling counters/histograms. Contention benchmark included; no default p95 speedup claimed.

Native optimized FP32

The native adapter is a separate execution path using original safetensors checkpoints. Set laya_native_options::provider to laya_native_provider::cuda and precision to laya_native_precision::optimized_fp32 to select upstream compensated Tensor Core projections. Its specialized FP32 attention kernel covers sequence lengths up to 128 tokens; longer sequences use cuBLAS attention. This boundary selects a kernel and does not limit the model context to 128 tokens.

The native runtime keeps weights resident and reuses its current compute graphs and allocations. Shape changes can rebuild graphs. One backend serializes calls to protect this state; use a single bounded batching worker as a starting point and tune with realistic inputs. Native ggml CUDA Graph replay is distinct from ONNX Runtime CUDA Graph capture.

Optimized FP32 and BF16 are explicit modes, not automatic promotions after a benchmark. BF16 additionally requires the upstream validated toolchain. Validate matching-precision probabilities and domain quality. Upstream’s published GPU results do not establish this adapter’s speed on your hardware.

Why the ONNX adapter does not enable CUDA Graphs

ORT requires every node to run on CUDA, fixed input/output shapes and device addresses for each graph ID, and serialized calls on the captured session. The profiled Laya export assigns shape operations to CPU, so it does not meet that requirement. Host I/O binding does not fix placement or provide stable device buffers. See the official CUDA Graph constraints.

Persistent device buffers can be evaluated independently of graph capture, but require bounded allocation, ownership through execution, and synchronization before reading or reusing outputs. Capturing each new shape is not an unbounded cache strategy: ORT retains captures for the session lifetime. The current ONNX adapter exposes reusable host buffers, not a persistent-device-buffer API.

What to measure for each change

Record model hash, runtime/provider versions, hardware, input length, field count, concurrency, warmup, sample count and timing boundaries. Report p50, p95 and p99 with errors and queue rejections, not successful calls alone. Include cold start and memory separately. Compare identical inputs and a labeled quality set; probability parity on a single fixture is not enough.

For a service, distinguish queue delay, tokenization, model execution and end-to-end response latency. Increase concurrency until the queue or latency budget fails. Do not infer throughput simply from reciprocal p50.

Safe caching

Result caching can help repetitive workloads, but requires explicit semantics: include the complete relevant state, schema, model revision and policy in the cache key, bound retention, and protect sensitive context. Do not silently cache decisions for state that changes independently, such as account balances or incident status. This library currently provides no result cache.