Production readiness
JevT++ is pre-1.0 software. Passing unit tests is not evidence that a model meets your business accuracy, availability or latency requirements. Pin a release and model revision; validate your own deployment configuration.
Required validation before deploying
Section titled “Required validation before deploying”- Exercise the real application inputs: Unicode, long context, malformed backend output, empty state and all abstention/error paths.
- Run the Laya contract/parity tests with the exact shipped runtime and model. Keep a labeled holdout set covering every business category and escalation boundary. Assess calibration and abstention as well as accuracy.
- Load-test representative lengths and field counts at expected concurrency. Define a queue limit, timeout policy, memory ceiling and rejection behavior.
- Verify shutdown with in-flight inference and active diagnostic connections. The library does not cancel model inference on an application deadline.
- Test the installed CMake package, not just an in-tree executable. Deploy the matching backend runtime libraries and model/tokenizer artifacts explicitly.
- Run sanitizer and OS/toolchain checks. Optional backends need their own checks; a green core-only matrix does not establish backend portability.
Security and operational boundaries
Section titled “Security and operational boundaries”Accept only trusted model bundles. Hash verification establishes artifact identity, not that an arbitrary ONNX graph is safe. Pin artifacts from a source you trust and apply the runtime vendor’s security updates.
Limit input size before tokenization. The adapter’s token budget does not stop
an application from accepting and copying a very large request first. Do not
put raw customer data into diagnostic tags. Configure bounded retention and
histogram bounds suited to actual latencies.
DiagnosticsOptions::recent_sample_every samples only retained recent traces;
all recorded calls still contribute to exact counters and histogram buckets.
Zero disables recent traces, as does recent_capacity = 0.
Input/output token counters sum observed backend reports, not billing charges.
Remote usage shared across independently coalesced evaluations can be observed
more than once; the library does not estimate a per-field cost allocation.
The diagnostics HTTP server is a small, single-client-at-a-time loopback utility, not an authenticated public service. It uses socket idle timeouts and interrupts its active connection during shutdown. Those are not a substitute for authentication, TLS, rate limits or a total request deadline. Prefer exporting snapshots through the service’s existing telemetry system.
Report vulnerabilities through the repository’s security policy.
Integration boundaries still open
Section titled “Integration boundaries still open”| Area | Current boundary |
|---|---|
| Accelerator support | CPU by default; explicit CUDA uses either a compatible ONNX Runtime GPU build or the optional native laya.cpp build |
| Async integration | Bounded workers, futures and optional Boost.Asio completion tokens/coroutines; no forced interruption of running local inference |
| Quality | Eight-ticket example set is a smoke test, not a representative benchmark |
| Confidence | Distribution concentration/threshold policies, not a domain accuracy guarantee |
| Input structure | Core JSON is serialized context; remote parses structured payloads, while application schema validation belongs to the caller |
| Compatibility | Pre-1.0 APIs may change; review release notes and recompile consumers |
The Laya adapter selects CUDA with
laya_options::provider = jevt::laya_provider::cuda. It checks available
providers and appends the CUDA execution provider; unavailable or failed CUDA
initialization is reported rather than silently retrying construction on CPU.
use_tf32 defaults to false. Local C++ CPU/CUDA parity validation observed
a maximum probability difference of 7.15e-7; this is evidence for that tested
configuration, not a guarantee across models, devices or runtime versions.
Optional use_io_binding binds reusable host tensors. ONNX Runtime still
performs device transfers; this is not zero-copy or CUDA-graph execution.
Test the chosen provider, thread settings, batch sizes and representative
inputs before enabling them in a service.
The native laya.cpp adapter uses a safetensors checkpoint and separate build dependencies. Pin the runtime, ggml submodule, model revision and CUDA compiler/math-library versions. Its CPU strict-FP32 default, CUDA optimized-FP32 mode and opt-in BF16 mode are distinct validation targets. BF16 is rejected when the upstream required toolchain profile is unavailable. Initialize native CUDA backends before other CUDA consumers because the pinned runtime sets process-wide CUDA math policy. Validate initialization order when embedding multiple GPU libraries in one service.
One native instance serializes calls and reuses mutable graph state. Exercise changing lengths, batch sizes, option counts and repeated calls when qualifying a build; a single warmed shape does not cover this behavior. Native ggml CUDA Graph replay is not ONNX Runtime CUDA Graph support. The latter remains unavailable for the profiled ONNX graph’s mixed CPU/CUDA placement.
Upstream laya.cpp supports other accelerators and serving features; JevT++’s native wrapper currently exposes in-process CPU/CUDA inference only. Do not treat upstream benchmarks on different hardware as deployment evidence for this wrapper.
jevt::batching_backend supplies bounded queue admission and an optional
worker pool around a backend. Configure request capacity, worker count, batch
size and batch-formation delay explicitly, and handle overload errors.
Shutdown drains accepted work and waits for backend completion; application
deadlines do not interrupt an active inference call. See the
concurrency guide for ownership and shutdown requirements.
The remote backend is opt-in and sends application context to the configured provider. Supply credentials explicitly, review the provider’s data-handling terms, set byte/time/retry limits and avoid sensitive data in questions or criteria unless that transfer is authorized. Retries may repeat a billable request. Local mock and loopback CI does not establish live-provider compatibility, availability or billing behavior. Qualify your chosen endpoint and model separately before deployment.
Release evidence
Section titled “Release evidence”Archive the commit, toolchain/runtime versions, model hashes, test results, quality report and load-test results for each deployment. CI is reproducible evidence of the scenarios it executes, not a blanket production certification. No new release tag is created merely by publishing these docs.