Concurrency, coroutines and Asio

Current contract

choose() and evaluate() are synchronous. A call owns the CPU thread until the backend returns. Both Laya backends call blocking model inference; wrapping it in a coroutine does not make its computation asynchronous.

Capability	Current status
Concurrent calls through a bound handle	Supported only when the supplied backend is thread-safe
ONNX Laya tokenizer access	Serialized by an internal tokenizer mutex
ONNX Laya inference	Shared ORT session; exclusive per-call tensor buffers can be reused after completion
Native laya.cpp inference	Calls on one backend are internally serialized to protect mutable graph state
Diagnostics recording and snapshots	Internally synchronized
choose_async(std::string)	std::async(std::launch::async) returning std::future
Bounded workers and microbatching	Optional jevt::batching_backend wrapper
Queued future API	batching_backend::submit() owns request text and returns std::future
Coroutine / co_await API	Optional Boost.Asio completion-token adapter with use_awaitable
Boost.Asio / standalone Asio adapter	Boost 1.74+ supported; standalone Asio not provided
Queue admission	Configurable capacity; overload and shutdown return structured errors
Request deadlines and cancellation	Queue preflight and supported transports honor deadline/stop token; arbitrary running local inference cannot be interrupted

Input string views only need to remain alive for the synchronous call. choose_async() instead takes an owned string and copies the bound handle, keeping its backend alive. A custom backend must not retain request views after returning. Synchronizing the diagnostics registry does not make an application callback thread-safe.

The existing async convenience API

auto pending = routing.choose_async(std::string{message});
// Do independent work on this non-event-loop thread.
auto result = pending.get(); // Blocks until inference is finished.

This API has no admission control and may create a thread per call. Destroying the last future produced by std::async may also wait for completion. Do not discard these futures expecting fire-and-forget behavior, and do not call get() on an I/O event-loop thread. Prefer the synchronous API behind your service’s controlled executor for sustained load.

Bounded workers and microbatching

Include <jevt/batching.hpp> and wrap a shared backend in jevt::batching_backend to use a bounded queue and worker pool. Configure worker_count, queue_capacity, max_batch_size and max_delay through jevt::batching_options. Optional length_bucket_width groups requests by total text bytes to reduce length variation within a batch; results retain caller order. With multiple workers, the wrapped backend must be thread-safe.

The native laya.cpp backend is safe to share but serializes execution on one instance. Increasing worker_count against that instance does not add simultaneous GPU runs. Start with one worker and tune batch size and delay. Separate native instances duplicate resident model/graph state; measure memory and contention before using several on the same device.

submit(request) copies all request text, including option labels, before returning a future. predict() and predict_batch() remain synchronous and wait for their queued results. A full queue returns error_code::overloaded; after shutdown starts, new requests receive error_code::shutting_down. Batch admission is atomic: an oversized batch or insufficient queue capacity rejects the whole batch.

queue_capacity bounds waiting requests. Up to worker_count * max_batch_size additional requests can execute. It is a request-count limit, so enforce input byte limits in the application too. max_delay controls batch formation, not an end-to-end request deadline. stats() reports queue depth, in-flight requests, admissions, rejections, completions and executed batches.

shutdown() stops admission, drains accepted work and joins workers. It waits for the wrapped backend to return and does not cancel model inference. Do not destroy the wrapper from one of its backend callbacks or while callers still access it. Recursive synchronous calls and shutdown from its own workers are rejected. The optional Boost.Asio adapter exposes the same worker pool through completion tokens without a future-waiter thread.

Integrating an Asio-based service

Enable JEVT_ENABLE_ASIO=ON, link jevt::asio and use jevt::asio_backend. Its async_predict() accepts a callback or boost::asio::use_awaitable and posts the result to the handler’s associated executor. Input is copied before returning, including when a coroutine token defers initiation. See the Asio guide for compilable examples and lifetime rules.

For service integration:

1. Validate request size and deadline on the I/O executor.
2. Use a bounded inference queue, such as batching_backend, and handle overload explicitly.
3. Keep blocking inference off the I/O executor; the adapter runs it on bounded workers and owns the request data.
4. Keep the associated executor and application captures alive until completion.
5. Pass request deadline and stop token. Queue preflight rejects expired work; an active local model call may still run to completion. The supplied remote Curl transport also observes cancellation and deadlines during I/O.
6. On shutdown, stop admissions and drain/join inference workers before destroying resources used by their completion handlers.

Budget both the number of workers and ORT intra-op threads. Eight inference workers each creating their own large ORT pool can worsen p95 through CPU oversubscription. Measure queue delay separately from execution time.

Native graph reuse also depends on batch/sequence/option shape. Length buckets can reduce shape variation but are byte-based hints, not fixed tensor-shape guarantees. Warmup and steady-state measurements should cover the actual shapes your service sends. Native ggml replay is owned by the serialized backend; it does not change the synchronous lifetime contract.

The portable cancellation mechanism is the request’s std::stop_token. Boost cancellation slots and standalone Asio are not part of this adapter’s contract. The coroutine API yields result<inference_response>; an error is not a typed negative answer or abstention.

Lifetime and global state

Prefer explicit jevt::context in services, plugins and tests. jevt::init() replaces the process-wide default used for subsequent global binds; existing bindings retain their shared backend. Keep the returned app alive while using the global binding convenience functions. Keep any executor and callback captures alive until their scheduled work completes.