Mirrored from docs/serving.md at v0.6.16, the latest stable release.
neurobrix run loads a model, runs one request and exits. neurobrix serve starts a daemon: a background process
that loads one model once and answers requests over a local socket until it is stopped or stays idle too long.
neurobrix chat is an interactive client of that daemon for language models, and neurobrix stop ends it.
neurobrix serve --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 # load once; returns when the model is loaded
neurobrix chat # multi-turn chat with the loaded model
neurobrix run --prompt "What is a KV cache?" # answered by the daemon (same model)
neurobrix stop # unload and exit
The wire format (framing, every method's fields, error strings, versions, the ready record) is specified in
the daemon wire protocol. This page describes what the commands do, what "warm" means,
and how the conversation and its KV cache behave today.
neurobrix serveneurobrix serve --model <name> [--hardware <profile>] [--timeout <seconds>] [--foreground]
[--compiled | --sequential | --triton | --triton-sequential]
[--ready-fd <fd>] [--ready-file <path>]
| flag | meaning | default |
|---|---|---|
--model |
the model name (as neurobrix list shows it); required |
— |
--hardware |
hardware profile ID | the default profile of this machine, created at first use |
--timeout |
idle timeout in seconds (see Lifecycle) | 1800 |
--foreground |
serve in this process and block the terminal | off: the daemon goes to the background |
--compiled |
PyTorch engine, fused graph | the default where PyTorch is installed |
--sequential |
PyTorch engine, one operation at a time (debugging) | — |
--triton |
NeuroBrix Triton engine | the default where PyTorch is not installed |
--triton-sequential |
Triton engine, one operation at a time (debugging) | — |
--ready-fd |
POSIX: an inherited pipe fd; the ready record (one JSON line) is written there once the daemon accepts, then the fd is closed | — |
--ready-file |
a file the ready record is written to, whole, once the daemon accepts | — |
At most one engine flag may be given; two or more are refused. When none is given, serve prints which engine it chose.
Choosing the PyTorch engine where PyTorch is not installed is refused before the daemon starts.
serve refuses to start (exit code 1) when a daemon of the same instance is already running.
Without --foreground, serve starts the daemon in the background, prints its PID and the path of its log file, waits
until the model is loaded and the socket accepts, then prints one of:
[Serve] Model loaded — <model> (<family>, warm serving)
[Serve] Model loaded — <model> (<family>, cold serving)
[Serve] Failed: <error>
and exits 0 (loaded) or 1 (failed). warm and cold are explained in Warm and cold serving.
On Linux the background daemon is a fork of the serve process; on macOS and Windows it is a new process running
neurobrix serve ... --foreground (with the same model, the resolved hardware profile, engine and timeout).
--ready-fd and --ready-file are for programs that launch the daemon themselves; serve also prints the ready record
on stdout as one line NBX-READY <json>. See Readiness.
neurobrix chatneurobrix chat [--max-tokens N] [--temperature T] [--repetition-penalty P]
| flag | meaning | default |
|---|---|---|
--max-tokens |
maximum tokens per answer | the model's default |
--temperature |
sampling temperature | the model's default |
--repetition-penalty |
repetition penalty (1.0 = none) | the model's default |
chat needs a running daemon (otherwise it prints how to start one and exits 1) whose model is of the llm family
(otherwise it tells you to use neurobrix run and exits 1). It then reads lines from the terminal; each line is one
user message, and the full answer is printed when it is complete (chat answers are not streamed).
| command inside chat | effect |
|---|---|
/new |
start a new conversation: the daemon's message history is cleared |
/context |
number of messages, tokens, percentage of the ceiling and tokens held by the cache (see The /context figure) |
/status |
model, family, engine mode and the device memory in use |
/quit, /exit, /q |
leave chat (the daemon keeps running) |
End-of-file or Ctrl-C also leaves chat. A turn past the session's ceiling is refused by name (see
When the conversation outgrows the cache).
A coding agent that works in a project with tools is not part of the engine: it will be its own project, NBX Code,
which speaks to neurobrix serve through the OpenAI-compatible API (forced tool calls are honoured there).
neurobrix stopneurobrix stop
No flags. When no daemon is running it says so, removes the instance's leftover files (socket, pid, port file) and
exits 0. Otherwise it escalates:
shutdown request on the socket (3 s socket timeout), then up to 5 s for the process to exit;SIGTERM), then up to 3 s;SIGKILL; taskkill /F on Windows).It then removes the instance's files. If the process is still alive it prints the command to kill it by hand and
exits 1.
neurobrix run while a daemon is running--model is omitted, run takes the model of the running daemon.run sends its request to the daemon ([Run] Using warm daemon (PID ...))--output path; text is printed on stdout and not written to --output.run refuses to load a second copy beside it andrun --certified-only is never answered by a daemon: it is refused while one is running.One daemon serves one model. The environment variable NBX_SOCKET_PATH names an instance: set it to the same path
for the serve and for every client command of that daemon, and two daemons with different paths run side by side.
Unset, the default instance is used. See Where the daemon listens.
llm family it runs one one-token generation"warmup") so that the weights are on the device before the first request. It prints<model> | <profile> | <strategy> | Ready in <s>s.shutdown request, on SIGTERM/SIGINT, or on the idle timeout. Before exiting it stops everyserving.drain_timeout_s for their replies to be delivered, closes the connections andshutdown request it also unloads the weights.Idle timeout. The accept loop waits for a new connection for --timeout seconds. When that wait elapses and no
connection is open, the daemon exits; while any connection is open (a chat session, a running request) it does not.
Load failure. The daemon reports failed with the error, and exits; the launching serve prints the error and
exits 1. The log file keeps the daemon's output.
All in the engine home ($NEUROBRIX_HOME, else the home of ~/.neurobrix/paths.json, else ~/.neurobrix), for the
default instance:
| file | content |
|---|---|
daemon.sock |
the Unix socket (macOS, Linux) |
daemon.pid |
the daemon's PID |
daemon.log |
the daemon's stdout and stderr |
daemon.ready |
Windows only: the ready record while the launcher waits for it |
A named instance (NBX_SOCKET_PATH=<path>) uses <path> as its socket and <stem>.pid, <stem>.log,
<stem>.ready (and on Windows <stem>.port) beside it.
When the daemon loads, Prism solves the plan with serve_mode=True. Each plan carries a loading mode:
run; the executor'sCold serving — weights reload per request, setup/context persists.In serve mode Prism prices single_gpu with every weight resident at once (the sum of all weights plus the
largest activation), not one component at a time as for a single run. Prism ranks the
strategies by what they do with the weights and takes the fastest plan that fits within the best rank (lower rank
first):
| strategy | rank | loading mode when serving | daemon |
|---|---|---|---|
single_gpu |
1 | eager | warm |
single_gpu_lifecycle |
1 | eager | warm |
component_placement |
2 | eager | warm |
block_scatter |
2 | eager | warm |
pipeline_parallel |
2 | eager | warm |
weight_sharding |
2 | eager | warm |
op_level_tiling |
3 | eager when the plan's memory is at most 90% of the summed capacity of the plan's CUDA devices, otherwise lazy (so lazy on a device that is not CUDA) | warm or cold |
zero3 |
4 | lazy | cold |
component_placement_lazy |
4 | lazy | cold |
lazy_sequential |
4 | lazy | cold |
layer_streaming |
4 | lazy (always) | cold |
cpu_execution |
6 | lazy | cold |
cpu_streaming |
6 | lazy (always) | cold |
The cold fallback. When no strategy fits under the serve-mode budgets, Prism says so in a warning and solves
again with the budgets of a single run. The plan it finds then is lazy whatever its strategy, so the daemon serves cold.
The result is visible in three places: the serve line (warm serving / cold serving), the ready record's
"serving" field, and the status method's warm_serving and strategy fields.
The plan is solved once, at load. On a machine busy at that moment, Prism re-plans on the memory really free (as it does
for run); it is not re-planned between requests.
| platform | transport |
|---|---|
| macOS, Linux | Unix domain socket, daemon.sock in the engine home (or NBX_SOCKET_PATH) |
| Windows | TCP on 127.0.0.1 only: a fixed port for the default instance, a port chosen by the OS (written to <stem>.port) for a named instance |
Nothing listens on a network interface unless serve --http opens the HTTP API. Each message is a 4-byte big-endian length followed by
one UTF-8 JSON object of at most 64 MiB. Every request carries protocol (today 1), method and params.
| method | kind | what it does |
|---|---|---|
generate |
work | one request of any family, as neurobrix run would make it; "stream": true sends one event per produced token before the reply |
chat |
work | llm family only: adds message to the daemon's conversation and answers it (see Chat and the KV cache) |
complete |
work | families that declare the agent mode: one answer over a message list the caller owns (roles, tool results, optional tools) |
template |
control | the model's chat template text ("" when it has none) |
new_chat |
control | clears the daemon's conversation |
status |
control | model, profile, family, engine mode, warm_serving, strategy, device memory in use, the conversation's context, the live operations, the engine and protocol versions |
cancel |
control | stops one operation by op_id (below) |
shutdown |
control | stops running work, unloads, exits |
chat and complete read temperature, repetition_penalty, top_k, top_p, max_tokens and seed; a parameter
the engine does not read is refused by name before anything runs. The full field lists are in
Methods.
One inference at a time, on one thread. Every engine call — the load, generate, chat, complete,
new_chat, the unload — runs on one long-lived inference thread of the daemon, in arrival order, whichever door
(the socket or the HTTP API) and connection sent it: a work request that arrives while
another runs waits (its state is queued). status, cancel, template and shutdown are answered while a
generation runs, from any other connection.
Operations. Every work request is an operation with an id: the client's (op_id in the request) or one the daemon
assigns (<pid>-<n>). The reply, its errors and its stream events carry it. Its state is queued, running, then one
of completed, cancelled, orphaned (its client went away), stopped (daemon shutdown) or failed. The daemon
remembers the last serving.operations_retained finished operations.
A Python program can use the client the CLI uses, neurobrix.serving.client.DaemonClient (connect, generate,
generate_stream, chat, complete, template, new_chat, status, cancel, shutdown, close).
cancel with params.op_id, sent on a connection other than the one running the work, sets that operation's stop
flag. The autoregressive decode loops of both engines (the text generators, and the vision-language, audio-language,
encoder-decoder and text-to-speech decode loops) read the flag once per token, after the token is recorded, and end as if
their length bound were reached: the reply carries what was produced so far, with status: "cancelled" and steps.
The cancel answer waits until the decode loop has returned and every device of the plan has been synchronised, so
cancelled means no kernel of that generation is still running. Its outcome is one of:
| outcome | meaning |
|---|---|
cancelled |
this request stopped it |
already_cancelled |
an earlier cancel stopped it |
already_finished |
it had ended (state says how: completed, orphaned, stopped, failed) |
unknown_op |
no remembered operation has this id |
cancel_pending |
the stop was not honoured within serving.cancel_ack_timeout_s; not a confirmation |
A flow without a per-token decode loop (image or video denoising, for example) does not read the flag: the work runs
to its end and the cancel then answers already_finished, or cancel_pending if the timeout comes first.
A client that disconnects during a work request (its socket closes, or a stream event cannot be sent) orphans the
operation: the daemon stops it at the next token through the same flag and records it orphaned, never cancelled.
Pressing Ctrl-C in neurobrix chat while an answer is generated closes the client, so the daemon stops that answer
this way.
Shutdown stops every live operation (stopped), waits up to serving.cancel_ack_timeout_s for it, then unloads.
The dispatcher's limits are read from the serving: section of config/system.yml (each is required; a missing or
non-positive value is refused by name):
| key | shipped value | meaning |
|---|---|---|
cancel_ack_timeout_s |
120 | how long a cancel (and a shutdown) waits for the stop |
operations_retained |
256 | finished operations remembered for a late cancel |
listen_backlog |
16 | pending connections the socket queues |
orphan_poll_s |
0.5 | how often a non-streamed request's connection is checked for a vanished client |
drain_timeout_s |
30 | how long an exiting daemon waits for replies to be delivered |
ready_poll_s |
0.5 | Windows: how often the launcher looks for the ready file |
This is the state after 0.6.2 (unreleased): the KV cache belongs to the chat session (scope A of
docs/design/prism_kv.md). Host and disk tiers and sharing between sessions are not built.
The daemon holds one conversation (a list of user and assistant messages), created at the first chat request.
Every client's chat reads and extends that same history; new_chat (or /new) clears it. The chat method takes no
system message. generate does not touch it.
The session keeps its KV cache between turns. For each chat request the daemon:
max_tokens fit the session's ceiling (below);Turn n therefore costs a prefill of its own new tokens, not of the conversation. A prefill that continues a cache
attends over the cached keys read back as whole cache pages, the keys past the true end masked, so the bytes do not
depend on where a turn ended. With a planned prefill chunk (KVCachePlan.prefill_chunk, chosen by Prism for a
serving daemon) a turn served from the session gives the same bytes as the same turns prefilled together with the
same chunk boundaries, on both engines — an equality between two reads cut at the same positions, which a
continued turn and a whole re-read of its prompt do not share (the paragraph below). complete keeps one cache record for the agent path the same way: a fixed system prompt and tools contract
are prefilled once.
What that reuse does not promise is the same bytes as a fresh read: a prompt whose beginning is served from the kept
cache and the same prompt read whole reach the same logits by two different sequences of operations, which do not
round alike. Where two tokens are nearly tied the answer can part at that token — the same request sent twice in a
row to one daemon may give two fluent answers that differ from one word on, at temperature: 0 too (measured
2026-11-11 on a model streamed layer by layer). Within one daemon it is deterministic: the answer is a function of
the request and of what the cache held when it arrived.
Any other request on the same engine (a generate, a warmup) that reuses the cache empties it, and the next turn
then prefills the whole conversation again. A failed turn leaves the session holding nothing.
On the Triton engine under --certified-only, a continued prefill needs the kernel settings of attention with fewer
query rows than keys; the census does not derive them yet, so such a run stops naming the missing setting.
Prism sizes the decoder's KV cache in the plan (KVCachePlan), from the KV state the container declares (layers,
key-value heads, key and value head dimensions, window):
max_cache_len): when serving, as many positions as the memory left by the plan affords, capped by the2 × max_tokens + prompt_margin positions (prompt_margin from the container's defaults,max_tokens). A strategy whose remaining memory cannot holdinitial_cache_len): when serving, max_tokens + prompt_margin positions (at most the ceiling).For comparison, a single run caps the cache at that request's max_tokens + prompt_margin instead of the window.
The session's ceiling is stated when serve starts ("served context: up to N positions a session"): the plan's
session_reach (the positions a serving session holds in the room the cache was sized from, within the declared
window; its max_cache_len when the decoder declares no window) plus, past them, the positions the KV host tier's
share of host memory holds — the oldest pages move there and decoding goes on, exactly. Over HTTP a request past it is
refused before it runs with 400 context_length_exceeded. A turn whose
rendered conversation plus max_tokens (the request's, else the container's) passes it is refused by name, with
the ceiling and the plan fact it comes from; the conversation is left as it was. Nothing is summarised or dropped:
start a new conversation (/new). The reply's summarized field stays in the protocol and is always false.
/context figure/context (the reply's and status's context record) shows:
| field | content |
|---|---|
turns |
the number of messages in the history |
tokens |
the conversation's token count, rendered by the chat template |
max_context |
the session's ceiling (above) |
usage_pct |
tokens / max_context, in percent |
cached_tokens |
the positions the session's KV cache holds |
last_turn |
the last turn's prompt, matched, prefilled, padded and held token counts |
neurobrix serve --http opens, on the same daemon, an HTTP server that speaks the OpenAI API: the same loaded model,
the same queue and the same inference thread as the socket, so an OpenAI client, neurobrix chat and neurobrix run
share the warm weights and wait for each other.
neurobrix serve --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --http # http://127.0.0.1:8765/v1
It listens on serving.http_host:serving.http_port of config/system.yml (127.0.0.1:8765) unless --http-host /
--http-port say otherwise. An address other than loopback is refused without --api-key-file <file>; with a key,
every request sends Authorization: Bearer <key> (401 otherwise). A port already taken is refused at start, naming
the process that holds it. serve prints OpenAI-compatible API: <base_url>; the ready record and status carry
http. A request within the last --timeout seconds keeps the daemon from exiting idle.
Which endpoints a model serves is read from its family's declaration (config/families/<family>.yml), not from its
name. An endpoint the loaded family does not serve answers 409 model_does_not_serve_endpoint, naming why.
| endpoint | served for | what runs |
|---|---|---|
GET /v1/models, GET /v1/models/{id} |
every model | the one loaded model |
POST /v1/chat/completions |
a family declaring the agent mode (llm) |
your messages through the model's chat template (the daemon keeps no history for them); tools too |
a family declaring chat (vlm, audio_llm) or a multimodal family's text mode |
ONE user message: its text and the one image_url (a data: URL) or input_audio part the family needs |
|
POST /v1/completions |
llm |
the raw prompt, no chat template |
POST /v1/images/generations |
a family with a prompt-only mode that writes an image (image, a multimodal family's image mode) |
data[].b64_json (PNG); size = WIDTHxHEIGHT; n images, seed, seed+1, ...; extensions steps, guidance_scale |
POST /v1/audio/speech |
a family with a prompt-only mode that writes audio (tts) |
input spoken with voice as the speaker; the body is WAV |
POST /v1/audio/transcriptions |
a family that answers an audio input with text (stt, audio_llm) |
the uploaded file (multipart); audio_llm needs prompt, its instruction; response_format json or text |
POST /v1/embeddings |
a family declaring an embedding output — none today |
refused by name |
Every request names the loaded model in model; another name answers 404 model_not_found. The response header
X-NeuroBrix-Op-Id carries the request's operation id (a client may choose it by sending the same header): the
socket's cancel stops it, and a client that disconnects stops it at the next token (orphaned).
Chat and completions read max_tokens / max_completion_tokens, temperature, top_p, seed, stream,
stream_options.include_usage, tools, tool_choice (auto, none, required, a named function),
parallel_tool_calls: false (with a forced call only; refused by name otherwise), response_format json_schema,
user, and the engine's top_k and repetition_penalty. A forced call and a json_schema answer are enforced by the
grammar engine, the nbxgrammar package pip install neurobrix installs (501 constrained_decoding_not_installed on a
platform it has no wheel for). A parameter the engine does
not honour is refused by name (400, param names it) — stop, logprobs, n > 1, frequency_penalty /
presence_penalty other than 0, logit_bias, response_format json_object (no schema to enforce). Their neutral
values (n: 1, stop: null, ...) are
accepted. A request that finds serving.http_max_queued others already waiting gets 429 server_busy. An engine
failure — a device out of memory, a named refusal — answers 500 with its message, and the daemon serves on.
Errors are always the OpenAI error object: {"error": {"message", "type", "param", "code"}}.
The server parses the answer. The tool-call format is the one the model's chat template declares, read from the
template's markers: <tool_call>{"name", "arguments"}</tool_call> (JSON), <tool_call><function=…><parameter=…>
(XML; each value is read with its JSON-schema type from your tools), <tool_call>NAME<arg_key>…</arg_key><arg_value>… </arg_value></tool_call> (key and value pairs, GLM's form; each value read with its type, as in XML),
[TOOL_CALLS]NAME[ARGS]{…} (Mistral's form, one per call), and a marker token followed by a JSON list
(<|tool_call|>[...]). A template that declares none refuses tools by name. In the two forms that write a tool's
name bare before a marker token, a forced call (tool_choice: "required" or a named function) refuses, by name, two
tools whose names begin the same way (read and read_file): they cannot be told apart before the marker. Calls come back as
message.tool_calls with finish_reason: "tool_calls"; a block that opened like a call but does not parse is kept in
content, never dropped. When the template declares thinking markers (<think> … </think>), the reasoning is in
reasoning_content and content holds the answer only. Send the history back as OpenAI messages (an assistant's
tool_calls, then role: "tool" messages): the template receives the arguments as objects, as it expects.
With "stream": true the answer is Server-Sent Events: data: <chat.completion.chunk> lines — the first with
delta.role, then delta.content / delta.reasoning_content as the text becomes final, each tool call whole once
its block closes (delta.tool_calls[{index, id, type, function}]), a last chunk with finish_reason, a usage chunk
when stream_options.include_usage is set — then data: [DONE]. Text is held back only while the last token may
still change it or while it could be the start of a marker: a token or a few, never the turn.
curl -s http://127.0.0.1:8765/v1/models
curl -s http://127.0.0.1:8765/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
"messages": [{"role": "user", "content": "What is a KV cache?"}], "max_tokens": 128}'
curl -sN http://127.0.0.1:8765/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0", "stream": true,
"messages": [{"role": "user", "content": "Count to five."}]}'
curl -s http://127.0.0.1:8765/v1/images/generations -H 'Content-Type: application/json' \
-d '{"model": "PixArt-alpha/PixArt-Sigma-XL-2-1024-MS", "prompt": "a lighthouse at dusk"}' \
| python3 -c 'import sys, json, base64; open("out.png", "wb").write(base64.b64decode(json.load(sys.stdin)["data"][0]["b64_json"]))'
curl -s http://127.0.0.1:8765/v1/audio/transcriptions -F model=openai/whisper-large-v3-turbo -F [email protected]
curl -s http://127.0.0.1:8765/v1/audio/speech -H 'Content-Type: application/json' \
-d '{"model": "hexgrad/Kokoro-82M", "input": "Hello from NeuroBrix.", "voice": "af_heart"}' -o hello.wav
With the OpenAI Python client (pip install openai, on the client side; the engine does not need it):
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8765/v1", api_key="unused") # the key file's content when the door has one
model = client.models.list().data[0].id
tools = [{"type": "function", "function": {
"name": "get_weather", "description": "Current weather of a city",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
reply = client.chat.completions.create(model=model, tools=tools,
messages=[{"role": "user", "content": "Weather in Paris?"}])
print(reply.choices[0].finish_reason, reply.choices[0].message.tool_calls)
for chunk in client.chat.completions.create(model=model, stream=True,
messages=[{"role": "user", "content": "Tell me a story."}]):
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
In the serving: section of config/system.yml, read when the door opens (each is then required):
| key | shipped value | meaning |
|---|---|---|
http_host |
127.0.0.1 |
default address |
http_port |
8765 | default port (not 5000 or 7000, which macOS's AirPlay receiver holds) |
http_max_queued |
8 | HTTP requests allowed to wait at once (both doors' waiting requests counted); one more gets 429 |
http_max_body_mb |
64 | the largest request body (an upload, a base64 image) |
The in-run re-plan that neurobrix run gets on a busy machine is not armed for a served request yet; a request that
does not fit the memory left answers 500 with the named refusal, and the daemon stays loaded.
A warm language model, a few turns, then stop:
neurobrix serve --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --timeout 3600
neurobrix chat --max-tokens 256 --temperature 0.7
# You: What is a KV cache?
# You: /context
# You: /new
# You: /quit
neurobrix stop
The Triton engine and an explicit hardware profile:
neurobrix serve --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --hardware v100-32g --triton
Two daemons side by side (one model each):
NBX_SOCKET_PATH=/path/to/llm.sock neurobrix serve --model TinyLlama/TinyLlama-1.1B-Chat-v1.0
NBX_SOCKET_PATH=/path/to/image.sock neurobrix serve --model PixArt-alpha/PixArt-Sigma-XL-2-1024-MS
NBX_SOCKET_PATH=/path/to/llm.sock neurobrix chat
NBX_SOCKET_PATH=/path/to/image.sock neurobrix run --prompt "a lighthouse at dusk" --output lighthouse.png
NBX_SOCKET_PATH=/path/to/llm.sock neurobrix stop
NBX_SOCKET_PATH=/path/to/image.sock neurobrix stop
From Python, a streamed generation cancelled from a second connection:
import threading
from neurobrix.serving.client import DaemonClient
work = DaemonClient(); work.connect()
control = DaemonClient(); control.connect()
def stop_soon():
print(control.cancel("my-op")) # {'outcome': 'cancelled', 'op_id': 'my-op', ...}
for kind, event in work.generate_stream("Tell me a long story.", op_id="my-op", max_tokens=512):
if kind == "token":
print(event.get("text", ""), end="", flush=True)
if event["step"] == 20:
threading.Thread(target=stop_soon).start()
else:
print("\n", event["status"], event.get("steps")) # "cancelled" and the tokens produced
work.close(); control.close()
© 2025–2026 Hocine Benkelaya. NeuroBrix engine released under the Apache License 2.0.
Developed by Hocine Benkelaya · Supported and funded by Neural Network Holding Ltd · Commercial distribution by WizWorks OÜ