Streaming, structured output & tools

LLMBoost supports the rich parts of the OpenAI API, not just plain completions.

Streaming

Stream tokens as they're generated for responsive UIs.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
stream = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.2",
    messages=[{"role": "user", "content": "Explain KV cache in one sentence."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Structured output (JSON)

Force valid JSON — guided decoding constrains the model to your schema, so you get parseable output every time.

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.2",
    messages=[{"role": "user", "content": "Give the capital of France."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "capital",
            "schema": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    },
)

{"type": "json_object"} works too when you just need "valid JSON".

Tool / function calling

Pass tools and let the model decide when to call them.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="<a tool-capable instruct model>",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
)

note

Tool-call emission needs a tool-trained instruct model and the matching tool-call parser enabled on the server. LLMBoost forwards the tool schema either way — the model decides whether to call.

Vision (multimodal)

Image-to-text models are supported — send image content parts per the OpenAI multimodal message format. Serve a vision model the same way:

lbh serve <vision-model>