Skip to main content

Run manually

Prefer to manage the container yourself instead of using LLMBoost Hub? Run the LLMBoost image directly with docker run.

Tailor the examples to your setup:

Start the server​

The image ships the llmboost server; pass llmboost serve <model> as the command. On AMD Instinct (ROCm):

# For deepseek-ai/DeepSeek-V3.2 on your GPU — obtain <llmboost-image> with: lbh prep deepseek-ai/DeepSeek-V3.2
docker run -it --rm \
--network host \
--group-add video --ipc host \
--cap-add SYS_PTRACE --security-opt seccomp=unconfined \
--device /dev/kfd --device /dev/dri \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_TOKEN=$HF_TOKEN \
<llmboost-image> \
llmboost serve deepseek-ai/DeepSeek-V3.2 --port 8000
The image

The image above is filled in for the model + GPU you picked above, from the same lookup LLMBoost Hub uses. If the pair shows <llmboost-image>, that combo has no published image yet — get it with lbh prep <model> (LLMBoost Hub pulls the right image); it is also provided with your license. --network host publishes the port directly — use -p 8000:8000 for bridge networking instead.

The server starts and listens on http://localhost:8000.

Add serve flags​

Any vLLM-compatible serve flag works — for example, to expand serving across multiple GPUs:

llmboost serve deepseek-ai/DeepSeek-V3.2 \
--tensor-parallel-size 4 \
--max-model-len 8192

See the full, version-correct list with llmboost serve --help, or the Configuration reference.

Call it​

The server speaks the OpenAI API on /v1 — point your existing client at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
stream = client.chat.completions.create(
  model="deepseek-ai/DeepSeek-V3.2",
  messages=[{"role": "user", "content": "Explain KV cache in one sentence."}],
  stream=True,
)
for chunk in stream:
  print(chunk.choices[0].delta.content or "", end="", flush=True)

See the OpenAI API guide for the full surface.

Licensing

LLMBoost activates a license on startup (a free trial is provisioned automatically). Keep the host online for first activation; serving then runs locally on your GPUs.