Skip to main content

Troubleshooting

Quick fixes for the issues people hit most. Still stuck? Grab the server's console output and contact us.

GPU out of memory at startup​

The model and KV-cache allocations don't fit in GPU memory. These techniques help, in order of preference:

llmboost serve <model> --max-model-len 8192        # use a smaller context window
llmboost serve <model> --gpu-memory-utilization 0.85
llmboost serve <model> --tensor-parallel-size 2    # spread across GPUs

A large model (e.g. deepseek-ai/DeepSeek-V3.2) needs multiple GPUs — see Multi-GPU.

GPU out of memory during runtime​

Startup succeeds, but the server aborts mid-run once traffic ramps up — KV-cache growth under concurrent requests exhausts GPU memory. On AMD you'll see a runtime abort like:

Callback: Queue 0x7f5f68200000 Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. Code: 0x1008 Available Free mem : 0 MB.

Lower the client-side concurrency (fewer in-flight requests), or give the server more headroom and a smaller KV-cache/weight footprint:

llmboost serve <model> \
  --disable-auto-config
  --gpu-memory-utilization 0.9 \
  --max_num_seqs 512

"Model not found" / download fails​

  • Use the correct Hugging Face model ID, e.g. deepseek-ai/DeepSeek-V3.2 — or a local path you pre-downloaded, via lbh serve <Repo/Model> -m /path/to/model.
  • The model is gated? Accept its license on Hugging Face, then authenticate: 
    hf auth login      # or: export HF_TOKEN=...
  • Behind a proxy? Ensure the host can reach huggingface.co.

License activation fails​

  • First activation needs outbound network — keep the host online once.
  • A free trial is provisioned automatically; if it's denied, your image may be past end-of-life — pull the latest, or contact us for an enterprise seat.

Port already in use​

The default port is 8000; serve on another port:

llmboost serve <model> --port 8001

Server is up but requests hang or 5xx​

  • Give the model time to finish loading — the first request after start can be slow (weights + warmup). Poll GET /health until 200.
  • Check the server's console output for the underlying error.
  • Reproduce with the built-in smoke test: lbh test <model>.

Chat requests fail with "no chat template"​

The model has no built-in chat template — usually a base model rather than an instruct one. Two options:

  1. Use an instruct model (e.g. …-Instruct) for chat, which ships a template. Base models still work fine on /v1/completions.

  2. Supply a chat template to use a base/custom model with the chat endpoints. Save a Jinja2 chat template to a file and pass it on serve:

    llmboost serve <model> --chat-template ./chat_template.jinja

    Templates for popular model families are widely available (check the model's card or community template collections); the format is the standard Jinja2 chat template the chat endpoints expect.

Disable Auto Config​

LLMBoost may fail to start if user-provided arguments conflict with its automatic configuration. In such cases, either remove the conflicting arguments or disable automatic configuration using --disable-auto-config flag.

#lbh
lbh serve ... -- --disable-auto-config

#llmboost
llmboost serve ... --disable-auto-config

Docker can't see the GPU​

First, verify the GPU is healthy on the host, then check the container's GPU access:

rocm-smi

Ensure ROCm 6.3+ and that the container has /dev/kfd + /dev/dri access.