uncloseai.

How We Run Inference

How we run inference

Optional reading, for anyone reproducing our setup or contributing GPU time.

Topology

Two consumer cards, one model. Either hostname gives you turboderp/Qwen3.8-27B-exl3 (read live from /v1/models):

HostnameCardModel
hermes.ai.unturf.comRTX 4090 (Ada)turboderp/Qwen3.8-27B-exl3
qwen.ai.unturf.comRTX 3090 (Ampere)turboderp/Qwen3.8-27B-exl3

Yes, hermes.ai.unturf.com serves Qwen, deliberately. Hostnames are routing labels; always read id from /v1/models.

One port per model identity. Qwen binds 18888, Hermes 18889. A hostname whose model is down gets a 502 instead of silently answering as whatever else is up.

vLLM Setup: Qwen3.8-27B EXL3 on a 24 GiB card

TL;DR: turboderp/Qwen3.8-27B-exl3, a 3.0 bpw exllamav3 trellis checkpoint, served by stock vLLM 0.26 through our vllm-exl3 plugin (--quantization exl3): vLLM keeps the scheduler, paged fp8 KV, prefix caching and the OpenAI API; exllamav3's CUDA kernels run the linears. Same 64k window and 12 sequences as the INT4 build it replaced. Measured 2026-09-23 on the 3090 against that INT4 build with the same flags on an idle box: decode +21% to +41% per stream, prefill +26% (1,653 vs 1,317 prompt tok/s), eight concurrent 7k-prompt tasks in 30.5 s vs 46.0 s, tool-call fidelity 6/6 on both.

# the exllamav3 wheel built on this box and the vllm-exl3 plugin, into the vLLM venv
pip install --no-deps /home/fox/exl3-build/dist/exllamav3-1.5.1-cp312-cp312-linux_x86_64.whl
pip install --no-deps -e /home/fox/exl3/vllm-exl3

vllm serve turboderp/Qwen3.8-27B-exl3 \
    --host 0.0.0.0 --port 18888 \
    --max-model-len 65536 \
    --max-num-seqs 12 \
    --gpu-memory-utilization 0.80 \
    --kv-cache-dtype fp8 \
    --dtype float16 \
    --quantization exl3

Our systemd unit adds (CUDA 13.3 on the 4090, 12.8 on the 3090):

[Service]
User=fox
Environment=CUDA_HOME=/usr/local/cuda-13.3
Environment=PATH=/usr/local/cuda-13.3/bin:/usr/local/cuda-13.3/nvvm/bin:/usr/bin:/bin
Environment=CPATH=/usr/local/cuda-13.3/include
Environment=LIBRARY_PATH=/usr/local/cuda-13.3/lib64
Environment=LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64
Environment=VLLM_EXL3_BACKEND=exllamav3
Environment=VLLM_EXL3_MGEMM=3-32
Environment=VLLM_EXL3_CUBLAS16_ROWS=256
Environment=VLLM_PREFIX_CACHE_RETENTION_INTERVAL=6400
Environment=PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
TimeoutStartSec=3600
Restart=always
RestartSec=5
MemoryMax=16G
OOMScoreAdjust=-500

Why ninja and our CUDA env: vLLM 0.26 JIT-compiles fp8 kernels through FlashInfer at warmup, which needs ninja, nvcc with its nvvm/bin siblings, headers and -lcudart. Missing any one kills our engine at warmup. Single-quote JSON args in a unit file; systemd strips bare double quotes.

vLLM Setup: Hermes-3-Llama-3.1-8B

Our previous production model, kept as a one-line swap. solidrust/Hermes-3-Llama-3.1-8B-AWQ: ~6 GiB weights, Marlin INT4 kernels native on Ampere and Ada. (FP8-Dynamic quants token-loop on a 3090; sm_86 has no fp8 tensor cores.)

vllm serve solidrust/Hermes-3-Llama-3.1-8B-AWQ \
    --host 127.0.0.1 --port 18889 \
    --max-model-len 64000 \
    --max-num-seqs 32 \
    --gpu-memory-utilization 0.72 \
    --kv-cache-dtype float16 \
    --dtype auto \
    --quantization awq_marlin \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

Proxy Setup

An edge Caddy terminates TLS and rate-limits, then forwards over TLS to a Caddy beside each GPU, which proxies to vLLM on loopback. vLLM never binds a public interface.

keepalive off on every hop that reaches vLLM. A pooled connection can be half-open; Go will not retry a POST on it, so our request vanishes and our client hangs forever. A fresh handshake is trivial next to a multi-second generation.

hermes.ai.unturf.com {
    rate_limit {
        zone hermes_public {
            key {remote_host}
            events 3
            window 1s
        }
    }
    reverse_proxy https://<gpu-host> {
        header_up X-Real-IP {http.request.remote.host}
        header_up X-Forwarded-For {http.request.remote.host}
        transport http {
            keepalive off
            dial_timeout 5s
            response_header_timeout 1260s
        }
    }
}

Access logs delete Authorization, Cookie and api-key headers via Caddy's format filter.

Model Discovery

Swagger docs at /docs: hermes.ai.unturf.com/docs, qwen.ai.unturf.com/docs.

curl https://hermes.ai.unturf.com/v1/models
curl https://hermes.ai.unturf.com/version

{
  "object": "list",
  "data": [
    {
      "id": "turboderp/Qwen3.8-27B-exl3",
      "object": "model",
      "owned_by": "vllm",
      "max_model_len": 65536
    }
  ]
}

Pass id as your model name. We omit --served-model-name: an alias hides which checkpoint you have.

Thinking mode: roughly 8x slower. Our clients send "chat_template_kwargs": {"enable_thinking": false} by default.

Rate Limiting & Access

Next Steps

Ready to add text-to-speech to your application?

📖 Read the Text-to-Speech documentation →