Skip to content

Connect a local model server

Goal

Route the primary Colossus model through a local server that implements the OpenAI-compatible model-catalog and Chat Completions contracts Colossus uses.

Prerequisites

  • A local server running on a known loopback port with the intended model loaded.
  • Working GET /v1/models and POST /v1/chat/completions compatibility.
  • A schema version 2 configuration generated by colossus config init.
  • The exact local model identifier, context and output limits, and supported tool-call and streaming behavior.

Ollama, LM Studio, and vLLM are common examples of servers that can expose compatible APIs. They are not guaranteed first-class Colossus integrations: compatibility depends on the server version, selected model, and enabled features.

Steps

1. Start the server and load the model

Use the server's own startup and model-loading workflow. Confirm it listens on the loopback address and port you intend to grant. The example below uses http://127.0.0.1:11434/v1 and assumes no authentication.

2. Configure the loopback route

Run colossus -w . config effective, edit the reported resolution.configPath, and keep its required storage block and any intended custom settings. Apply this validated overlay, then replace local-model and its limits with the exact selected catalog entry.

schemaVersion: 2
providers:
  profiles:
    local-provider:
      kind: open_ai_compatible
      baseUrl: http://127.0.0.1:11434/v1
      credentialReference: null
      chatCompletionsOutputTokenParameter: max_tokens
models:
  profiles:
    local:
      providerProfile: local-provider
      model: local-model
      contextWindowTokens: 32768
      maxOutputTokens: 4096
      capabilities:
        toolCalls: true
        streaming: true
  roles:
    primary: local
sandbox:
  networkDestinations:
    - http://127.0.0.1:11434

Set toolCalls: true only if both the server and selected model handle function tools and tool history correctly. Set streaming: true only if the endpoint implements the streaming Chat Completions contract; otherwise set either capability to false.

This is a CLI-managed loopback connection: you start the server, and credentialReference: null means no authentication. A desktop or other embedding host may instead manage server lifecycle and inject a host:IDENTIFIER credential. The standard CLI does not resolve host: credentials.

The example uses the backward-compatible max_tokens request field. Select max_completion_tokens when the exact server and model require the modern field, or omit when the server rejects both limit parameters. The configured model maxOutputTokens remains Colossus's canonical output reservation in every mode.

3. Inspect routing and readiness

colossus -w . models route primary
colossus -w . provider models local-provider
colossus -w . provider doctor local-provider
colossus -w . models doctor local

4. Send one bounded model turn

colossus -w . run \
  "Reply with exactly: connected"

Expected result

The catalog exposes the exact local model, diagnostics succeed against loopback, and the bounded run returns connected.

Verification

Stop the server and confirm provider doctor local-provider fails rather than silently using another endpoint. Restart and reload the model, rerun both doctor commands, and then explicitly resubmit the smoke test.

Failure path

  • Connection refused: start the server, verify its bind address and port, then rerun provider doctor local-provider.
  • HTTP 503 while loading: wait for the server to report the model ready, rerun models doctor local, and explicitly resubmit the turn. Colossus reports this as recoverable but does not retry implicitly.
  • HTTP 400 or malformed response: verify the exact model ID and the server's model catalog, Chat Completions token parameter, tool-call, and streaming compatibility. Correct chatCompletionsOutputTokenParameter, disable unsupported capability flags, and rerun models doctor local.
  • The sandbox denies loopback under isolation: grant the exact scheme, address, and port shown in baseUrl, without its /v1 path.

Next step

Review the exact compatible-server contract in Providers and models, apply deployment guidance from Providers and routing, or compare this path with Other OpenAI-compatible endpoints.