LLM Gateway

LLM Gateway is a centrally managed proxy that routes LLM requests from your applications and AI agents to external providers. Instead of each application holding its own provider API keys, the CData administrator registers providers once and governs all traffic through a single endpoint. Send requests to https://cloud.cdata.com/gateway/v1 and Connect AI forwards them to the configured provider. Authenticate using the virtual key you issue on the Virtual Keys tab. Your application never holds a raw provider API key.

The LLM Gateway supports the following external providers: Anthropic, Azure OpenAI, Cohere, Custom OpenAI-compatible providers, Google Gemini, and OpenAI.

The LLM Gateway page contains the following tabs:

Projects: Contains a list of projects, each scoped to a routing rule and identified by a user-defined name.
Virtual Keys: Contains a list of virtual keys for authentication. Virtual keys are the credentials a client sends to the LLM Gateway.
Routing Rules: Contains a list of routing rules that determine where a project’s traffic goes: to a single provider, a priority-failover chain, or based on a model allowlist.

Get Started Using LLM Gateway

The following is an overview of setting up the LLM Gateway. Refer to the linked sections for details.

Register a provider. On the LLM Providers page, click + Add Provider and enter your provider name, type, and API key. This is the credential Connect AI uses to forward requests to the LLM provider. Next, select which of the provider’s models to use (such as claude-sonnet-5 for Anthropic or gpt-4o for OpenAI).

Create a routing rule. On the Routing Rules tab of the LLM Gateway page, click + Add Routing Rule and select a routing name, description, and type. Select Single provider to route all traffic to one provider, Priority Failover to route to a primary provider with automatic failover, or Model Allowlist to approve specific model–provider pairs. With Model Allowlist, the client names the model in each request, and the gateway forwards it to the paired provider or refuses it if the model is not on the list. Then configure the targets for your routing rule. If the organization-level default routing rule already suits your needs, you can skip this step and assign it when creating your project.

Create a project. On the Projects tab of the LLM Gateway page, click + Add Project and follow the instructions for creating a project. Add a project name and description, and then assign the routing rule you created in the previous step, or use the default routing rule. Save the project.

Issue a virtual key for the project. On the Virtual Keys tab of your project, or on the Virtual Keys tab of the main LLM Gateway page, click + Issue Virtual Key and follow the instructions for issuing a virtual key.

Send requests through the gateway. Point your LLM client at the Base URL for your LLM and authenticate with the virtual key, as follows:

Choose your Base URL. On the Projects tab of the LLM Gateway page, click View Endpoints to open a dialog showing the Gateway URL: https://cloud.cdata.com/gateway/v1.This URL is already complete. Do not add a suffix.

Configure your client. Set your client’s Base URL to the endpoint you copied. Pass the virtual key as the Authorization: Bearer <virtual-key> header or as the x-api-key header. The gateway selects the provider and model based on the project’s routing rule.

A Connect AI Personal Access Token is not accepted here; only the virtual key binds a request to its project.

Make a test request. The following examples show a minimal request using different SDKs against the same base URL. Replace <virtual-key> with the key you issued.

curl https://cloud.cdata.com/gateway/v1/chat/completions \
  -H "Authorization: Bearer <virtual-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="https://cloud.cdata.com/gateway/v1",
    api_key="<virtual-key>",
)

response = client.chat.completions.create(
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

import anthropic

client = anthropic.Anthropic(
    base_url="https://cloud.cdata.com/gateway/v1",
    api_key="<virtual-key>",
)

message = client.messages.create(
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)
print(message.content)

Getting Started

Data Sources

Explore and Model

Integrations

AI

Security and Governance

Virtual Databases

Operations

Get Started Using LLM Gateway

​Get Started Using LLM Gateway

Get Started Using LLM Gateway