Skip to main content

​
vLLM Configuration

OpenAI Compatible Configuration Interface The OpenAI API Compatible configuration screen in Novaplan AI where you’ll enter your vLLM endpoint URL, API Key, and Model Name Novaplan AI allows you to integrate with vLLM (High-throughput and memory-efficient inference engine) using its OpenAI-compatible API endpoint. vLLM is designed for fast LLM inference and serving, making it ideal for self-hosted deployments.

​
What is vLLM?

vLLM is an open-source library for fast LLM inference and serving. It provides:
  • High throughput serving with PagedAttention
  • Continuous batching of incoming requests
  • Optimized CUDA kernels for faster inference
  • OpenAI-compatible API server
  • Support for various popular open-source models

​
Prerequisites

Before configuring vLLM in Novaplan AI, ensure you have:
  1. A running vLLM server instance
  2. An endpoint URL that Novaplan has confirmed is accessible from your managed workspace
  3. (Optional) API key if you’ve configured authentication on your vLLM server
  4. The model name/path used when starting your vLLM server

​
Starting a vLLM Server

If you haven’t started a vLLM server yet, here’s a quick example:
From the vLLM server itself, this example is available at http://localhost:8000/v1/. A managed Novaplan AI workspace needs a separately reachable network address; confirm it with Novaplan.

​
Required Fields

​
Endpoint URL *

The Endpoint URL is the base API endpoint of your vLLM server. Format: http://your-server:port/v1/ Examples:
  • On the vLLM server itself: http://localhost:8000/v1/ (for local checks only)
  • Remote server: http://192.168.1.100:8000/v1/
  • Domain-based: https://vllm.yourdomain.com/v1/
Important:
  • The endpoint URL must include the /v1/ suffix
  • Use https:// for production deployments with SSL/TLS
  • Ensure the server is accessible from where Novaplan AI is running
  • Check firewall rules if connecting to a remote vLLM server

​
API Key *

The API Key field is used to authenticate requests to your vLLM server. Configuration options:
  • If your vLLM server was started with --api-key, enter that key here
  • If your vLLM server was started without authentication, you can enter any placeholder value (e.g., no-key or dummy)
Security Note: Use a strong, unique API key for the OpenAI-compatible endpoints, and restrict network access to the server. The vLLM CLI reference warns that --api-key does not authenticate every endpoint on the server, including /invocations; it is not a complete server access control. Coordinate the network path and any additional protection with Novaplan and your model-server administrator.

​
Model Name *

The Model Name must match the identifier exposed by your vLLM server’s API. This is usually the model used at startup, but a server configured with --served-model-name can expose a different name. Examples:
  • Qwen/Qwen3-8B
Finding your model name: You can query your vLLM server to list available models:
Important: Use an ID returned by /v1/models; do not guess from the startup command alone.

​
Optional Features

​
Multimodal

Enable this checkbox if your vLLM server is running a model that supports multimodal input (text + images). When to enable:
  • You’re using a vision-language model (e.g., LLaVA, Qwen-VL)
  • The model was specifically trained for multimodal understanding
  • You need to process documents with images or visual content
Model choice: Use a vision-language model supported by your installed vLLM version. The Qwen/Qwen3-8B server example above is a text-generation model; do not enable Multimodal for that example solely because it belongs to the Qwen family. Consult the selected model’s card and the vLLM supported-model list. Note: Standard text-only models do not support multimodal capabilities. Verify your model’s documentation before enabling this feature.

​
Reasoning

Enable this checkbox if your model has enhanced reasoning capabilities. When to enable:
  • You’re using a reasoning-focused model (e.g., DeepSeek-R1)
  • The model is designed for complex problem-solving tasks
  • Your use case involves mathematical, logical, or multi-step reasoning
Note: Reasoning models typically take longer to generate responses as they perform additional internal reasoning steps.

​
Configuration Steps

As shown in the image above:
  1. Select “OpenAI API Compatible” as your Provider Type from the dropdown
  2. Enter the vLLM Endpoint URL that Novaplan has confirmed is reachable from your workspace (for example, https://vllm.example.com/v1/)
  3. Enter your API Key (or a placeholder if authentication is disabled)
  4. Specify the exact Model Name used when starting your vLLM server
  5. (Optional) Check “Multimodal” if using a vision-language model
  6. (Optional) Check “Reasoning” if using a reasoning-focused model
  7. Click “Add Model” to complete the setup
All fields marked with an asterisk (*) are required to successfully configure the vLLM integration. You must complete these fields to proceed with the setup.

​
Supported Models

vLLM supports a wide range of open-source models. For the most up-to-date list of supported models, check the vLLM documentation.

​
Performance Considerations

Optimizing your vLLM deployment:
  • GPU Memory: Ensure adequate GPU memory for your model size
  • Batch Size: vLLM automatically manages batching for optimal throughput
  • Tensor Parallelism: For large models, use multiple GPUs with --tensor-parallel-size
  • Quantization: Use quantized models (GPTQ, AWQ) to reduce memory usage
  • Context Length: Adjust --max-model-len based on your use case
Example with optimizations:

​
Troubleshooting

Connection Issues:
  • Verify the endpoint URL is correct and includes /v1/
  • Check that the vLLM server is running: curl http://localhost:8000/health
  • Ensure network connectivity between Novaplan AI and vLLM server
  • Check firewall rules and port accessibility
  • For remote servers, ensure proper DNS resolution
Authentication Errors:
  • Verify the API key matches what was set with --api-key when starting vLLM
  • If no authentication was configured, any placeholder value should work
  • Check vLLM server logs for authentication failures
Model Not Found:
  • Confirm the model name exactly matches the one used to start the vLLM server
  • Query available models: curl http://localhost:8000/v1/models
  • Restart vLLM server if the model was changed
Performance Issues:
  • Monitor GPU memory usage and utilization
  • Check vLLM server logs for warnings or errors
  • Consider using a smaller model or quantization
  • Adjust --max-model-len if seeing out-of-memory errors
  • Use tensor parallelism for large models
Server Not Starting:
  • Verify CUDA/GPU drivers are properly installed
  • Check you have sufficient GPU memory for the model
  • Review vLLM server logs for detailed error messages
  • Ensure the model is compatible with your vLLM version
For additional support, refer to the vLLM documentation or contact Novaplan AI support.