NOXGLE / PROJECTSllama.
Containerized llama.cpp server with CUDA and an OpenAI-compatible API.
Local inferenceDocker / CUDA / llama.cpp / Shell
The repository documents a Docker-based llama.cpp deployment with CUDA, model configurations and an OpenAI-compatible API.
Problem
A local model needs a repeatable GPU environment and an interface other tools can use.
Use case
A self-hosted inference endpoint for OpenAI-compatible clients.
Simplified diagram based on the documentation- API client
- Docker / llama.cpp
- CUDA / GPU + model
GPU and memory requirements depend on the model. README performance figures are author claims, not hub benchmarks.
Source: project README ↗ · Checked 2026-11-16
DockerCUDAllama.cppShellVERIFIED
Architecture
Docker Compose runs a llama.cpp server. Model configurations and operational scripts are maintained in the repository.
Features
- Containerized llama.cpp server with CUDA and an OpenAI-compatible API.
Requirements
Review dependencies and requirements in the source before installing.
Installation
Follow the current upstream installation documentation. This project was not executed during the audit.
License: MIT · Updated: 2026-11-06 · Verified: 2026-11-16
→ Reading the llama deployment architecture