N_ NOXGLE.
NOXGLE / PROJECTS

llama.

Containerized llama.cpp server with CUDA and an OpenAI-compatible API.

Local inferenceDocker / CUDA / llama.cpp / Shell

llama ↗

The repository documents a Docker-based llama.cpp deployment with CUDA, model configurations and an OpenAI-compatible API.

Problem

A local model needs a repeatable GPU environment and an interface other tools can use.

Use case

A self-hosted inference endpoint for OpenAI-compatible clients.

Simplified diagram based on the documentation
  1. API client
  2. Docker / llama.cpp
  3. CUDA / GPU + model

GPU and memory requirements depend on the model. README performance figures are author claims, not hub benchmarks.

Source: project README ↗ · Checked 2026-11-16

DockerCUDAllama.cppShellVERIFIED

Architecture

Docker Compose runs a llama.cpp server. Model configurations and operational scripts are maintained in the repository.

Features

  • Containerized llama.cpp server with CUDA and an OpenAI-compatible API.

Requirements

Review dependencies and requirements in the source before installing.

Installation

Follow the current upstream installation documentation. This project was not executed during the audit.

License: MIT · Updated: 2026-11-06 · Verified: 2026-11-16

→ Reading the llama deployment architecture