ExLlama
A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
Projects
GPU Inference & Serving listings
Text Generation Inference
Large Language Model Text Generation Inference
User submitted (not verified)
Triton Inference Server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
User submitted (not verified)
vLLM
A high-throughput and memory-efficient inference and serving engine for LLMs
User submitted (not verified)