TensorZero

Build industrial-grade LLM applications: one API for every LLM, observability, optimization (prompts, models, etc.), evaluations, and A/B testing — all open source. Turn metrics and human feedback into smarter, faster, and cheaper LLMs. Get started in minutes.

TensorZero is an open-source stack designed for building industrial-grade LLM applications. It provides a unified API for accessing any LLM provider, along with tools for observability, optimization, evaluation, and experimentation. The platform enables users to refine models using production metrics and human feedback, ensuring better performance, lower latency, and cost efficiency. Built with Rust for high performance, it supports features like streaming, structured generation, caching, and fallbacks. Users can integrate TensorZero incrementally into their workflows, leveraging its Python client, OpenAI-compatible SDKs, or HTTP API. The platform also offers built-in A/B testing, dynamic prompt optimization, and fine-tuning to enhance model outputs. It stores inferences and feedback in a user-managed database, facilitating debugging and performance monitoring. Fully self-hosted and open-source, TensorZero is production-ready and backed by a team with expertise in machine learning and scalable systems. It caters to both simple applications and complex deployments, with support for GitOps and third-party integrations.