Add llmtest - pytest for LLM applications

[sazed5055]

Description

Add llmtest to the Testing section under "Testing Frameworks".

llmtest is a Python testing framework for LLM applications that helps developers systematically test for:

• Grounding failures (hallucinations)
• Prompt injection attacks
• Safety violations (PII leaks)
• Model regressions

Why add llmtest?

• Python-native testing framework: Built with Python developers in mind, integrates naturally with existing Python test workflows.
• Solves emerging problem: As Python dominates LLM application development (OpenAI, Anthropic, LangChain, etc.), systematic testing for LLM behaviors is becoming critical.
• Active development: v0.1.1 released March 2026 with CI/CD integration and comprehensive examples.
• Zero-cost exploration: Includes mock provider for testing without API costs.
• Fits the list: Natural addition to Testing Frameworks alongside pytest, hypothesis, and Robot Framework.

Features

• ✅ YAML-based test configuration - Simple, declarative test definitions
• ✅ Multiple test types - Grounding, injection, safety, regression tests
• ✅ Multiple LLM providers - OpenAI, Anthropic, custom HTTP endpoints
• ✅ CI/CD integration - GitHub Actions workflow templates included
• ✅ Reporting - HTML and JSON output formats
• ✅ Mock provider - Test your test suite without API calls

Repository

• GitHub: https://github.com/sazed5055/llmtest
• Documentation: Comprehensive README with quickstart and examples
• Examples: Customer support bot testing, RAG application validation, CI/CD integration
• License: MIT
• Python: 3.9+

Installation

pip install 'git+https://github.com/sazed5055/llmtest.git@v0.1.1#egg=llmtest[all]'

Example Usage

from llmtest import TestRunner
from llmtest.providers.openai_provider import OpenAIProvider

provider = OpenAIProvider(model="gpt-4o-mini")
runner = TestRunner(provider)
results = runner.run_from_file("llmtest.yaml")

[JinyangWang27]

@sazed5055 Thanks for your submission. However, this project does not meet our acceptance criteria:

Repository age: The repository was created 1 day ago. We require a minimum of 3 months (or 6 months for Hidden Gem submissions) to ensure project stability.

GitHub stars: 3 stars (minimum 100 required, or strong justification for Hidden Gem).

Please see our CONTRIBUTING.md for full requirements. You're welcome to resubmit once the project has matured and gained community traction.