Pro
EvalCore makes LLM regression testing feel CI-friendly by replaying recorded model behavior offline without API keys or flaky network calls.
Reviews of EvalCore
4.3
Con
Its sweet spot is narrow: if you don't already fear prompt regressions, it may feel like a smoke alarm for a toaster.
Introduction
EvalCore is a practical test runner for teams building LLM apps and agents. I like that it records live model behavior, then replays it offline in CI so tests are less flaky and do not need API keys. Developers can define YAML evals, compare models, and block regressions before prompt, model, or dependency changes ship.
What is the most typical application scenario of EvalCore?
An LLM app developer can use EvalCore in pull request CI to replay saved model behavior offline and catch prompt or model regressions without flaky API calls.
Features of EvalCore
Define eval suites and scoring rules in YAML
Record live model calls and replay them offline
Run repeatable CI checks with local cassette storage
Test OpenAI-compatible, REST, shell, or trace targets
Compare baselines, trials, and model versions
Use cases for EvalCore
Add LLM regression tests to every pull request
Run CI evals without live API keys or network calls
Check prompt or model changes before release
Snapshot-test agent behavior against saved baselines
Pricing info of EvalCore
FreePublic pricing information collected from the official website.
evalcore.cc Pricing
Open source
$0
- Apache-2.0 open source
- in CI offline replay
- any language
- One small binary
- No telemetry
- No signup
- No server
Last checked Jul 24, 2026. Visit the official website for the latest terms.