- Adds a lightweight evaluation framework for app-level tracking and benchmarking.
- Changes: 28 files, +1455 additions, -66 deletions.
- Branch: add-evaluations -> main.
- PR: https://github.com/chanzhi82020/FastGPT/pull/1
Applications built on FastGPT need repeatable, comparable benchmarks to measure regressions, track improvements, and validate releases. This initial implementation provides the primitives to define evaluation scenarios, run them against app endpoints or model components, and persist results for later analysis.
I updated the PR description to emphasize that the evaluation system is targeted at FastGPT-built apps and expanded the explanation of the core pieces so reviewers understand the scope and intended use. The new description outlines the feature intent, core components, and how results are captured and aggregated for benchmarking.
- Evaluation definitions
- Define evaluation tasks that reference an app (app id, version, endpoint), test datasets or input cases, expected outputs (when applicable), and run configuration (parallelism, timeouts).
- Support for custom metric plugins so teams can add domain-specific measures.
- Runner / Executor
- Executes evaluation cases against app endpoints or internal model interfaces.
- Captures raw responses, response times, status codes, and any runtime errors.
- Computes per-case metrics (e.g., correctness, latency) immediately after each case run.
- Metrics & Aggregation
- Built-in metrics: accuracy/success rate, latency (p50/p90/p99), throughput, error rate.
- Aggregation produces per-run summaries and per-app historical summaries for trend analysis.
- Allows combining metrics into composite scores for high-level benchmarking.
- Persistence & Logging
- Stores run results, input/output pairs (when needed), timestamps, environment info, and app/version metadata so runs are reproducible and auditable.
- Logs are retained to facilitate debugging and root-cause analysis of regressions.
- Reporting & Comparison
- Produces aggregated reports suitable for CI gating, release notes, or dashboards.
- Supports comparing multiple app versions or deployments side-by-side.
- Extensibility & Integration
- Designed to plug into CI (automated runs on PRs or releases), dashboards, and downstream analysis tools.
- Easy to add new metrics, evaluators, or dataset connectors.
By centering the evaluation system on FastGPT apps, teams can benchmark full application behavior (not only raw model outputs), correlate metrics with deployment configurations, and make informed release decisions.
- Expand built-in metric suite (e.g., F1, BLEU/ROUGE where applicable), add dataset connectors, and provide example evaluation scenarios for sample apps.
- Integrate with CI pipelines and add basic dashboarding for trend visualization.
Related Issue: N/A
Co-authored-by: Archer <545436317@qq.com>
chanzhi82020