- Adds a lightweight evaluation framework for app-level tracking and benchmarking.
- Changes: 28 files, +1455 additions, -66 deletions.
- Branch: add-evaluations -> main.
- PR: https://github.com/chanzhi82020/FastGPT/pull/1
Applications built on FastGPT need repeatable, comparable benchmarks to measure regressions, track improvements, and validate releases. This initial implementation provides the primitives to define evaluation scenarios, run them against app endpoints or model components, and persist results for later analysis.
I updated the PR description to emphasize that the evaluation system is targeted at FastGPT-built apps and expanded the explanation of the core pieces so reviewers understand the scope and intended use. The new description outlines the feature intent, core components, and how results are captured and aggregated for benchmarking.
- Evaluation definitions
- Define evaluation tasks that reference an app (app id, version, endpoint), test datasets or input cases, expected outputs (when applicable), and run configuration (parallelism, timeouts).
- Support for custom metric plugins so teams can add domain-specific measures.
- Runner / Executor
- Executes evaluation cases against app endpoints or internal model interfaces.
- Captures raw responses, response times, status codes, and any runtime errors.
- Computes per-case metrics (e.g., correctness, latency) immediately after each case run.
- Metrics & Aggregation
- Built-in metrics: accuracy/success rate, latency (p50/p90/p99), throughput, error rate.
- Aggregation produces per-run summaries and per-app historical summaries for trend analysis.
- Allows combining metrics into composite scores for high-level benchmarking.
- Persistence & Logging
- Stores run results, input/output pairs (when needed), timestamps, environment info, and app/version metadata so runs are reproducible and auditable.
- Logs are retained to facilitate debugging and root-cause analysis of regressions.
- Reporting & Comparison
- Produces aggregated reports suitable for CI gating, release notes, or dashboards.
- Supports comparing multiple app versions or deployments side-by-side.
- Extensibility & Integration
- Designed to plug into CI (automated runs on PRs or releases), dashboards, and downstream analysis tools.
- Easy to add new metrics, evaluators, or dataset connectors.
By centering the evaluation system on FastGPT apps, teams can benchmark full application behavior (not only raw model outputs), correlate metrics with deployment configurations, and make informed release decisions.
- Expand built-in metric suite (e.g., F1, BLEU/ROUGE where applicable), add dataset connectors, and provide example evaluation scenarios for sample apps.
- Integrate with CI pipelines and add basic dashboarding for trend visualization.
Related Issue: N/A
Co-authored-by: Archer <545436317@qq.com>
* feat: enhance wallet billing features with new dataset and points options
* chore: removed local state for dataset month and replaced it with form state management
* chore: remove redundant state
* Match SSE for FastGPT (#5168)
* Match SSE for FastGPT
* Modify the judgment
* Optimize logic for SSE transmission
* Refactor imports
* directly use workflowStreamResponse from props
* improve error handling and streamline onStreamData logic
* Refactor API client configuration
* perf: system tool support sse
* update doc
---------
Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com>
* New chatinput (#4995)
* feat: Change border style
* refactor: Improve layout and styling of ChatInput component
* style: Update ChatInput component styling and mobile layout
* fix: update key detection for Enter key in ChatInput component
* feat: 添加 WelcomePage 组件，支持变量输入和欢迎信息展示
* style: Updated the PC voice input interface of the VoiceInput component and optimized the layout and style
* style: Optimize the layout and style of the WelcomePage component, and adjust the responsive design
* feat: Dynamically load the WelcomePage component and optimize the welcome information display logic
* refactor: Adjust the style and behavior of the ChatInput component and delete the WelcomePage component
* style: Modify the minimum height setting of the ChatInput component to simplify responsive design
* style: Optimize the layout and style of PC voice input components, and enhance the processing and drawing logic of waveform data
* style: Adjust ChatInput component's margin and textarea height logic for improved layout and responsiveness; refine PCVoiceInput component's positioning and display elements
* style: Enhance PCVoiceInput component's time display styling with custom font properties
* feat: Change border style
* refactor: Improve layout and styling of ChatInput component
* style: Update ChatInput component styling and mobile layout
* fix: update key detection for Enter key in ChatInput component
* feat: 添加 WelcomePage 组件，支持变量输入和欢迎信息展示
* style: Updated the PC voice input interface of the VoiceInput component and optimized the layout and style
* style: Optimize the layout and style of the WelcomePage component, and adjust the responsive design
* feat: Dynamically load the WelcomePage component and optimize the welcome information display logic
* refactor: Adjust the style and behavior of the ChatInput component and delete the WelcomePage component
* style: Modify the minimum height setting of the ChatInput component to simplify responsive design
* style: Optimize the layout and style of PC voice input components, and enhance the processing and drawing logic of waveform data
* style: Adjust ChatInput component's margin and textarea height logic for improved layout and responsiveness; refine PCVoiceInput component's positioning and display elements
* style: Enhance PCVoiceInput component's time display styling with custom font properties
* style: Add new 'xxl' size to theme spacing for improved layout options
* style: Update close icon fill color to use currentColor for better theming support
* style: Enhance voice input functionality and UI responsiveness; improve waveform sensitivity and amplitude
* style: Conditionally render file preview based on voice input state
* style: 优化移动端音频波形渲染，增强清晰度和敏感度
* style: Update comments to English to enhance code readability and consistency
* style: Adjust the mobile audio waveform update frequency and optimize rendering performance
* style: Optimize the file preview rendering logic in voice input mode to enhance user experience
* style: Optimize the file preview rendering logic in voice input mode to enhance user experience
* style: Adjust the chat input box placeholder color and border color to enhance visual effects
* fix: pg test
* Test secret (#5011)
* add http header auth config (#4982)
* add http header auth config
* optimize code
* add mcp tools header auth
* fix build
* fix ui
* fix
* teamid
* secret value encrypt (#5002)
* perf: secret code
* header auth ui (#5012)
* header auth ui
* fix i18n
* doc
* perf: type
* header secret ui
* reset ui
* perf: check secret invalid
---------
Co-authored-by: heheer <heheer@sealos.io>
* feat: cq and extrat AI memory (#5013)
* fix: login xss
* feat: Users can download the invoice by self (#5015)
* Users can download the invoice by themselves
* Direct file stream implementation for transmission presentation
* i18n
* Chatbox-fix (#5018)
* feat: Change border style
* refactor: Improve layout and styling of ChatInput component
* style: Update ChatInput component styling and mobile layout
* fix: update key detection for Enter key in ChatInput component
* feat: 添加 WelcomePage 组件，支持变量输入和欢迎信息展示
* style: Updated the PC voice input interface of the VoiceInput component and optimized the layout and style
* style: Optimize the layout and style of the WelcomePage component, and adjust the responsive design
* feat: Dynamically load the WelcomePage component and optimize the welcome information display logic
* refactor: Adjust the style and behavior of the ChatInput component and delete the WelcomePage component
* style: Modify the minimum height setting of the ChatInput component to simplify responsive design
* style: Optimize the layout and style of PC voice input components, and enhance the processing and drawing logic of waveform data
* style: Adjust ChatInput component's margin and textarea height logic for improved layout and responsiveness; refine PCVoiceInput component's positioning and display elements
* style: Enhance PCVoiceInput component's time display styling with custom font properties
* feat: Change border style
* refactor: Improve layout and styling of ChatInput component
* style: Update ChatInput component styling and mobile layout
* fix: update key detection for Enter key in ChatInput component
* feat: 添加 WelcomePage 组件，支持变量输入和欢迎信息展示
* style: Updated the PC voice input interface of the VoiceInput component and optimized the layout and style
* style: Optimize the layout and style of the WelcomePage component, and adjust the responsive design
* feat: Dynamically load the WelcomePage component and optimize the welcome information display logic
* refactor: Adjust the style and behavior of the ChatInput component and delete the WelcomePage component
* style: Modify the minimum height setting of the ChatInput component to simplify responsive design
* style: Optimize the layout and style of PC voice input components, and enhance the processing and drawing logic of waveform data
* style: Adjust ChatInput component's margin and textarea height logic for improved layout and responsiveness; refine PCVoiceInput component's positioning and display elements
* style: Enhance PCVoiceInput component's time display styling with custom font properties
* style: Add new 'xxl' size to theme spacing for improved layout options
* style: Update close icon fill color to use currentColor for better theming support
* style: Enhance voice input functionality and UI responsiveness; improve waveform sensitivity and amplitude
* style: Conditionally render file preview based on voice input state
* style: 优化移动端音频波形渲染，增强清晰度和敏感度
* style: Update comments to English to enhance code readability and consistency
* style: Adjust the mobile audio waveform update frequency and optimize rendering performance
* style: Optimize the file preview rendering logic in voice input mode to enhance user experience
* style: Optimize the file preview rendering logic in voice input mode to enhance user experience
* style: Adjust the chat input box placeholder color and border color to enhance visual effects
* New chatinput (#4995)
* feat: Change border style
* refactor: Improve layout and styling of ChatInput component
* style: Update ChatInput component styling and mobile layout
* fix: update key detection for Enter key in ChatInput component
* feat: 添加 WelcomePage 组件，支持变量输入和欢迎信息展示
* style: Updated the PC voice input interface of the VoiceInput component and optimized the layout and style
* style: Optimize the layout and style of the WelcomePage component, and adjust the responsive design
* feat: Dynamically load the WelcomePage component and optimize the welcome information display logic
* refactor: Adjust the style and behavior of the ChatInput component and delete the WelcomePage component
* style: Modify the minimum height setting of the ChatInput component to simplify responsive design
* style: Optimize the layout and style of PC voice input components, and enhance the processing and drawing logic of waveform data
* style: Adjust ChatInput component's margin and textarea height logic for improved layout and responsiveness; refine PCVoiceInput component's positioning and display elements
* style: Enhance PCVoiceInput component's time display styling with custom font properties
* feat: Change border style
* refactor: Improve layout and styling of ChatInput component
* style: Update ChatInput component styling and mobile layout
* fix: update key detection for Enter key in ChatInput component
* feat: 添加 WelcomePage 组件，支持变量输入和欢迎信息展示
* style: Updated the PC voice input interface of the VoiceInput component and optimized the layout and style
* style: Optimize the layout and style of the WelcomePage component, and adjust the responsive design
* feat: Dynamically load the WelcomePage component and optimize the welcome information display logic
* refactor: Adjust the style and behavior of the ChatInput component and delete the WelcomePage component
* style: Modify the minimum height setting of the ChatInput component to simplify responsive design
* style: Optimize the layout and style of PC voice input components, and enhance the processing and drawing logic of waveform data
* style: Adjust ChatInput component's margin and textarea height logic for improved layout and responsiveness; refine PCVoiceInput component's positioning and display elements
* style: Enhance PCVoiceInput component's time display styling with custom font properties
* style: Add new 'xxl' size to theme spacing for improved layout options
* style: Update close icon fill color to use currentColor for better theming support
* style: Enhance voice input functionality and UI responsiveness; improve waveform sensitivity and amplitude
* style: Conditionally render file preview based on voice input state
* style: 优化移动端音频波形渲染，增强清晰度和敏感度
* style: Update comments to English to enhance code readability and consistency
* style: Adjust the mobile audio waveform update frequency and optimize rendering performance
* style: Optimize the file preview rendering logic in voice input mode to enhance user experience
* style: Optimize the file preview rendering logic in voice input mode to enhance user experience
* style: Adjust the chat input box placeholder color and border color to enhance visual effects
* fix: pg test
* Test secret (#5011)
* add http header auth config (#4982)
* add http header auth config
* optimize code
* add mcp tools header auth
* fix build
* fix ui
* fix
* teamid
* secret value encrypt (#5002)
* perf: secret code
* header auth ui (#5012)
* header auth ui
* fix i18n
* doc
* perf: type
* header secret ui
* reset ui
* perf: check secret invalid
---------
Co-authored-by: heheer <heheer@sealos.io>
* feat: cq and extrat AI memory (#5013)
* refactor: Refactored the ChatInput component, optimized the layout of the text area and button group, and improved the user experience
* refactor: Updated ChatInput component, optimized layout and style, and enhanced user experience
* feat: update docs
---------
Co-authored-by: archer <545436317@qq.com>
Co-authored-by: heheer <heheer@sealos.io>
* input ui
* fix: chat input ux
* Return in JSON format to handle checkres (#5019)
* Users can download the invoice by themselves
* Direct file stream implementation for transmission presentation
* Return in JSON format to handle checkres
* fix: invoice
* fix: ui
* doc
* update package
* fix: ts
* fix: login checker
* fix: team plan
* perf: aiproxy ux
---------
Co-authored-by: Theresa <63280168+sd0ric4@users.noreply.github.com>
Co-authored-by: heheer <heheer@sealos.io>
Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com>
chanzhi82020