How to: Evaluation and Monitoring
* [AlpacaEval](https://github.com/tatsu-lab/alpaca_eval) ![](https://img.shields.io/github/stars/tatsu-lab/alpaca_eval.svg?cacheSeconds=86400) - AlpacaEval is an automatic evaluator for instruction-following language models.
* [ANN-Benchmarks](https://github.com/erikbern/ann-benchmarks) ![](https://img.shields.io/github/stars/erikbern/ann-benchmarks.svg?cacheSeconds=86400) - ANN-Benchmarks is a benchmarking environment for approximate nearest neighbor algorithms search.
* [ARES](https://github.com/stanford-futuredata/ARES) ![](https://img.shields.io/github/stars/stanford-futuredata/ARES.svg?cacheSeconds=86400) - ARES is a framework for automatically evaluating Retrieval-Augmented Generation (RAG) models.
* [BEIR](https://github.com/beir-cellar/beir) ![](https://img.shields.io/github/stars/beir-cellar/beir.svg?cacheSeconds=86400) - BEIR is a heterogeneous benchmark containing diverse IR tasks. It also provides a common and easy framework for evaluation of your NLP-based retrieval models within the benchmark.
* [Code Generation LM Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness) ![](https://img.shields.io/github/stars/bigcode-project/bigcode-evaluation-harness.svg?cacheSeconds=86400) - Code Generation LM Evaluation Harness is a framework for the evaluation of code generation models.
* [COMET](https://github.com/Unbabel/COMET) ![](https://img.shields.io/github/stars/Unbabel/COMET.svg?cacheSeconds=86400) - COMET is an open-source framework for machine learning evaluation.
* [C-Eval](https://github.com/hkust-nlp/ceval) ![](https://img.shields.io/github/stars/hkust-nlp/ceval.svg?cacheSeconds=86400) - C-Eval is a comprehensive Chinese evaluation suite for foundation models.
* [Deepchecks](https://github.com/deepchecks/deepchecks) ![](https://img.shields.io/github/stars/deepchecks/deepchecks.svg?cacheSeconds=86400) - Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling you to test your data and models from research to production thoroughly.
* [DeepEval](https://github.com/confident-ai/deepeval) ![](https://img.shields.io/github/stars/confident-ai/deepeval.svg?cacheSeconds=86400) - DeepEval is a simple-to-use, open-source evaluation framework for LLM applications.
* [DomainBed](https://github.com/facebookresearch/DomainBed) ![](https://img.shields.io/github/stars/facebookresearch/DomainBed.svg?cacheSeconds=86400) - DomainBed is a test suite containing benchmark datasets and algorithms for domain generalization
* [EvalAI](https://github.com/Cloud-CV/EvalAI) ![](https://img.shields.io/github/stars/Cloud-CV/EvalAI.svg?cacheSeconds=86400) - EvalAI is an open-source platform for evaluating and comparing AI algorithms at scale.
* [Evalchemy](https://github.com/mlfoundations/evalchemy) ![](https://img.shields.io/github/stars/mlfoundations/evalchemy.svg?cacheSeconds=86400) - Evalchemy is a unified and easy-to-use toolkit for evaluating post-trained language models.
* [EvalPlus](https://github.com/evalplus/evalplus) ![](https://img.shields.io/github/stars/evalplus/evalplus.svg?cacheSeconds=86400) - EvalPlus is a robust evaluation framework for LLM4Code, featuring expanded HumanEval+ and MBPP+ benchmarks, efficiency assessment (EvalPerf), and a secure, extensible evaluation toolkit.
* [Evals](https://github.com/openai/evals) ![](https://img.shields.io/github/stars/openai/evals.svg?cacheSeconds=86400) - Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.
* [EvalScope](https://github.com/modelscope/evalscope) ![](https://img.shields.io/github/stars/modelscope/evalscope.svg?cacheSeconds=86400) - EvalScope is a streamlined and customizable framework for efficient large model evaluation and performance benchmarking.
* [Evaluate](https://github.com/huggingface/evaluate) ![](https://img.shields.io/github/stars/huggingface/evaluate.svg?cacheSeconds=86400) - Evaluate is a library that makes evaluating and comparing models and reporting their performance easier and more standardized.
* [Evidently](https://github.com/evidentlyai/evidently) ![](https://img.shields.io/github/stars/evidentlyai/evidently.svg?cacheSeconds=86400) - Evidently is an open-source framework to evaluate, test and monitor ML and LLM-powered systems.
* [GAOKAO-Bench](https://github.com/OpenLMLab/GAOKAO-Bench) ![](https://img.shields.io/github/stars/OpenLMLab/GAOKAO-Bench.svg?cacheSeconds=86400) - GAOKAO-Bench is an evaluation framework that uses Chinese National College Entrance Examination (GAOKAO) questions as a dataset to assess large models' language comprehension and logical reasoning abilities.
* [Giskard](https://github.com/Giskard-AI/giskard)![](https://img.shields.io/github/stars/Giskard-AI/giskard.svg?cacheSeconds=86400) - Giskard is an open-source Python library that automatically detects performance, bias & security issues in AI applications.
* [guidellm](https://github.com/vllm-project/guidellm) ![](https://img.shields.io/github/stars/vl