Atla has introduced Selene 1, a cutting-edge LLM Judge designed specifically to evaluate generative AI responses. Selene 1 achieves state-of-the-art performance across 11 commonly used benchmarks for AI evaluation, outperforming models including OpenAI's o-series, Anthropic's Claude 3.5 Sonnet, and DeepSeek's R1 on average. This makes it one of the most advanced tools available for assessing AI-generated content.

A General-Purpose Evaluator with Strong Performance

Selene 1 excels in a variety of evaluation tasks, including:

1. Absolute scoring (e.g., rating responses on a scale of 1-5)
2. Classification (e.g., answering yes/no questions)
3. Pairwise preference judgments (e.g., selecting the better response between two options)

It backs up its evaluations with chain-of-thought reasoning, providing actionable critiques that enhance transparency and usability. Selene 1 can be applied to tasks such as detecting hallucinations in retrieval-augmented generation (RAG) systems, assessing logical reasoning in AI agents, and verifying correctness in domain-specific applications. It supports both reference-based and reference-free evaluation, making it a flexible tool for different use cases.

Fine-Grained Steering and Customization

Selene 1 is designed to be highly customizable, responding well to fine-grained instructions that allow users to adjust evaluation criteria according to specific needs. Atla has also introduced the Alignment Platform, a tool that simplifies the creation and refinement of custom evaluation metrics. Users can describe their task, and the platform assists in generating and testing tailored evaluation prompts, requiring little to no prompt engineering.

Benchmark Performance Highlights

Selene 1 delivers state-of-the-art performance across key benchmarks, including:

1. FLASK Benchmark: Achieves a ~0.71 Pearson correlation with human scores, demonstrating strong alignment with human judgment.
2. MT-Bench: Excels at evaluating complex multi-turn conversations.
3. Auto-J and RewardBench: Outperforms other models in capturing human preferences across domains like chat, reasoning, and safety.

Integration and Accessibility

Selene 1 is designed for seamless integration into existing AI evaluation workflows, offering:

1. API & SDK Access: Provides structured input-output formats for easy adoption.
2. Compatibility with Popular Tools: Works smoothly with frameworks like DeepEval and Langfuse.
3. Alignment Platform: Available to all users, enabling the creation and refinement of custom evaluation metrics.

Atla’s release of Selene 1 offers a robust, customizable, and high-performance solution for developers and researchers focused on improving the reliability of AI-generated content.