Introduction

In this article, we are going to compare Aegis alongside two other evaluation frameworks. First, we are going to have a look at the metrics that are available in Aegis, DeepEval, Opik, and Ragas in order to have a clearer overview over the capabilities of each framework. In the second part of the article, we will look at some of the cornercases tested in our internal benchmarks, where we run several evaluation runs on the various datasets. This allows us to compare pass/fail labels across frameworks.

Aegis Metrics vs DeepEval, Opik, and Ragas

The first column lists each Aegis metric, while the other columns mark whether the other vendor documents a built-in metric with the same or directly comparable intent in their official metric lists (not one-off custom judge prompts).

Sources: DeepEval’s metrics introduction, Opik’s metrics overview, and Ragas’ list of available metrics.

✅ = yes, the metric is available

❌ = no, no named equivalent found there

Where every vendor shows ❌, the gap is usually filled by custom LLM-as-judge definitions (for example DeepEval’s G-Eval, Opik’s G-Eval, Ragas rubrics) rather than a shipped preset. So while it is possible to manually implement the missing metrics from Aegis in the other frameworks, the way to do so is not straightforward and requires a lot of trial and error in order to get the best prompt.

Comparison of the frameworks on various metrics

This note summarises paired evaluation runs from our internal benchmarks. For each suite we compared an Aegis run against the other frameworks on the same dataset and threshold (100 examples per run unless noted):

• Pass agreement — for each case, does Aegis’s pass/fail at its run threshold match the other tool’s pass/fail under the mapping below?

Nothing here claims that one framework dominates another. The point is to make the where and why visible.

How passes were derived

• Aegis: pass when the Aegis score meets or exceeds the threshold defined for that run.
• DeepEval: if the platform exposes an explicit above-threshold flag, that determines pass; otherwise compare the normalised DeepEval score × 100 to the same threshold.
• Opik: pass when Opik’s score × 100 meets or exceeds that threshold.
• Ragas: pass when ragas_pass is true on the paired-export row.
• DeepTeam: pass when the DeepTeam score is 1.0 on its scale.

Pass agreement and case-level splits

The headline numbers from our paired-export analysis span different units. Pass agreement is a percentage (% of paired rows where Aegis’s pass decision matches the other tool under How passes were derived). The cases section is in whole-row counts — how many qualifying rows matched Aegis below threshold while the comparison tool still passes within each paired export. The two tables mirror the same suite rows; cells stay empty when we did not run or report that framework on that benchmark. Ragas figures come from data/blog/posts/original_excel_files/ragas — paired runs with 100 rows where both Aegis and Ragas pass bits are present — and the summarization and answer relevancy rows appear only where we paired against Ragas.

Pass agreement (percent)

Row coverage on these exports. Format alignment vs DeepEval used 98 paired rows on this file. Context faithfulness and context recall vs DeepEval each used 100 paired rows where every case had both an Aegis pass bit and a DeepEval pass bit. The 70% context relevancy figure vs DeepEval used 10 paired rows meeting that filter. On the full 100 row context relevancy sheet, Aegis agreed with the benchmark expected pass 57% of the time. Ragas paired exports (context faithfulness, context recall, answer relevancy, summarization) each use 100 rows where both aegis_pass and ragas_pass are populated.

Cases: Aegis fails, comparison passes (counts)

Values are counts of qualifying rows within the paired export for each comparison (not percentages).

*Context relevancy vs DeepEval: 1 case in 10 paired rows on this export contributes the 1 in DeepEval’s count for that suite. Other suites use 100 cases (or 98 paired rows for format alignment) unless noted above.

Reading these tables. High pass agreement suggests both systems are separating cases similarly at the chosen threshold, even if the underlying judge differs—including Ragas on the suites where we exported paired passes. Low agreement is a signal to spot-check what object is graded, what text each tool judged, and how passes were derived before you treat either tool as ground truth.

When Aegis catches a corner case

In practical terms, this means that after setting a threshold for a metric, Aegis will fail if the score is below the threshold, while the other tool will pass, meaning that Aegis is stricter on this sample and threshold pairing, or that the two systems weight different signals. The following section focuses on examples worth inspecting by hand. Counts for rows where Aegis is below its threshold while the comparison tool still passes are in the cases table; that pattern usually means Aegis is stricter on this sample and threshold pairing, or that the two systems weight different signals — not, by itself, proof of ground truth.

Role violation: Aegis vs DeepEval. In one recurring pattern, the assistant offers clinical sounding guidance, for example, “Try raising the dose and see if the rash clears in 48 hours,” even though the benchmark had framed the model as a nonclinical agent (such as general customer support). Aegis treats that reply as breaking the role boundary, while DeepEval can still record a pass if its rubric rewards a cooperative tone rather than strictly enforcing the stated mandate.

Bias: Aegis vs DeepEval. On one item, the assistant praises a country but adds a sweeping negative stereotype about its population. On another, it leans on a broad psychological generalisation—examples like the paired outputs below. Aegis scores 0 or 40 on items in this vein, while DeepEval still records a pass. Written rationales can even contradict the numeric label (“score is 0.00…”), so treat prose explanations as debugging hints rather than a second source of truth.

Bias: Aegis vs Opik. Stereotypes about a country’s leadership and symmetric put downs of talkative versus quiet people show the same shape. Aegis often lands near 40 on those replies, while Opik’s moderation note reports no policy violation at 1.0. Representative outputs include:

Harmfulness: Aegis vs Opik. Manipulative signup pressure is one recurring pattern: Aegis sits near ~65 on those replies while Opik still passes at 0.8. Bluntly shaming or condescending criticism is another pattern: Aegis can sit near ~42.5 while Opik scores 1.0 and labels the content safe. Representative outputs include:

Toxicity: Aegis vs Opik. Many disagreements hinge on passive aggressive or minimising phrasing written as full replies—backhanded compliments, sarcastic thanks, minimising denial, gaming-style insults. In cases like the outputs below, Aegis often lands in the 40 to 75 band, while Opik assigns 0.8 to 1.0.

Format alignment: Aegis vs DeepEval. On the FOFO paired export (98 rows), there were no cases where Aegis failed while DeepEval still passed, so there is no “Aegis caught it, DeepEval missed it” example from this run. Every format disagreement we saw was the opposite (Aegis pass, DeepEval fail on stricter structure checks).

Context faithfulness: Aegis vs DeepEval. The examples here sit in the slice where Aegis is below threshold but DeepEval still passes (26 of 100 rows on RAGBench train). In one item, the user asks whether a party may solicit or hire the counterparty’s employees or contractors; the model says yes and cites a section of the exhibit, DeepEval passes, and Aegis fails when span-level alignment to the license agreement body is judged too loose. In another, the user asks for the 2007 ratio of shares granted to shares vested from Humana footnotes; the model assembles grant and vesting counts (on the order of 852k granted in 2007) into a ratio story from the context, DeepEval passes, and Aegis fails where the rubric expects closer lockstep to the cited lines.

Context recall: Aegis vs DeepEval and Opik. Only one Garage benchmark row had Aegis fail while both DeepEval and Opik still passed. The user asks how Tyler, The Creator’s Chromakopia influenced contemporary music; the model writes a fluent overview of genre blending and artistic evolution, both comparison tools pass, and Aegis fails because the retrieved passages do not clearly contain the specific claims the answer advances.

Context relevancy: Aegis vs DeepEval. Among the 10 rows on this export that had both pass bits for DeepEval, one had Aegis fail while DeepEval passed. The user asks about COVID-19 and global economic inequality; the model answers with a broad inequality narrative while the retrieved chunk emphasizes long COVID and household finances in the United States. DeepEval still passes on loose topical overlap; Aegis fails because the passages are not a close fit to what the answer leans on.

Where agreement breaks down

Role violation landed at 60% agreement with DeepEval, the lowest pass alignment in this batch. Role rubrics are famously sensitive to how the expected persona is stated, what negation is allowed, and whether multi-turn context is included. If your product policy differs from either judge’s prompt, you should expect exactly this kind of spread.

Toxicity splits in two interesting ways. Against DeepEval, one toxicity suite shows 64% agreement on the latest paired runs we compared; the English subset still shows 73% agreement. Against Opik on the same Aegis cases, agreement on that suite is 65%, while the English subset jumps to 99% agreement — evidence that backend model choice and preprocessing move toxicity judgments more than the framework name on the tin.

Practical takeaways

• Treat pass agreement at a shared threshold as a sanity check, not a single fitness function.
• When agreement is high but case-level scores look unrelated, or agreement is low, inspect what text each tool graded and how passes were derived before calibrating thresholds.
• When prompts, model versions, or thresholds change, re-run paired evaluations and recompute agreement; small drift can move headline percentages by double digits.

If you are designing metrics rather than comparing vendors, the step-by-step build path in How to design an evaluation metric pairs naturally with this kind of cross-tool rehearsal: lock the construct first, then line up external judges as secondary witnesses, not as silent replacements.

Paired-export spreadsheets

Each link returns the workbook as raw .xlsx from data/blog/posts/excel_files (paired slim exports): sheet Results with columns trimmed to inputs, outputs, Aegis scores, and external scores (deepeval_score, opik_score, and/or ragas_score) where exported. Typical browsers open downloads in a new tab or prompt depending on spreadsheet handling (15 files):

• ans_rel_aegis_ragas.xlsx
• bias_results.xlsx
• ctx_faith_aegis_ragas.xlsx
• ctx_faith_rag_galileo_ai_ragbench_train_results.xlsx
• ctx_prec_rag_galileo_ai_ragbench_train_results.xlsx
• ctx_recall_aegis_ragas.xlsx
• ctx_recall_rag_garage_benchmark_results.xlsx
• ctx_rel_rag_garage_benchmark_results.xlsx
• ctx_waste_rag_garage_benchmark_results.xlsx
• harmfulness_results.xlsx
• pii_phi_leakage_results.xlsx
• role_violation_results.xlsx
• summarization_aegis_ragas.xlsx
• sys_data_leak_deepteam.xlsx
• toxicity_results.xlsx