Top 5 Essential Metrics for Evaluating LLMs
No two products need exactly the same dashboard. A code assistant, a customer chatbot, and an internal RAG search all care about different failure modes. Still, there are a few questions that keep showing up in production: Did the model answer the right thing? Were the claims sound? Did it stay inside the documents and the rules? Was it safe? Did it respect the format you asked for?
Below are five metric areas that cover most of that ground. On the Aegis metrics page we group implementations into categories such as general performance, RAG, safety, security, and structural integrity. The names below match the metrics that exist in the product today.
1. Answer relevancy (did it address the actual request?)
Also called answer relevance or task fit in various toolkits. You are asking whether the completion is on-topic for this prompt, not whether it is eloquent in isolation.
Why it matters. The model can look polished while ignoring a constraint in the question (“under 100 words,” “only use the provided table,” “refuse if the user asks for medical diagnosis”). Relevance-type metrics target that gap.
Watch for. Over-weighting length or confidence; a long off-topic answer should not score well because it “sounds authoritative.”
2. Factfulness (are claims supported?)
Sometimes tied to truthfulness, factual consistency, or overlap with trusted references. Here the target is whether stated facts line up with reality or with an agreed evidence set (product docs, clinician-approved copy, policy text).
Why it matters. Generative models can be fluent and wrong. If your product is exposed to customers or regulators, factual drift is often a higher risk than middling writing quality.
Watch for. Passing only when a single gold answer exists; open-ended tasks still need a plausible way to check claims (retrieved passages, tool output, or human spot checks on a sample).
3. Context faithfulness & retrieval fit (for RAG and grounded answers)
For systems that retrieve documents or chunks first, you usually want at least two complementary views:
- Faithfulness / groundedness: Does the answer stick to what the context actually says, without inventing unsupported details?
- Context quality: Was the right material retrieved? In Aegis, related RAG metrics include Context Relevancy, Context Recall, Context Sufficiency, Context Faithfulness, Context Waste, and Context Ranking Precision.
If you are not using retrieval, you can lean more on factfulness and tool traces; if you are using RAG, skipping this layer leaves a large hole.
Why it matters. Many “hallucinations” in production are really context failures: wrong chunk, missing chunk, or answer that drifts off the cited text.
4. Safety (harm, toxicity, bias, and role fit)
In Aegis, Safety is its own category. It is separate from Security (prompt attacks, PII/PHI leakage, role hijacking, and similar). The safety metrics currently available are:
- Misuse — whether the model is used for something outside the intended domain of a specialised chatbot (you supply the domain to check against).
- Manipulation — whether the output uses misleading, manipulative, or overly persuasive tactics (including misinformation-style pressure in how the answer is framed).
- Role Violation — whether the assistant stays within the role implied by the prompt or an explicit expected role you provide.
- Toxicity — toxic language in the output (with optional control over whether quoted citations count toward the score).
- Harmfulness — how harmful the content is, weighed by intensity and exposure.
- Bias — stereotyping or unfair treatment along dimensions such as gender, race, religion, profession, and related characteristics.
Why it matters. Even a factually correct answer can be toxic, biased, off-role, or harmful in ways that do not show up in a relevance or RAG score. These metrics are the usual line between an internal experiment and something you put in front of users.
Watch for. Each safety metric has its own 0–100 score and threshold in Aegis. A strong score on one (for example Toxicity) does not excuse a weak score on another (for example Bias). Review explanations per metric to see which construct actually failed.
5. Structural integrity & instruction following
Here you verify machine-consumable shape: valid JSON/XML, schema fields present, enumerated values in range, required sections in a template, API-safe strings, etc. It overlaps with prompts that forbid certain behaviours (“never output Markdown,” “always cite source IDs”), which some platforms label under alignment or format metrics.
Why it matters. Downstream automation breaks on malformed output far more often than humans notice sloppy prose. These checks are cheap to automate and catch a different class of bugs than subjective quality scores.
Watch for. Valid JSON that is semantically wrong still needs factual or relevance scoring; structure and meaning are complements, not substitutes.
How to use the five without drowning in work
Start with scenarios that burned you before, or that your risk review flagged, and map each scenario to one relevance issue, one grounding/fact issue, one safety/format issue as relevant. Expand when regressions repeat or when you add retrieval, tools, or new locales.
If you want to browse how these map to concrete implementations and categories, see the metrics overview and the Aegis documentation for API specifics.