LLM Evaluation Framework: Build Your First Eval System

What is an LLM evaluation framework?

Much has been discussed in software about the importance of a robust and thorough testing suite to the long term health and scalability of any product. When a team is able to trust the existing unit and integration tests, this speeds up development. It allows for new features to be developed faster, and refactors on previously done work to be done with confidence.

This holds true, even when we start to integrate our systems with large language models (LLMs), or to build entirely new applications based on these new technologies. There is an inherent difference, however, between classical software projects and those that use LLMs. The difference being that LLMs are non-deterministic at its core.

This difference means that we have to relearn and redevelop our body of knowledge as Software Engineers for how to build a reliable and trustworthy testing suite. When we test a deterministic system, unit and integration tests work well. When testing a non-deterministic system, we must use evaluation tests, also called evals.

Evals are a way to give confidence that a non-deterministic AI system will work to a given level of quality and success given a data set supplied. This blog post outlines a basic system that I am using in order to go from zero to a basic level of evals in a system using LLMs primarily based on prompt engineering.

I will first outline the overall steps before diving into a case study, with examples of prompts I might call, and steps I might take in order to build this system.

Five parts of a basic LLM evaluation system

The steps are as follows:

At a high level, come up with a basic understanding of what your system can do by defining it in a matrix. A simple matrix might consist of features, scenarios, and personas.
Using prompt engineering, generate a large amount of comprehensive test scenarios. This can also be a multi step prompt of first generating a series of Given-When-Then triads followed by the actual scenario setup data.
Come up with a list of deterministic pass / fail tests that can be run against all of the outputs. For example, the presence or absence of a series of words. Whether or not a function was called. Whether or not the database was updated. The response length, and the number of sentences returned. The semantic similarity compared to a specified string.
Create a system of versioning for prompts, and generate a report for each evaluation run. Track this over time. Decide on a minimum pass / fail threshold, and start to iterate on the prompt until you reach that threshold. Set up the system to give a scheduled daily, weekly, or monthly report.
Create a dashboard where its easy for the team to view real conversations logged from the system. Allow anyone to pull in real data into the testing system with one click. Make sure to add obfuscation if the conversation contains sensitive personally identifiable information.

At the end of this process, you will have:

A basic testing suite for your non-determininistic system
A report that can be run ad-hoc, with a history of how different generations of the system fared and how it improved over time
A dashboard that allows for real conversations to be added into this testing suite.

Admittedly, at the end of this process, there are a few issues that we must address. The solutions to this I will discuss in a future blog post.

The first real problem is that the data generated initially is not real data, and must be replaced with real data or fake data that is provably very close to the real data as possible.

The second problem is that deterministic tests are able to superficially test a non-deterministic system, but to fully test the success of a conversation requires another non-deterministic system. This can be a human being or another LLM, a grader LLM.

The solutions to these two problems I will discuss in a later blog post, and require an extension of the current system to accomodate those problems.

Case Study: T-Shirtify

Lets start with a customer service chatbot for a fictional e-commerce company that sells t-shirts called "T-Shirtify".

Step 1: Define the system

The first step is to define the system. We will start with a simple matrix of features, scenarios, and personas.

Features (What the system can do):

Order Tracking. The user can ask about the status of their order and get an update on the latest status information.
Return Processing. The user can ask to start a return process, and the system will guide them through the initial steps, asking any required information.
Product Recommendations. The user can ask a general query about if a specific product exists, and the system will provide a recommendation.

Personas (Who the system is designed to help):

Joe, Confused First-Time User. Joe is a 52-year-old high school teacher from Portland who rarely shops online and prefers calling customer service. He's a first time user of the system, and is confused about how to use the system.
Karen, Frustrated Customer. Karen is a 45-year-old busy mother of two from suburban Chicago who values efficiency and clear communication. She's a frustrated customer, and is angry about the system not working as expected.
Eva, Wholesaler. Eva is a 35-year-old small business owner from Miami who runs a local boutique and frequently orders inventory online. She's a wholesaler, and is looking to buy a large quantity of t-shirts for resale. Scenarios (A mapping of feature -> persona -> what they are trying to do):
Joe's Product Recommendations. Joe is looking for a t-shirt that matches a specific image or vibe, and the system will provide a recommendation.
Karen's Return Processing. Karen is looking to return a t-shirt that she received, and the system will guide her through the process.
Eva's Order Tracking. Eva is looking to track the status of an order, and the system will provide an update on the latest status information.

Step 2: Generate test scenarios

Now that we have defined the system, we can generate test scenarios. We will use a prompt engineering tool to generate a large amount of test scenarios. Here is an example of a prompt that can be used for this:

Using the following variables, I was able to generate 20 test scenarios that I've linked below:

<Context>
    <ApplicationFeature>
        <Name>Product Recommendations</Name>
        <Description>The user can ask a general query about if a specific product exists, and the system will provide a recommendation.</Description>
    </ApplicationFeature>
    <Persona>
        <Name>Joe, Confused First-Time User</Name>
        <Description>Joe is a 52-year-old high school teacher from Portland who rarely shops online and prefers calling customer service. He's a first time user of the system, and is confused about how to use the system.</Description>
    </Persona>
    <FeatureScenario>
        <Name>Joe's Product Recommendations</Name>
        <Description>Joe is looking for a t-shirt that matches a specific image or vibe, and the system will provide a recommendation.</Description>
    </FeatureScenario>
</Context>

Step 3: Add deterministic checks

Start with checks that code can answer without another model. These tests are cheap, repeatable, and easy to debug.

For each generated response, check requirements such as:

Whether the expected tool or function was called
Whether required fields are present
Whether prohibited words or phrases appear
Whether the response stays within length limits
Whether a database or application state changed as expected

Deterministic checks will not tell you whether a conversation was helpful. They will catch mechanical failures before you pay for a more complex evaluator.

Step 4: Version prompts and compare evaluation runs

Store the prompt version, model, model settings, test dataset version, and result for every run. Without these details, a higher pass rate is difficult to reproduce or explain.

Compare one controlled change at a time. If you change the prompt, model, and dataset together, you will not know which change affected the result.

A useful report includes the total pass rate, pass rate by scenario, repeated failure types, token cost, and links to failed examples. Set a release threshold for each critical test rather than relying only on one average score.

Step 5: Replace synthetic examples with real failures

Synthetic scenarios are useful for starting the system, but production conversations reveal cases you did not predict. Give reviewers a simple way to add a failed conversation to the evaluation dataset after removing personal or sensitive information.

Over time, the dataset should move from mostly generated scenarios to reviewed examples drawn from real use. Keep the difficult examples even after they pass. They become regression tests for future prompt and model changes.

Where LLM judges fit

Some requirements cannot be checked with code alone. Questions such as whether an answer is helpful, well-supported, or appropriate for the user require human judgement or an LLM evaluator.

Use an LLM judge only after deterministic checks pass. Define the evaluation criterion clearly, require a decision with reasoning and evidence, and validate the judge against a small labelled set. The RED framework covers this binary evaluation pattern in more detail.

When the task is comparative, such as selecting the best response from several candidates, use pairwise LLM ranking instead of asking the evaluator for an arbitrary score.

This gives you a practical first evaluation system: defined scenarios, repeatable checks, versioned results, real regression examples, and model-based judgement only where it is needed.