How we evaluate digital twins

august is built to get you closer to your customers, and their insights, faster. Modeled on real people who matter to your business, digital twins help anyone on your team surface the kind of overlooked insight that usually takes weeks to find, in minutes. In an earlier post we explained how a Digital Twin is trained. This post answers the question that should come immediately after: how do we know a twin actually behaves like the person it models?

Three premises shaped every design decision in our evaluation:

1. The only ground truth for what a person would do is what that person actually did. We test against held-out real behavior, never against opinions about whether an answer sounds right.
2. A twin has to beat the model it is built on. If Claude given the same information does just as well, the twin is a costume.
3. A test is only as good as its ability to be failed. Every control must be a genuine competitor, not a prop.

The rest of this post is those three premises turned into an instrument.

How the test works

The primary idea is to see how well a twin can identify the behavior of the customer it models. Twins were trained on a fixed number of sessions and made to identify the next actual session that followed.

Building a twin that cannot know the answer

Every case starts with one customer and at least 51 real sessions. The twin is built from the first 50, exactly the way production builds every twin: same pipeline, same stages, same persona, memories and reflections. The next session is held out as the answer. Nothing that touches the twin may contain information from that session or anything after it: purchases, searches and page views are pulled from the training window only, and every build is audited for leaks from the future.

At test time the twin also behaves exactly as it does in production. It does not get its whole history pasted into the prompt; it retrieves the memories and reflections most relevant to the question at hand, ranked by relevance, importance and recency, the same way it would in an interview or a focus group.

The prefix

A prefix of the 51st session i.e. the held-out session was created.

What is a prefix? A prefix is a set of initial events in the held-out session’s complete sequence of events. It is the opening slice of the held-out session: its first events, rendered only from what had happened up to that point.

Why is a prefix necessary? The prefix exists to make the comparison fair. Asked which of four sessions belongs to a person it knows nothing about, with no glimpse of the visit, a default model can only guess, and the results show exactly that: default Claude’s accuracy always ranged between ~20% and 30%, and basic probability shows that picking 1 out of 4 options has 25% accuracy. Revealing the same opening equips each arm with legitimate evidence, so what the test measures is what the twin adds on top of it.

We cut each session by event count rather than clock time, because activity comes in bursts, sessions can include significant idle time - a cut at 20% of elapsed minutes would hand some sessions nearly all their events and others almost none. Everything the models see is written only from events before the cut. Every candidate ending is written only from events after it. Overlap between question and answer is impossible by construction, not because we asked the model nicely not to peek.

The decoys

The decoys come from sessions of other customers of the same org, never from the twin's own person, and they are chosen to be hard. Purely random sessions can make the test easy. A visit that opens on ski bindings is easy to tell from one that opens on garden hoses. So candidates are matched by comparing openings to openings: each session's own prefix is embedded and scored against the real one, filtered to the same kind of device, at most one session per customer. The matching algorithm never reads the endings. The canonical design uses the three strongest matches based on the prefix. The endings the models actually read are freshly written from each session's post-cut events by the same rendering machinery production uses for memories, blind to which option is the answer.

Scoring

Scoring is simple. The model picks an option, the option either is or is not the real session. There is no text generation or semantic evaluation needed to compare quality. It is simply a forced choice test, deliberately straightforward.

What we tried first

The test setup we landed on was not the first pass at evaluating the twins. The obvious way to test a twin is to ask it to predict the future outright, so that was what we built first.

Each arm (twin, random twin and default Claude) was shown the prefix and asked to predict the rest of the session: the likely actions, the likely products, the likely outcome and how the visit ends. We converted that prediction into a vector embedding. A pool of real session summaries from all customers of the same org was also converted into vector embeddings. We then ran a similarity search to rank each of the session summaries in the pool. The higher the true session's summary ranked in that search, the better the arm's score.

The twins performed exceptionally well. Across pool sizes of 1000, 10,000, and ~90,000 session summaries, the true session summary was regularly ranked within top 10. However, so did default Claude. The true session summary was ranked just as high by a twin and Claude. Often, the twins did not beat Claude on that benchmark.

Before believing the result, we ran a control the design itself called for: embed just the bare prefix, no prediction from the twins, and run the same search. The control beat every arm. Copying the question ranked the real session first 57% of the time; the best model-written prediction managed 36%.

This finding was foundational in the final test design setup. Here is what we learned from the original runs:

A session summary describes the whole session, opening included. The winning strategy was to just restate the prefix, and every word the twins or default Claude wrote about the future pushed its prediction further from the target. To prevent this, we now ensure the prefix is removed from the options shown to the twins during the test. The prefixes are sliced and the twins are only shown events that come after the slice.

Similarity search measures how much two pieces of text mention the same things, not whether the behavior in them is the same. So a prediction that named the right products but the wrong outcome scored higher than one that called the outcome correctly in different words. Similarly, naming the wrong products or pages also pushed the prediction further away from the true session summary. The clearest tell came when stripping detail out of a twin's context improved its accuracy. As a result, we switched the test from similarity search to a deterministic test that picked the right option from a restricted pool of session summaries.

The baselines

Why not just prompt Claude

The obvious objection to any twin is that a well-prompted frontier model would do just as well. So default Claude is our primary baseline, and it is treated as a real competitor: the same model, the same instructions, the same prefix and the same four options. The only thing it lacks is the knowledge of the customer’s past sessions. If the twins consistently outperform a default llm model in predicting the correct session that belonged to the customer it models, that perfectly captures its value. It proves that each customer’s behavioral footprint is unique and is being captured correctly by august twins.

Aren't all twins the same underneath?

The second objection is that twins are one clever prompt wearing different names. The random twin control answers it: run the identical pipeline, but hand the model a different customer's twin. If twins were generic, the random twin would score the same as the correct one. Instead it falls below blind guessing. The persona and memories of a different twin i.e. a different human actively mislead it into making the wrong choices. This proves that the signals are truly personal. The value of the twins lie in the fact that they model their underlying human accurately by capturing the uniqueness and nuances of their behavior. If the twins failed at doing this OR there really weren’t any differences in user behavior, there truly would be no difference in their performance in these evals.

The results

Here is what happened at a large specialty retailer, where 47 customers passed every eligibility gate.

The hardest version of the test shows the twin no opening at all: just the four candidate endings and nothing else. The twin uses only the unique customer footprint it captures. The twin picked the correct session 66% of the time.

Claude without a twin scored 25.5%. With no context of the particular session, or information about the specific customer, there is nothing to reason from. With a 1 in 4 chance of picking the right session, Claude sits squarely at the ~25% probability mark, as expected. The twin's 66% accuracy is person-knowledge alone, and the 40 point gap is statistically significant.

Hand that same test to a random twin, the identical case with the wrong customer, and accuracy falls to 19.1%, below blind guessing. The wrong person's context is worse than none: the model demonstrably uses the persona and memories it is given, and the wrong ones actively mislead it, leading to a significant 47 point gap with the real twin. If twins were interchangeable prompting, that swap would have cost nothing.

Then we made the fight fair for Claude. Reveal the first three events of the visit and Claude climbs to 38.3% because now it has evidence to reason over. On the same test, the twin scores 57.4%. That is a 19 point gap in a contest Claude is genuinely equipped to win.

When the twin and Claude disagreed about which ending was real, the twin was the one holding the right answer 3 times out of every 4 disagreements.

None of this is the artifact of one lucky configuration. We reran the study under three instrument shapes: every decoy hand-picked to match the opening, a mix of matched and random decoys and a five-option version where chance drops to 20%, and the thin-context advantage stayed statistically significant in all three.

Arm	Without prefix	With prefix
Twin	66.0%	57.4%
Different Twin	19.1%	32.6%
Default Claude	25.5%	38.3%

Why this matters

august digital twins are created to embody your real customers. For how a twin is built in the first place - traits, memories and reflections - see How we train Digital Twins. It earns its claim through these validation tests. The results say something precise: twins capture the nuances of individual customer behavior, the fine grain of how one person browses, compares and decides, reliably enough to pick that person's real session out of a lineup of lookalikes. When teams use twins to pressure test a concept, a flow, or a price, they are designing against how their customers actually behave rather than how an average says they should. That is what it takes to develop and curate experiences genuinely tailored to the preferences and behavior of the people the product is for.