august in the shipping loop
Ansuman M.
September 2, 2026
august began with a simple interface : teams could ask their customer twins a question and get their responses back.
This interface was sufficient for one time conversations, but repeated usage didn’t follow at the scale we expected. Our hypothesis was that the first main problem was access : asking for twin feedback required people to step outside where they were already working. Naturally, we wanted to meet our users where they work and built out a suite of entry points - MCP, Slack App, Figma Plugin, CLI, Claude/Codex connectors and the like.
That helped, but it did not remove the underlying blocker. Someone still had to recognise that a decision would benefit from customer context, remember that august was around, switch context, and frame the right questions.
We wanted to remove this dependency and test a different model : what if august could participate automatically in a workflow, with enough context to say something useful, while the decision was still in progress (and reversible).
A PR review just so happens to satisfy all these requirements. It already involves a meaningful engineering change, contains the implementation in code, and occurs before the change has reached users. The goal, however, would not be to review the code for technical correctness. Instead, august would try to understand what the change means for users and get feedback from relevant customer twins while there is still time to act.
This led to the august GitHub App. For each product change, an agent investigates the repository and produces a product-change spec. august uses that spec to identify and interview the most relevant twins, then posts the feedback directly on the PR as a comment.
The same product-change spec is persisted beyond the merge. A weekly Release Impact Report reuses it to connect what the team intended to ship with how users actually behaved afterwards, and to propose a concrete next test.
Throughout this post, we use a change to the Shopping Assistant Conversations Analytics in our dashboard as the running example as it crosses enough layers of our tech stack to explain the hard parts of the review problem.
1. Why a diff is not enough
The Shopping Assistant Conversations Analytics change touched nine files.
Those files included data warehouse models, analytics lookups, Cube configuration, Hasura queries, tests, and a roughly 700-line dashboard UI change. Together, they enabled a brand user to load a recent conversation list, filter it, and open a read-only transcript drawer.
The pull-request title could describe that at a high level, but it could not sufficiently answer all the questions the review needed: where the change appears in the user journey, what is the user trying to achive in that workflow, and which group of users might encounter this change most frequently.
So why not shove the code diff into an LLM? 2 problems.
The first obvious one is context. For a sufficiently large PR, the full diff may not even fit into a single model call. Even when it does, the largest files occupy most of the context window regardless of how much they contribute to understanding the user-facing change. Furthermore, relationship between two hunks in the diff may depend on code that did not change at all. We needed a more surgical exploration of the repository.
The second problem is harder to detect. A model can produce a seemingly coherent explanation without accounting for the whole change/diff. The output may sound right even when a subset of hunks might not have been read. So we also needed to guarantee that each and every change has been accounted for.
2. How an agent investigates a repository
The exploration is done by an agent, henceforth referred to as the ‘planner agent’, or just ‘planner’. It receives two inputs : the exact base and head revisions, and a complete manifest of changed files and hunks. The manifest here acts as both a starting point and a coverage checklist. Every meaningful item in it must eventually be connected to the product change interpretation or explicitly exempted.
The planner receives enough seed information from the manifest to start, forms a working plan, and requests incremental repository context as the investigation progresses. It does it via six simple tools it has access to:
- Read a diff hunk
- Read the base version of a file
- Read the head version of a file
- Search literal text in the repository
- List a directory
- Submit semantic coverage
During a run, the planner doesn’t blindly read whichever version of a branch happens to exist when a tool call is executed. Rather the reads are tied to the base or head revision only. This is important as it prevents a force-push, a follow-up commit, or branch deletion to change the evidence underlying an in-progress review.
Each operation is constrained by line ranges, result size limits, pagination, and a wall-clock deadline. The agent can move around a large repository, but it cannot dump an oversized result into its context window or investigate indefinitely.
As for the other problem of coverage - the planner does not decide for itself that the investigation is complete. It instead submits a coverage ledger.
For every changed path and changed hunk, the ledger requires one of two explanations : how the item contributes to the inferred product behavior, or why it is safely exempt. Coverage is rejected if the agent leaves a hunk unread, or invents a repository path, or omits a manifest entry. The planner is sent back into the loop till the completion criteria is met.
For the Conversations Analytics pull request, its output looked like this:
Summary: Make the existing Shopping Assistant Conversations tab operational with a filtered conversation list and read-only transcript drawer.
Surface: Web dashboard → Shopping Assistant → Conversations
Workflow: A brand user opens Conversations, reviews the default seven-day list, changes the date range or filters by device, add-to-cart, or purchase, then opens a conversation to read its transcript.
Before: The routed Conversations tab showed a message saying that the conversation list and transcript functionality would ship later.
After: For brands with Shopping Assistant enabled, the tab shows up to 100 recent conversations with time, device, add-to-cart and conversion badges, turn count, and the first question. Users can filter the list and open a side drawer containing the shopper and assistant turns.
Affected users: Brand-dashboard users with Shopping Assistant enabled, especially people investigating shopper interactions and commercial outcomes.
Selection query: Select active brand-dashboard users whose organization has Shopping Assistant enabled, especially users who visit the Conversations route or analyze conversations, device mix, add-to-cart activity, or purchases.
3. What happens before merge
Once the planner has validated the product-change spec, august can run the pre-ship review.
The GitHub webhook delivery queues a background job pinned to the PR’s exact base and head revisions. If GitHub delivers the same event twice, august updates the existing review instead of creating a duplicate. And if a new commit is pushed, the review is rerun against the new head.
august then uses the product-change spec to select relevant customer twins, based on prior evidence that they have used the affected workflow. Each twin reviews the change independently and their findings are synthesised and posted as one coherent review.
4. Reusing the spec after merge
Each validated spec is stored against the pull request’s exact head revision. When that revision merges, the Weekly Release Impact Report reuses the spec as its record of what the team intended to ship.
Some pull requests merge before the GitHub App completes a review. In that case, the planner agent generates the missing spec against the merged head. It does not select twins, create a GitHub check, or post a comment. It only produces the artifact needed by the post-merge workflow.
The spec gives the report one side of the comparison i.e. the intended product change. august’s observation memories provide the other side : what users did after the merge. The next problem is deciding which observations, if any, are actually related to the change.
5. Choosing the behavioral evidence unit
august processes session replay into several layers of memory:
A session arc observation summarizes the session as a whole.
An incident observation isolates an interaction with enough evidence to stand on its own, such as a visible error or a qualifying rage cluster (“incident” does not necessarily mean “bug”). Usually a session produces several observations because and each visit may span multiple different workflows (potentially relevant to different releases)
august also builds reflections across multiple observation memories. These reflections help understand broader behavioral patterns for the twin. The distinction between a reflection and an observation matters here.
The Release Impact Report needs to know precisely what behavior occurred before or after a specific merge. A reflection such as “this user usually checks anomalies before changing configuration” might be useful context, but it’s a lossy compression of the observation memories’ incidents and timestamps. The report therefore uses active observation memories as units of evidence.
An example, On august 6, one internal dashboard session produced this observation:
Recall label: Filtered Conversations and reviewed AI-search analytics
Memory text: Spent about 16 minutes working in the AI-search area on desktop, moving between app and settings pages. Dug into the Shopping Assistant’s Conversations view, opened Filters, selected the Purchased filter, and applied it. Then checked AI-search analytics, switched the date range to Last 30 days, and read the information tooltip on one metric.
Some observations also carry a richer session exemplar with route sequences, top clicks, workflow labels, outcomes, and a compact transcript attached which we may use during the association phase (see below)
6. Associating a release with behavior
There are two separate questions while associating a release with observations : Could this observation belong on the ‘before’ or ‘after’ side of this release at all? And, does the observation actually describe the workflow changed by the release?
We tackle them one step at a time.
6.1 Temporal eligibility logic
For the proof of concept, a merge to main was treated as the deployment time. This is a deterministic approximation. Every reviewable pull request was given a 30-day baseline ending at merge and about 14 days of follow-up beginning at merge. We apply those temporal filters to observations as the first step.
For example, change A merged on august 1, change B merged on august 9, and an observation occurred on august 5. The observation is after A and before B. The report may later group A and B into one rollout for readability but it cannot rewrite the august 5 observation as post-release evidence for B.
6.2 Several pull requests grouped as one feature release
A release window often contains separate data, backend, frontend, and visual-polish PRs for one customer-facing feature.
An LLM call receives all complete PR specs and clusters them into feature groups. We validate that every reviewable PR appears in only one group. If it cannot be reliably grouped with other PRs then it stays a singleton.
Changes F through H in the figure are illustrative. Only A through E belong to the dogfood run.
For our running example feature, five pull requests formed one Shopping Assistant analytics rollout:
| Change | Product contribution | Merged |
| A | Product navigation and routes | Jul 31, 07:00 UTC |
| B | Analytics KPIs and definitions | Jul 31, 07:11 UTC |
| C | Conversations list, filters, and transcript drawer | Jul 31, 07:23 UTC |
| D | Transcript search, sorting, and pinned controls | Jul 31, 20:49 UTC |
| E | Analytics design refresh | Aug 4, 19:40 UTC |
6. 3 Vectorized retrieval
Once we have our feature clusters, the next step is to plausibly match temporally eligible observations to each product change group. To prune obvious non-matches, the first pass is a quick embedding filter.
We embed only the selection query from each product-change spec (to reduce noise) and observation embeddings already exist from memory ingestion. Then the engine forms two normalized matrices:
A vectorized temporal mask removes cells outside the baseline and follow-up window for each pull request. This evaluates the logical N-by-M candidate space with matrix multiplication instead of nested application code.
For each observation and feature pair, the engine takes the highest score among the feature’s temporally eligible contributing changes. The default threshold is 0.50.
There is no top-K cap nor any winner-takes-all assignment by design, since one session can produce several observations, and one observation may genuinely relate to several changes.
6.4 Semantic reranking
Embedding similarity only gives plausible matches which we then have to examine more closely. Every shortlisted observation is judged against the matched feature group with a structured-LLM reranking call.
The reranker receives the complete observation and its structured evidence, rich session exemplar if available, context for the twin whose observation is being matched, product change specs in the feature group, and cosine scores for each change from the embedding filter step.
The aforementioned August 6 observation had a feature-level similarity of 0.553, so it passed retrieval for the five-change Shopping Assistant group.
The reranker associated it with:
- Change A, because the user navigated the dedicated Shopping Assistant product area.
- Change B, because the user reviewed KPI content and metric definitions.
- Change C, because the user opened Conversations, selected Purchased, and applied filters.
- Change E, because the user exercised the refreshed analytics surface.
NOTE : The result says that this exact evidence supports an association with A, B, C, and E. We do not claim that those releases caused the user to behave that way.
7. Asking the twins who used the workflow
Observed behaviour / evidence can tell us facts like a user filtered the Conversations list post release, opened a transcript, returned to the list, and opened another one - but it does not directly tell us what decision they was trying to take, or what change would make the workflow better. This is where the twins come back to the picture again.
Here the selection criteria for twins is expectedly different from the pre-ship review. Before launch, august selects twins by relevance because proving direct use is impossible, but after launch, relevance by itself isn’t sufficient. The report requires evidence that the represented user actually exercised the changed workflow in the post release window.
For the Conversations workflow, the question planner started with 26 post-release observations across six users. It retained only observations that directly showed the workflow being exercised which resolved cleanly to two twins.
Both twins received the same question independently:
Think of the last time you opened the Shopping Assistant Conversations tab and clicked into an individual conversation to read the shopper and assistant messages. What were you trying to learn or decide? Where did you feel unsure or work around something? If you could change one thing about how conversations are listed or displayed, what would make that job easier?
User A connected a repeated open-and-return behavior to a specific job:
I’m usually trying to figure out where the assistant lost the shopper: whether it is a content problem, a persona problem, or a carousel or action problem.
She described the list as too thin to support that triage:
There’s a lot of open, skim, back, open, skim, back. I’d put more of the transcript signal in the row before I click.
User B described a similar investigation:
I was trying to decide whether the assistant actually understood the shopper’s intent and handled the thread cleanly.
Her preferred change was:
Keep the filtered list pinned on the left, open the transcript on the right, and highlight the exact message or outcome that made the conversation match the filter.
Obviously, these answers are not a consensus of all customers. They are hypotheses from two observed users, grounded in ten sessions worth testing next. And it’s strictly better than flying blind.
8. Turning the result into a test
Post the interviews, the generated report keeps three separate layers:
| Layer | What the evidence supports |
| Observed behavior | Users filtered Conversations, opened transcripts, returned to the list, and opened others. |
| Twin interpretation | Two observed-user twins said they were triaging failures and lacked enough row context. |
| Proposed test | Try richer rows and a side-by-side transcript layout before adding more controls. |
The observation is recorded behavior. The interview supplies an interpretation of the behaviour. The proposed action-set is a hypothesis that still needs to be tested in the product.
For this example, the actions proposed would be:
- Put richer rows behind a feature flag, including the last shopper message, last assistant message, and a reason or outcome tag.
- Test a pinned list with a side-by-side transcript, highlighting the moment that caused the conversation to match the active filter.
- Interview three to five more confirmed Conversations users before committing to a larger redesign.
And the result to be monitored is : “Over two weeks and at least 20 confirmed Conversations users, the treatment should reduce very short transcript opens without scrolling. At least three of five follow-up interviewees should also say they can triage more conversations without opening every row. If neither measure moves, revert the flag and revisit the hypothesis”
Closing the loop
At Fermat, August has already run 344 pull-request reviews, involving 94 twins and 1,109 twin conversations.
To sum up, the systems documented above bring relevant customer context/caution into a change before it ships, follow that same change into observed usage, and turn what users do afterwards into the next product hypothesis or test. This extends august from a research primitive someone has to invoke, into a customer-feedback resource which chips in autonomously in the shipping loop.