A study tells you where you stand. Monitoring shows what changed over time using the same frozen prompt bank, models and codebook. A comparison does not prove what caused the change.
One instrument, run again
The study ends with a frozen prompt bank. Monitoring re-runs that same bank: same questions, same models, same codebook, read the same way. That is what makes two runs comparable. Without the frozen bank, a second measurement is a new anecdote, not a trend.
Between the two runs sits the work.
- Study, ending in a frozen bank
- The work on the sources
- Monitoring re-runs the same bank
The order is study, then the work on the sources the report points at, then monitoring: the re-run shows what changed after that work, without proving what caused the change. A second measurement without intervening work can still record changes in model behavior or third-party sources.
Recommended
Invented sampleSame frozen bank, same models. The difference does not prove causation.
44 of 396 responses in the first run, 75 of 396 in the second, after the work. The same 396 responses stand behind both numbers, so the difference is readable. Change the bank and the two numbers stop meaning anything together.
What breaks comparability:
Editing questions after the baseline
New questions start their own baseline, with their own first run. They never join the existing line.
Changing models or repetitions mid-trend
The instrument changed, so the numbers changed for reasons that are not yours.
Splicing banks into a trend
Never. We document the bank so any later edit is deliberate, not accidental.
Cadence, agreed per client
No fixed package. The cadence depends on how fast your category moves and when your decisions happen: budget cycles, campaign windows, retailer negotiations.
Two rhythms combine:
Periodic runs
- With the frozen bank, to keep the line moving between decisions.
Deeper re-runs
- At longer intervals, more models or more repetitions, for the numbers that support bigger decisions.
Alerts run between and across runs:
Drop
Your visibility drops
Against the previous comparable run.
Pattern
A specific critique repeats across assistants
One negative answer is noise; the same critique in several answers is a pattern with a source attached.
Rival set
A competitor enters or leaves the answer set
For questions that used to be stable.
Every alert names the question, the model and the responses behind it. Alerts are measurement with a citation, not a newsfeed.
What monitoring does not tell you
Kept visible on purpose:
It does not prove causation
A model update or new third-party content can move the same number. The comparison tells you what changed. It does not prove you caused it.
If nothing moved, the report says nothing moved
No chart gets stretched to show progress.
Third parties still write most of what gets cited
Part of the movement, up and down, sits outside anyone's control.
What the trend panels show
One panel per outcome rate (Mentioned, Recommended, Named as the best, With a purchase path). In each panel:
Points
Two points, one per run, connected by a straight line
Labels "run 1" and "run 2" with the run date.
Denominator
The rate above each point, the denominator printed under each point
"n = 396 responses". Denominators are part of the visual, not footnotes.
Delta
The delta labeled between the two points
In percentage points, with a plus or minus sign. Zero movement is drawn as zero.
Scale
A fixed scale per panel
So panel heights are not stretched to exaggerate change.
Caption
A fixed caption under every panel
"Same frozen bank, same models. The difference does not prove causation."
The panel above this section is that specification, drawn.
The honest annotation is the caption. It ships with every panel, including the ones where nothing moved.
Start with the baseline, or continue it
No baseline yet
- The study comes first: monitoring without a frozen bank has nothing to compare.
You already have one
- Ours or from another provider, monitoring starts with a conversation about the decision the numbers need to inform: is what we are doing working, and how do we know?