Earlier this month one of the providers we track started returning errors instead of answers. Not all of them, and not for long — but long enough that a scan came back with 50 of its 66 queries on that provider missing. The other two providers answered everything.
Here is the uncomfortable part. Nothing about that scan looked broken. It had a score. The score was lower. And a lower score on a chart looks exactly like a brand losing visibility, which is the one interpretation guaranteed to send someone hunting for a problem that doesn’t exist.
That gap — between a number being wrong and a number being plausible — is what this post is about.
TL;DR
- A partial scan is marked partial, with the count of answers actually collected, and it’s kept out of trend comparisons rather than averaged in silently.
- Failures are reported per provider, not per scan. “One provider answered 16 of 66” is diagnostic; “148 of 198” is just alarming.
- Failed queries are now re-run automatically — two attempts, ten minutes apart and doubling — so most outages are repaired before anyone opens the dashboard.
- Only missing answers are re-run. Never ones that answered unfavourably. This matters more than the rest of the list combined.
- A failed query costs nothing. Credits are spent on answers, not attempts.
- If recovery can’t fix it, you can re-run the failures yourself, or an org admin can remove the scan so it doesn’t sit in your history.
A 21-point drop, and nothing on the chart says it is not real. The obvious next move is to go looking for a content or ranking problem that never happened.
Why a partial scan is worse than a failed one
A scan that fails outright is a nuisance. You see an error, you re-run it, you lose an afternoon. Nobody makes a decision based on it.
A scan that half succeeds is a different animal, because it produces a number, and numbers get acted on. Visibility scores are averages over the answers collected. Take away a third of the answers and the average doesn’t announce that it’s now measuring something else — it just moves. Down, usually, because a brand can’t be mentioned in an answer that was never collected.
So the failure mode isn’t “we lost some data”. It’s “we quietly published a different metric under the same name, on the same axis, next to five points that were measured properly”. Everything downstream inherits it: the trend line, share of voice against competitors, the client report you exported that afternoon.
This is a category-wide problem and mostly an invisible one. Every tool in this space queries providers that fail sometimes; the difference is whether the tool tells you. A single number with no denominator can’t.
What happens now
The scan says what it collected. Status, answers collected against answers requested, and a per-provider breakdown. Two providers at 66 of 66 and one at 16 of 66 is an upstream fault, and it reads as one immediately — where a single aggregate figure would have you checking your prompts.
The provider rejected these requests rather than answering them. Nothing about the prompts changed — the same 66 ran clean on the previous scan, and the other two providers answered all 66 on this one.
Both are available to you, but you rarely need them — automatic recovery has already started.
The failures are re-run without you asking. This is the part that shipped most recently, and it’s the one that changes the experience: a transient provider failure is usually repaired before anyone looks. Two automatic attempts, the first after ten minutes and the delay doubling after that, re-running only the queries that came back empty.
The delay is the point. A provider having a bad minute recovers on its own; retrying instantly just collects the same errors again, more expensively.
One provider stopped answering partway through. 50 of its 66 queries came back as errors, so the scan is marked partial rather than complete. The 50 failures cost nothing: credits are spent on answers, not attempts.
You keep the manual controls. Recovery has bounds, and past them it stops. When it does, the failed queries are listed and you can re-run just those — not the whole scan. Asking is never capped, because a person asking has a reason.
And you can delete a scan. Some scans aren’t worth repairing. An org admin can remove one, and it comes out of the charts, the trend line, and any report generated afterwards. We’d rather give you a way to remove a bad measurement than have you explain it to a client every quarter.
The honest limitations
Recovery only re-runs queries that returned nothing. It cannot and does not re-run a query that answered in a way you wouldn’t like. If that distinction sounds like an implementation detail, consider the alternative: a system that retries unfavourable answers is a system that manufactures its own numbers, and every score it has ever produced would be worthless. Missing and unfavourable are different states, and only one of them is eligible.
It costs credits. Recovered answers bill exactly like any other answer — an answer is an answer. Failed attempts cost nothing, and the cap is two automatic attempts per scan, so the worst case is bounded and small. But it does mean recovery spends money without asking you first, which you should know rather than discover.
There’s no per-project off switch yet. Recovery is on for everyone. If the credit behaviour is a problem for how you work, tell us and we’ll add the setting — it’s a small change and nobody has asked yet.
Some failures are terminal. If a provider is refusing requests outright, a retry is just a slower way to collect the same error. Recovery stops early when an attempt gains nothing new, rather than spending its second attempt learning the same thing twice.
A recovered answer is collected later than its siblings. Minutes to hours later, which is why recovery only applies to scans that finished within the last twelve hours. Past that, a “repaired” answer would be measuring a different day to the rest of the scan, and we’d rather leave the gap visible than fill it with something that isn’t comparable.
Recovery is not a guarantee. A scan can end up partial anyway. When it does, it stays marked as partial — the mark isn’t removed just because we tried.
FAQs
Do I need to turn this on? No. Automatic recovery is live for every project on every plan.
Will it retry my old partial scans? No. Only scans that finished within the last twelve hours are eligible, for the comparability reason above. Older partial scans stay as they are, and you can still re-run them manually if you want the answers.
Did my charts have silent holes before this? Partial scans have always been marked partial — that isn’t new. What’s new is the per-provider detail, the automatic repair, and the ability to remove a scan that isn’t worth keeping. Before, a partial scan was something you could see and nothing you could do anything about.
Does a retried scan re-send my completion email? No. Recovery only ever touches scans that have already been finalised, which structurally puts it outside the code path where completion emails, report delivery, and automatic annotations happen. It can’t re-fire them because it never goes there.
Are the recovered answers scored differently? No, but the scoring is incremental: answers that were already scored aren’t re-scored, and recovered ones are added. You don’t get double-counted mentions from a retry.
How do I know whether a scan was repaired? The scan’s diagnostics show the attempts and what each one recovered. If a scan reached full coverage after recovery, it reports as complete, because it is.
Most of the trust in a measurement tool is built in the unglamorous places — what it does when something upstream breaks, and whether it tells you. The providers we track will fail again; that part isn’t up to us. What is up to us is whether a broken scan arrives on your chart looking exactly like bad news.