Nearly every published ecommerce seo case study shares the same structure: a client with a problem, a list of things the agency did, a green line going up, and a percentage. Almost none of them are falsifiable. You cannot check the baseline, you cannot see what else changed, and you cannot tell whether the result would have happened anyway.
This article does not present client numbers. It teaches you to interrogate the ones you are shown, and to design a test on your own store that would survive the same interrogation. That is more useful than another anonymised chart, and considerably harder to fake.
Why Most Published Case Studies Prove Nothing
Start with the structural problems. They are not usually dishonesty — they are the natural result of writing marketing material about a noisy system.
- No control. Traffic went up after the work. It might have gone up without it. Without a comparison group, the claim is a correlation dressed as a result.
- Convenient baselines. A period chosen to start at a seasonal trough, or immediately after a penalty, produces a spectacular percentage from arithmetic alone.
- Confounded changes. The same six months usually include a site redesign, a paid campaign, a PR push, new stock and a pricing change. Attributing the outcome to one of them is a choice, not a finding.
- Cherry-picked keywords. On a catalogue of five thousand pages, you can always assemble twenty terms that improved. Reporting those twenty is not measurement.
- Vanity outcomes. “Rankings improved 400%” is not a unit. Impressions are not revenue. Traffic is not orders.
- Survivorship. Nobody publishes the engagements that failed, which is why the visible sample looks so much better than the average result.
The Questions That Separate Evidence From Marketing
When someone shows you a case study — an agency pitching, a tool vendor, a conference talk — ask for these. The response tells you as much as the answer.
- What were the absolute numbers? Not percentages. “Up 300%” from 400 monthly sessions is 1,200 sessions. That may still be a good result; you just need to know which scale you are looking at.
- What was the exact date range, and why that range? Compare year-over-year for seasonal categories. Month-over-month in retail is mostly seasonality wearing a costume.
- What else changed in the same period? Redesigns, replatforms, price changes, new products, paid spend, PR, a competitor exiting.
- What was the control group? On a store there is almost always one available — comparable pages left untouched. If none was held, say so.
- Where is the revenue number? Organic revenue or orders. If the case study stops at sessions, ask why.
- What is the data source? Search Console and analytics are your ground truth. Third-party position and volume figures are modeled estimates from periodic crawls — fine for competitive comparison, weak as proof of a specific change.
- What did it cost, and over what period? A result delivered in twenty-four months for six figures is a different proposition from the same result in six months for twenty thousand.
- Did it hold? Twelve months later, is the site still there? Plenty of impressive charts flatten the quarter after publication.
A practitioner who has genuinely measured something will answer these comfortably, including the awkward ones. Evasiveness on question three or four is the tell.
Reading the Chart Itself
A few specifics that catch most inflated claims. Check whether the y-axis starts at zero — a truncated axis turns an 8% lift into a cliff face. Check whether the annotated “work started” marker sits before or after the inflection; frequently the rise begins weeks earlier, meaning something else caused it.
Look for the shape. Genuine organic growth from content and technical work is gradual and compounding. A vertical step in a single week is usually an indexation fix, a penalty recovery, a tracking change, or a measurement artifact — all legitimate, but different stories with different transferability to your situation.
And check that the traffic is commercially relevant. A store that tripled organic sessions by ranking a viral blog post about office chair ergonomics has tripled traffic that does not buy chairs. Segment matters more than total.
Designing a Test on Your Own Store
The most valuable case study you will ever read is one you ran. Ecommerce is unusually well suited to this because you have hundreds of comparable pages, which means you can build a genuine control group — something a single-site blog can never do.
Structure it like this:
- Pick one intervention. One. Rewriting manufacturer descriptions. A new category title template. Adding buying guidance to category pages. Testing three things at once yields three unattributable outcomes.
- Build two matched cohorts. Take a page type — say 200 product pages — and split them into a treatment group and a control group matched on current traffic, revenue, page depth, category and existing copy length. Randomise the split within those strata.
- Freeze the baseline. Export Search Console impressions, clicks, average position and organic revenue per URL for both cohorts across the preceding 8–12 weeks. Save the file. Baselines reconstructed after the fact are how people fool themselves.
- Deploy to the treatment group only, on a single date you record. Change nothing else on either cohort.
- Wait. Six weeks minimum for content changes, twelve for anything structural. Resisting the urge to peek and act at week two is the hardest part.
- Compare the change in the treatment group against the change in the control group, not against the treatment group’s own past. If both rose 15% because it is November, your intervention did nothing.
What to Measure, In Priority Order
Rank first: organic revenue per cohort. Then orders, then organic sessions, then Search Console clicks, then impressions, then average position. Most published case studies report that list backwards, because the metrics at the bottom move first and look most dramatic.
Impressions and average position are still useful — they are leading indicators, and they tell you whether a change registered at all before revenue has had time to respond. Just do not promote them to the headline.
Two honesty constraints to build in. Daily movement of two to three positions is ordinary noise, so use weekly aggregates and cohort averages rather than individual page positions. And accept that a null result is a real finding: knowing that rewriting mid-tier product descriptions did not move revenue on your catalogue saves you from rolling it out across four thousand pages.
Documenting It So It Is Actually Reusable
Write the test up while the details are fresh, in a format that would satisfy the questions above. Hypothesis, cohort definitions and sizes, baseline period and figures, intervention description, deploy date, measurement window, results for both cohorts in absolute numbers, confounders you know about, and your confidence level.
Include what you could not control. A competitor relaunching mid-test, a core update landing in week four, a stock-out across half the treatment cohort — these belong in the record. A case study that names its own weaknesses is worth ten that do not.
Practically, this is easier when the data lives in one place. Rank tracking that stores top-100 snapshots with movement deltas between checks, alongside connected Search Console and GA4 data, gives you both the competitive estimate and the ground truth for the same period — and a shareable read-only dashboard means a client or a stakeholder can see the same figures you did rather than a chart you assembled. Full-site crawls with per-issue evidence give you the pre-test inventory of which pages actually had the problem you are testing.
The Standard Worth Holding
Apply one rule to everything you read and everything you publish: would this convince a sceptic who wanted it to be false? Most case studies fail that test immediately, and the ones that pass are almost always more modest in their claims.
Run two or three properly controlled tests on your own store and you will have better evidence about what works on your catalogue than any external seo ecommerce case study can offer — because it comes from your products, your customers and your competitive set. SEO Rocket exists to make that measurement loop cheap to run at a flat US$50 a month; the discipline of running it honestly is still entirely on you.
