Company context
A large US cannabis marketplace and dispensary directory, free retailer listings on a 3% take rate. The client is the founder and CEO.
My role
I own diagnosis, evidence and the client-facing register, working alongside a project head, a crawl analyst and two engineers.
- 92,236 URLs across 12 families and 38 patterns, all template-generated, 55,892 of them AI-written blogs on one title formula
- An independent live audit overturned the kickoff assumption: 92,236 real URLs, not the 55,000 to 60,000 everyone was working from
- The previous agency is still engaged, so the mandate was support, not replacement: fresh issue IDs, three pages maximum, no named merchants
How it runs
At this scale you classify populations. You do not inspect pages.
AI is the classifier, and the second pair of eyes on my own arithmetic.
- 43 Python scripts, ~12,300 lines, in four reusable stages: recon, decode, build, render
- Decode classifies 94 vendor CSVs and ~1.4M rows by URL shape
- A standing index maps every vendor issue to the file holding it, validated 75 of 75
The same method keeps catching me:
- Disproved my own root cause, a pagination trap with zero such URLs in the set
- Withdrew a false P0: 1,300ms re-measured warm at ~12ms, reversing the strategy from crawl capacity to crawl demand
- Caught a colleague's export missing 23,800 rows
- Found the link crawler had seen 1 of 1,000 URLs in the Search Console sample
Integrations
Ahrefs exports · Google Search Console · Bing Webmaster · Screaming Frog · PageSpeed Insights API · CrUX · Claude Code · Python (openpyxl, python-docx) · curl under Googlebot UA · headless Chrome · Google Drive MCP
Vendor tools as data sources, not dashboards. Ahrefs is read through its exports and re-scoped to the templates, so the client never has to buy a bigger plan.
The site's own structured data was advertising 12,521 entry points into a soft-404 space no crawler could see.
Every sampled page declared a canonical pointing at an integer-ID route returning 200 with a 259-word placeholder for any integer, including 999999. Zero of forty matched their own canonical. Fixable in one serialiser.
Results
- 130,376 noindex URLs reconciled to zero residual from an 82MB export, without spending a crawl credit
- 81% of crawl surface and 475,951 internal links found dead-ending, fixable with a small front-end change
- 9,792 reported lost backlinks cut to 469 real, a ~20x overstatement, reframed as a churn dashboard
- Delivered a 9-category scored audit, a 43-ticket register with 12 P0, an 80-check migration validation, and a 3-page report with 6 evidence workbooks