File viewer

crawl_13082026_0133.md

/app/data/llm/analysis/firstpage/crawl_13082026_0133.md

Only one URL was crawled: the homepage, returning a 200 and marked indexable. That single data point is unremarkable. The problem is everything else.

What stands out - The crawl retrieved exactly 1 URL from a site known to have ~34,000 URLs. Even allowing for a 20-URL sample list, this is a near-total coverage failure. Without knowing the specific list input, we can’t determine whether the sample contained only the homepage or whether the 19 other listed URLs were unreachable, blocked, or redirected in ways that prevented a full fetch. Either scenario is a critical data gap. - Because only one URL was crawled, every other metric — duplicate titles, empty H1s, missing meta, directory distribution — is zero trivially. No structural, on-page, or indexation issues can be assessed from this output. The data is not just sparse; it is functionally absent for analytical purposes.

Connections and implications - This is consistent with a site that either aggressively blocks crawlers (e.g., via Cloudflare or bot management) or has a navigation structure that the Screaming Frog configuration couldn’t follow. The homepage returning 200 rules out a complete server outage, but does not rule out bot-blocking on deeper paths. - The discrepancy between a 34K-URL site and a 1-URL crawl is the single most important finding. It carries more weight than any internal SEO metric because it prevents diagnosis of those metrics entirely.

Hypotheses vs. evidence - Hypothesis: The site blocks automated crawlers. This is plausible given the 1-URL crawl, but the evidence is only suggestive. A single 200 from the homepage does not confirm or refute blocking. Server log analysis or a live probe with a controlled user-agent would be needed to verify. - It is equally possible that the sample list was misconfigured or omitted all but the homepage. We cannot rule that out without seeing the input list.

What matters most - The crawl data is unusable for technical SEO analysis. No prioritisation of fixes, no identification of index bloat, thin content, or internal linking issues is possible. The only actionable finding is the need to unblock or reconfigure the crawl to obtain representative data.

Key signals - 1 URL crawled from a site with ~34,000 URLs — the crawl coverage is functionally zero. - Homepage returns 200 and is indexable; no immediate accessibility crisis on the root. - All other on-page metrics (titles, H1s, meta, depth) are zero because no other pages were fetched, not because they are clean. - Potential featured finding: The site’s crawlability is either deliberately blocked or the crawl setup failed to reach internal pages. This single issue obscures every other technical SEO factor and must be resolved before any meaningful analysis can proceed.