Internal Link Crawl Explorer — Map Reachable Pages and Links

An internal link explorer maps hyperlinks that connect HTML pages on the same hostname, starting from a URL you select. Enter a permitted public page, choose a path scope, and set a strict limit of four, eight, or twelve attempted pages. The crawler retrieves pages sequentially through the existing restricted PHP inspection endpoint, records which responses were successfully inspected, and follows same-host HTTP(S) anchors found in those documents. Its report separates discovered links from fetched pages and lists failures instead of silently treating blocked or timed-out requests as successful. Because the crawl is deliberately bounded and does not execute JavaScript, it is a focused navigation map rather than a complete site audit.

Internal Link Explorer At a Glance

How Internal Links Form a Traversable Site Map

A website can be represented as a directed graph in which pages are nodes and hyperlinks are edges. If a page A links to page B, the crawler can discover B from A even when B never appears in a sitemap. The explorer starts at a selected page, extracts same-host destinations from its HTML, and puts previously unseen destinations into a queue. It then visits queued pages in order until the configured request cap is reached or the queue becomes empty. The report retains edges between source and destination URLs as well as details about the pages successfully retrieved. This graph is based on observed HTML links, not on an assumption that every site page has been discovered.

Choosing a Useful Starting URL

The starting URL determines what can be reached during a limited crawl. A homepage may expose global navigation and therefore produce many candidate URLs immediately, leaving only a few request slots for deeper pages. A category landing page, documentation index, or specific section might be a better starting point when the question concerns that area. Choose a stable publicly accessible URL rather than an authentication route or a page that depends entirely on client-side JavaScript. The crawler does not bypass login requirements, execute forms, or guess hidden links. If the initial page fails verification, the report must show zero successfully inspected pages; it cannot substitute an imagined site structure.

How the Page Limit Changes the Results

A queue limit is not a guarantee that every discovered page will be inspected. With a cap of eight attempts, the crawler might fetch the starting page and seven reachable pages while recording hundreds of additional candidates. Those candidates remain unvisited, and their status, content, and outgoing links are unknown. A failed request also consumes an attempt because the endpoint still expended time and network resources. The report therefore distinguishes pages inspected, requests failed, links discovered, and remaining queued URLs. This distinction prevents a small sample from being misrepresented as a complete crawl. Increase the cap only within the tool’s limits or run separate, carefully scoped analyses for different sections.

Path Scopes and Host Boundaries

By default, destinations must share the starting page’s hostname. A link to a different subdomain is not followed even if the same organization operates it. You can narrow the crawl further with a path prefix such as /docs/ so that discovered URLs outside that subtree are skipped. The prefix is matched against the resolved pathname, not against arbitrary text inside the URL. This makes the scope predictable but means a related page living elsewhere will not appear. Before interpreting an absent link, consider whether the page was outside the chosen scope or simply beyond the request cap. Cross-host crawling would also introduce additional rate, access, and permissions concerns and is deliberately excluded.

Worked Example: Category Navigation with Deeper Pages

Suppose /resources/ links to /resources/tutorials/ and /resources/reference/, and the tutorials page links to three individual lessons. Starting at /resources/ with a page cap of four might retrieve the landing page, both branch pages, and the first queued lesson. The resulting graph would show the other lessons as discovered destinations but not as inspected documents. If the reference page links back to the landing page, the queue should avoid requesting the already seen URL a second time. The export records these relationships so you can distinguish circular navigation from a duplicate-fetch problem. An unvisited lesson is neither proven healthy nor proven broken; its status remains unknown until an actual request succeeds.

Handling Query Strings and Fragments

A fragment such as #installation refers to a location within a page and does not require a separate HTML fetch. The explorer strips fragments when normalizing discovered destinations. Query parameters are different: they can change server-side content, but they can also create nearly unlimited permutations of filter and tracking URLs. By default, the crawler drops query strings from queued destinations to reduce duplicates and unintended load. You can explicitly preserve them when query-dependent pages are the intended subject of the inspection. Preserve mode should be used cautiously because many combinations can appear distinct even when the displayed page is practically identical. Neither option silently guesses canonical URLs for you.

The Role of the Restricted PHP Endpoint

The browser does not have unlimited permission to fetch and read arbitrary third-party pages. Instead, each live HTML request goes through the preexisting PHP site-audit endpoint, which performs public-address validation, DNS pinning, redirect checks, robots.txt access checks, request timing restrictions, and rate limiting. Pages rejected by any safeguard are listed as failed requests, with their error messages. The crawler does not try to bypass a blocked path or treat a network error as an empty page. Those restrictions are a fundamental part of responsible operation on shared hosting rather than incidental problems to be hidden from the user. On a live deployment, the endpoint must be tested against the host’s network and PHP configuration.

Reading the Crawl Graph and Page Summary

The inspected-page table shows the requested URL, the final URL, traversal depth, outgoing internal destination count, and whether an observed robots directive includes noindex. The edge table lists source and destination pairs extracted from successfully returned HTML. A URL may appear in the edge table without appearing in the inspected-page table, because discovery happens before the request cap is exhausted. When the starting page redirects, the tool retains its request information separately from the final content URL. These records support a practical discussion of link routes and how pages connect, but a directed edge alone does not prove that a target can be fetched or indexed.

What the Crawl Does Not Detect

The crawler reads the returned HTML rather than operating as a full browser with script execution. Links inserted by frameworks after hydration, accessible only through interactions, or enclosed in inaccessible iframes will not be discovered. It also does not parse JavaScript route definitions or submit forms. The observed noindex indicator comes from available meta/header data for fetched pages and should not be treated as a complete indexability judgment. Robots access rules, HTTP errors, and canonical selection remain independent concerns. The link attribute checker can inspect additional anchor-level signals on a specific page, including rel and target values, when a graph edge requires further context.

Using Exported Data to Investigate Navigation

The JSON export preserves inspected pages, failures, edges, and the count of still-queued destinations. The CSV export contains successfully inspected page rows for straightforward sorting. Together, these outputs can support questions such as whether a major section is reachable from its landing page, how many links a particular page declares, or where a navigation branch leads within the inspected sample. The reports should include the starting URL, path scope, query handling, page limit, and date of inspection. Without those settings, two crawls could produce different maps for reasons unrelated to website changes. Never infer that a page is orphaned merely because it does not appear in one bounded traversal.

Responsible Interpretation of Missing Pages

A missing page in the report may be outside the scope, discovered too late in the queue, absent from the initial HTML, blocked by robots rules, or hidden behind navigation that requires interaction. Only a successfully requested page can support assertions about its response content. Likewise, a link pointing to an external host is intentionally ignored by the crawler rather than followed across the internet. A meaningful remediation review starts with a reproducible question, checks the request limits and error list, and follows up with targeted inspection where necessary. A low number of traversed pages is not by itself evidence of poor internal linking; it is often the expected consequence of a deliberately conservative crawl limit.

Frequently Asked Questions

Conclusion

A bounded crawl can reveal genuine routes through a site without pretending to map every page. Keep failures and unvisited discoveries visible, then target specific navigation questions for subsequent verification.