Psychograph
DocsPsychograph MCP

Our ingest crawler

When you submit your website URL, our ingest crawler reads your site to build the first draft of your structured business profile. This document describes exactly what the crawler does, how to control its access, and what it extracts.

User-agent string

All requests from our ingest crawler identify themselves with this user-agent:

PsychographMCP-Ingest/1.0 (+https://mcp.psychogra.ph/docs/crawler)

The URL in the parenthetical is this page — any web operator who sees this user-agent in their logs and wants to know what it is will find the explanation here.

When the crawler runs

The crawler runs once: when you submit your website URL to begin onboarding. It does not run on a recurring schedule. If you want us to re-ingest your site to pick up major changes (new service pages, a website rebuild), contact us from your dashboard and we will trigger a manual re-crawl.

Updates to your profile after the initial crawl are made through your dashboard — not by re-crawling. You can add, edit, or remove any field without waiting for another crawl.

Crawl strategy

The crawler uses a sitemap-first, same-origin BFS strategy:

  1. Sitemap discovery — it first checks yourdomain.com/sitemap.xml. If a sitemap exists, all URLs listed in it are added to the crawl queue. This is the most reliable way to ensure we find your service pages even if they’re not linked prominently from the homepage.
  2. BFS same-origin link following — the crawler then follows links found on each page, staying within your domain. It does not follow links to external sites.
  3. Page cap: 25 pages — the crawler stops after 25 pages, regardless of how many are queued. For most local service businesses this covers every service page, the homepage, the about page, and the contact page. If you have a larger site, ensure your most important service pages are listed in your sitemap.

Request behavior

  • Fetch timeout: 10 seconds per page. Pages that don’t respond in time are skipped.
  • Redirects: followed automatically.
  • Content-type filter: only text/html pages are processed. PDFs, images, and other file types are skipped even if they appear in the sitemap.
  • Query string handling: query parameters are stripped before queuing a URL. example.com/services?ref=google and example.com/services are treated as the same page — only one is crawled.
  • Fragments: stripped (#anchors are ignored).
  • Duplicate detection: each normalized URL is only visited once.

File types we skip

URLs ending in these extensions are never fetched:

.pdf  .jpg  .jpeg  .png  .gif  .svg  .webp  .zip
.mp4  .mp3  .css  .js  .ico  .xml

What we extract from each page

From each HTML page, the crawler extracts:

  • Title — from the <title> tag, or the first <h1> if no title tag is present
  • Headings — all h1, h2, h3 elements (up to 200 characters each)
  • Body text — the visible text of the page after stripping scripts, styles, noscript blocks, SVGs, and iframes; up to 20,000 characters per page
  • Phone numbers — up to 5 phone numbers detected via pattern matching
  • Email addresses — up to 5 emails found in the body text

We do not extract images, parse JavaScript-rendered content, execute page scripts, or interact with forms. The crawler is a pure HTML reader — what’s in the HTML source is what we get.

If your website relies heavily on JavaScript to render content (single-page apps, heavy React or Vue frontends), some content may not be captured because it’s not present in the initial HTML response. In this case, your profile will be thinner from the crawl and you’ll fill in more fields manually during the review step. This is common and expected — the manual review step exists precisely for this reason.

Controlling crawler access

To block the crawler from specific pages

Use the standard robots.txt Disallow directive targeting our user-agent:

User-agent: PsychographMCP-Ingest
Disallow: /admin/
Disallow: /internal/
Disallow: /private-page

The crawler respects robots.txt Disallow rules. Pages blocked by your robots.txt are not crawled, and their content is not included in your profile draft.

To allow us after a previous block

If you previously blocked our user-agent and want to allow the crawler, remove the Disallow rules and contact us to trigger a re-crawl from your dashboard.

To block all crawlers except ours

User-agent: *
Disallow: /

User-agent: PsychographMCP-Ingest
Allow: /

Privacy and data handling

Crawled content is used exclusively to build your business profile. It is not shared with third parties, not used to train AI models, and not indexed in any public directory. The extracted text is stored in your account and is subject to the same data handling as the rest of your profile data.

Anything the crawler captures but you choose not to include in your published profile is never exposed to agents. You review and approve every field before it goes live. The crawler output is a first draft, not an automatic publication.