Skip to main content

Algolia Crawler

The WF-Algolia sync mirrors Webflow CMS collections into Algolia. Anything that is not a CMS item never reaches Algolia through it — and on most sites that is a lot of content:

  • Static pages built in the Designer
  • Landing and marketing pages
  • Help or policy pages written as fixed page content
  • Linked PDFs and other documents

The Algolia Crawler is Algolia's own tool for that content. You give it a starting URL; it fetches each page, extracts the parts you tell it to keep, follows links to find more pages, and writes the result into an index. It can run on a schedule.

Which one do I use?​

Both, usually. They cover different content and neither replaces the other.

WF-Algolia syncAlgolia Crawler
IndexesWebflow CMS itemsAny published page
ReadsStructured CMS fieldsRendered HTML
UpdatesAs items changeNext time it crawls
Configured inThis appThe Algolia dashboard

Use the sync for CMS collections: it reads fields directly, so it is faster, more precise, and reflects edits as they happen. Use the Crawler for everything else.

Give the Crawler its own index

Point the Crawler at an index no collection mapping writes to. WF-Algolia treats a mapped index as its own — a reindex clears out records the Crawler put there, and the two sources will otherwise fight over records with matching object IDs.

To search both at once, keep them in separate indices and use a federated dropdown, or give each element its own index with wf-algolia-index.

Set one up​

The Crawler is configured entirely in Algolia — there is nothing to switch on in this app.

  1. Open Crawler in the Algolia dashboard. The WF-Algolia app links straight there, above the Synced Collections table and from the Algolia account section of Settings.
  2. Create a crawler and give it your published site URL as the starting point.
  3. Point it at an index no collection mapping uses.
  4. Set the extraction rules — which parts of the page become which record fields.
  5. Run it, then check the records landed in Search → Index.

Its availability depends on your Algolia plan. Algolia's own Crawler documentation covers extraction rules and scheduling in full.

Searching crawled content​

Crawled records are ordinary Algolia records, so every WF-Algolia component works against them. Point an element at the crawler's index:

<div wf-algolia-index="site_pages">

The field names in your wf-algolia-text bindings must match the field names your extraction rules produce — the same rule that applies to CMS records. Open one crawled record in the dashboard and copy the names exactly.

Next​