URL Document Loader

Load a web page as plain text, ready to index into a knowledge base.

What it does

URL Document Loader downloads the page at URL and turns its HTML into clean plain text: no tags, no image or link addresses, line breaks kept. It puts the text in one output field, named by Output field name (text by default).

It is meant to feed the Indexer, but the output is ordinary text you can send to any node. It reads the page as the server sends it, so content that only appears after scripts run in the browser is missing.

When to use it

• Index a public web page, such as documentation or a pricing page, into a knowledge base.
• Refresh a knowledge base on a schedule with the latest version of a page.
• For the page open in your current tab, including pages behind a login, use Current Page Document Loader.

Inputs and settings

Einstellung	Typ	Was es bewirkt
URL PFLICHT(erforderlich)	Text	The URL to load text from.
Output field name	Text	The name of the output field that will hold the loaded text. Standard: text
Timeout	Number	The timeout for the request in milliseconds. Standard: 10000
Options	Group	The options for the node.
Options › Headers	Key-value pairs	The headers to send with the request.
Options › Include input fields	Toggle	Whether to include the input fields in the output. Standard: Off

• URL must be a full address including https://. It can be an expression, such as {{ $input.url }}, to load one page per incoming item.
• Timeout is in milliseconds: 10000 is 10 seconds.
• Use Headers for pages that need a header, for example Accept-Language: fr.

Outputs

• Output

Each item has the page text in the field you named (default text). With Include input fields on, the incoming fields are kept next to it. Pass it to the Indexer’s Input as {{ $input.text }}.

Dependencies and credentials

None. This node needs no connection and has no slots for other nodes.

Example workflow

1. Schedule runs every day.
2. URL Document Loader with URL https://example.com/pricing and Output field name text.
3. Indexer with Input {{ $input.text }}, Source name Pricing page and Source link https://example.com/pricing. Connect Local Knowledge to Knowledge and Recursive Character Text Splitter to Text Splitter.

Each day the same source is updated in place in the knowledge base.

Common issues

• “HTTP error! status: 404” (or another number). The server refused the page: 404 means not found, 403 means access denied. Check the address in a browser.
• “Invalid URL: …”. The address is malformed, or the page could not be reached at all (offline, or the site blocks the request). Check the address; if it opens fine in a tab, try running the workflow from the browser extension.
• “Request timeout”. The page took longer than Timeout. Raise it, for example to 30000.
• “No content found”. The page has no text in its HTML, often because it is built by scripts. Open it in a tab and use Current Page Document Loader.

Related nodes

• Indexer: save the text into a knowledge base.
• Get All Text from Link: the same page-to-text, when you don’t need to index.
• Current Page Document Loader: load the open tab instead.
• File Document Loader: load a PDF, HTML or text file.