Zum Inhalt springen
Agentic Workflowdocs
v0.8.2Kostenlos installieren

rezepte / extract data

Turn a list on a web page into structured data

Fetch a page by its address and pull each item's title and link into clean data you can reuse, without opening the page.

Dieser Inhalt ist noch nicht in deiner Sprache verfügbar.

Mittelstufe 7 Min. Keine Einrichtung Daten extrahieren

Workflow · web-scrape-to-json.awf 4 Schritte

Web Scrape to JSON: ein Workflow mit 4 Schritten, beginnend mit Run now und endend mit Show the data.

  1. Run now(When Started – zur Dokumentationsseite)
  2. Fetch the page(Get HTML from Link – zur Dokumentationsseite)
  3. Pull out the items(HTML Extract – zur Dokumentationsseite)
  4. Show the data(Display Object – zur Dokumentationsseite)
In AWFlow öffnen
Run it → fetch the page → pull out each item → show the data.

How it works

Section titled “How it works”

  1. Run it

    When Started Knoten-Referenz

    When Started has no settings. Click Test Workflow on the canvas.

    Passen Sie es an — run it every morning with a Schedule trigger.

    Step 1, close-up of the app: Run now (When Started), outlined.Step 1, close-up of the app: Run now (When Started), outlined.Vollbild
    Run now (When Started)
    The workflow editor with the Web Scrape to JSON recipe: sticky notes and the steps Run now, Fetch the page, Pull out the items, Show the data, linked left to right. The top bar shows the Workflow active switch and Save; Test Workflow is above the canvas.The workflow editor with the Web Scrape to JSON recipe: sticky notes and the steps Run now, Fetch the page, Pull out the items, Show the data, linked left to right. The top bar shows the Workflow active switch and Save; Test Workflow is above the canvas.

  2. Fetch the page

    Get HTML from Link Knoten-Referenz

    Get HTML from Link downloads https://news.ycombinator.com and puts its HTML in html. The page doesn't need to be open.

    Passen Sie es an — change URL to the page you want. Pages that need you to be signed in may return a login page instead.

    Step 2, close-up of the app: The URL field, outlined.Step 2, close-up of the app: The URL field, outlined.Vollbild
    The URL field
    The settings of the Fetch the page step: the URL field, Output field name, Timeout and Options.The settings of the Fetch the page step: the URL field, Output field name, Timeout and Options.

  3. Pull out the items

    HTML Extract Knoten-Referenz

    HTML Extract in list mode finds every .titleline and, inside each, reads the link text as title and its href as link.

    Passen Sie es an — set Container Selector to one repeated item on your page and add a field per value you want (price, date, author).

    Step 3, close-up of the app: The Container Selector field, outlined.Step 3, close-up of the app: The Container Selector field, outlined.Vollbild
    The Container Selector field
    The settings of the Pull out the items step: HTML, Mode Repeating list, Selector, Return, Attribute, Container Selector .titleline, and Fields with a title field read from the a element.The settings of the Pull out the items step: HTML, Mode Repeating list, Selector, Return, Attribute, Container Selector .titleline, and Fields with a title field read from the a element.

  4. Show the data

    Display Object Knoten-Referenz

    Display Object shows the items as a tree you can expand.

    Passen Sie es an — save them with Google Sheets or download them with Save as File.

    Step 4, close-up of the app: Show the data, outlined.Step 4, close-up of the app: Show the data, outlined.Vollbild
    Show the data
    The workflow editor with the Web Scrape to JSON recipe: sticky notes and the steps Run now, Fetch the page, Pull out the items, Show the data, linked left to right. The top bar shows the Workflow active switch and Save; Test Workflow is above the canvas.The workflow editor with the Web Scrape to JSON recipe: sticky notes and the steps Run now, Fetch the page, Pull out the items, Show the data, linked left to right. The top bar shows the Workflow active switch and Save; Test Workflow is above the canvas.

Test it

Section titled “Test it”

  1. Install the recipe and open it on the canvas. To import the file yourself, use Add Workflow › Import, choose the .awf and click Create & open. Then click Make it mine at the bottom of the canvas, tick the box and confirm, so you can edit it.

    Step 1, close-up of the app: Create & open, outlined.Step 1, close-up of the app: Create & open, outlined.Vollbild
    Create & open
    The Create a workflow dialog at step 3, Name & save, after importing web-scrape-to-json.awf: Source Imported .awf, Name “Web Scrape to JSON”, an optional Description, Where to save “Saved locally in this browser”, and the Back and Create & open buttons.The Create a workflow dialog at step 3, Name & save, after importing web-scrape-to-json.awf: Source Imported .awf, Name “Web Scrape to JSON”, an optional Description, Where to save “Saved locally in this browser”, and the Back and Create & open buttons.
    Step 1, close-up of the app: Make it mine, outlined.Step 1, close-up of the app: Make it mine, outlined.Vollbild
    Make it mine
    The Make this workflow yours? dialog over the Web Scrape to JSON workflow: it came from the Marketplace, It can currently lists what the workflow may access, two notes on what changes, the “I'm responsible for what this workflow does from now on.” checkbox, and Cancel and Make it mine.The Make this workflow yours? dialog over the Web Scrape to JSON workflow: it came from the Marketplace, It can currently lists what the workflow may access, two notes on what changes, the “I'm responsible for what this workflow does from now on.” checkbox, and Cancel and Make it mine.

  2. Click Test Workflow.

    Step 2, close-up of the app: Test Workflow, outlined.Step 2, close-up of the app: Test Workflow, outlined.Vollbild
    Test Workflow
    The workflow editor with the Web Scrape to JSON recipe: sticky notes and the steps Run now, Fetch the page, Pull out the items, Show the data, linked left to right. The top bar shows the Workflow active switch and Save; Test Workflow is above the canvas.The workflow editor with the Web Scrape to JSON recipe: sticky notes and the steps Run now, Fetch the page, Pull out the items, Show the data, linked left to right. The top bar shows the Workflow active switch and Save; Test Workflow is above the canvas.

  3. Open Show the data in the run results: you see about 30 items, each with a title and a link from the Hacker News front page.

Variations

Section titled “Variations”
Variation What to change Level
Scrape the page you’re on Replace Get HTML from Link with Get All HTML, which also works on pages you’re signed in to. Beginner
Keep only new items Store the last titles with Data Store and use Remove Duplicates. Advanced
Get a CSV Add CSV Build and Clipboard Write. Beginner

Troubleshoot

Section titled “Troubleshoot”
  • The list is empty. The site renders its list with JavaScript, so the fetched HTML has no items. Open the page and use Get All HTML instead.
  • Links are relative (/item?id=…). The site writes them that way. Prefix them with the site address in an Edit Fields step.
  • The fetch fails in the web app. Some sites block requests from other origins; run it in the extension. See browser compatibility.
Aria fragen