---
title: Remove Duplicates
description: Drop repeated items from a collection, optionally across runs
---

Drop repeated items from a list, and optionally skip items you already handled on an earlier run.

## What it does

Remove Duplicates looks at the whole incoming list and keeps the **first** item of each kind, dropping the later copies. The items that remain stay in their original order, so you can [Sort](/nodes/builtin/datatransformation/sort/) first to choose which copy wins.

You decide what "the same" means with **Compare**. The order of keys inside an object does not matter: `{ "a": 1, "b": 2 }` and `{ "b": 2, "a": 1 }` are duplicates.

With **Also Compare Against Previous Runs** on, the node also remembers what it let through before and drops those items on later runs. That turns any repeated scrape into a "new items only" feed.

## When to use it

* Clean a scraped list that shows the same row in several places on the page.
* Treat two records as the same even when a changing field, such as a timestamp, differs.
* Get notified only about new listings, posts or jobs, by turning on **Also Compare Against Previous Runs**.
* To see what changed between two lists (new, gone, edited), use [Compare Datasets](/nodes/builtin/flow/comparedatasets/) instead.

## Inputs and settings

* **Compare**: *All fields* compares the whole item. *Selected fields* compares only the names in **Fields**, and dots reach nested fields (`seller.id`). *All fields except* compares everything but the names in **Fields**; it can only leave out top-level fields, so `seller.id` would not work there.
* **Remember** and **History Size** only matter when **Also Compare Against Previous Runs** is on. With *Last N items*, **History Size** is a number of items (1000 by default, at most 2000). With *Time window*, it is a number of days (at most 365). Larger values are lowered to the maximum instead of failing.
* Each Remove Duplicates node keeps its own history, inside the workflow's stored data.

## Outputs

The same list with the copies removed, in the original order. Items are not changed. If every item was a duplicate, the output is an empty list.

## Dependencies and credentials

None. This node needs no connection and has no slots for other nodes.

## Example workflow

1. A [Schedule](/nodes/builtin/trigger/schedule/) trigger runs every hour.
2. [Get HTML from Link](/nodes/builtin/core/gethtmlfromlink/) and [HTML Extract](/nodes/builtin/datatransformation/html-extract/) read the listings on a page, each with a `url` and a `title`.
3. Remove Duplicates with **Compare** *Selected fields*, **Fields** `url`, and **Also Compare Against Previous Runs** on.
4. A [Browser Notification](/nodes/extension/browsernotification/) shows `{{ $input.title }}` for each listing that comes through.

The first run lets every listing through. After that, you only hear about listings you have not seen before.

## Common issues

* **"Compare mode "…" needs at least one field. Add a field, or switch to "All fields"."** You chose *Selected fields* or *All fields except* without filling **Fields**.
* **"Cannot compare these items: one of them contains a circular reference or a value that cannot be serialized."** An item holds something that cannot be written as plain data. Use *Selected fields* and compare on simple fields such as `id` or `url`.
* **Nothing is removed.** The items differ in a field you did not expect, such as a timestamp or a tracking parameter in a link. Compare only the fields that identify an item, or leave out the changing ones with *All fields except*.
* **Items you already saw come back on every run.** Remembering across runs only works when the node runs as part of a saved workflow; otherwise the run log warns "nothing was remembered for next time". Save the workflow and run it again.
* **You want old items to come through again.** Open the **Stored data** tab in the canvas side panel and delete this node's history entry (it starts with `remove-duplicates-history`).

## Related nodes

* [Sort](/nodes/builtin/datatransformation/sort/): choose which copy is kept.
* [Compare Datasets](/nodes/builtin/flow/comparedatasets/): sort two lists into new, gone and changed items.
* [Data Store](/nodes/builtin/core/datastore/): keep your own values between runs.
* [Limit](/nodes/builtin/datatransformation/limit/): keep only the first or last few items.