DevTools Logo

List & CSV Deduplicator

List & CSV Deduplicator

Remove duplicate entries from lists and CSV files by column — order-preserving, entirely in your browser

Input type

Options

Duplicate handling

Examples

Keyword list cleanup

Input
apple\nbanana\napple\n  cherry\nCherry
Output
3 unique: apple, banana, cherry · removed: apple ×1, cherry ×1

With trim + case-insensitive on; first-seen casing wins the output.

Customer CSV by email

Input
5,000 rows, key column: email
Output
e.g. 5,000 → 4,812 unique · 188 dropped

First row per email survives with all other columns intact; quoting round-trips.

About this tool

Two deduplication modes share one contract: keep the first occurrence, preserve original order, and report everything removed. Plain-list mode compares lines with optional trimming and case folding; CSV mode parses with full RFC 4180 quoting and dedupes by any column you choose.

Everything runs on a Set of normalized keys — linear time, comfortable with hundreds of thousands of rows — and nothing leaves the browser.

This tool stays focused on duplicate removal with a removal report — it keeps the first occurrence of each entry and leaves everything else about your list untouched.

How to use this tool

  1. Paste your data

    Plain list (one entry per line) or CSV with a header row.

  2. Choose options

    Trim, case-insensitive comparison, empty-line skipping; in CSV mode pick the key column.

  3. Review the report

    Summary counts plus a removed-duplicates list with repeat counts.

  4. Export

    Copy the clean output or download as .txt / .csv.

Use cases

Exported contacts

Collapse repeated email addresses while keeping the earliest row per contact.

ID list hygiene

Clean pasted user IDs or keywords from multiple log excerpts.

Modes

ModeDeduplicates by
Plain listWhole line (with optional trim / case folding / empty skip)
CSVAny single column you pick, with full quoting support

Common mistakes

Mistake:Deduping CSV on the whole row

Fix:Pick the key column — rows often differ in timestamps while being the same record.

Mistake:Expecting 'keep newest' behaviour

Fix:First occurrence wins by design; sort by date before pasting to keep the newest.

Frequently asked questions

Which duplicate is kept?

Always the first occurrence, in your original order. That makes the output predictable: dedupe a sorted-by-date CSV and the earliest row per key survives.

How do I dedupe a CSV by a specific column?

Switch to CSV mode, paste your file with its header row, then pick the key column — email, SKU, ID, anything. Rows repeating that value are dropped while all other columns stay untouched.

Is my data uploaded anywhere?

No. Parsing and deduplication run entirely in your browser with JavaScript. Nothing is sent to a server, which makes it safe for customer lists and other sensitive exports.

How big a list can it handle?

Hundreds of thousands of lines work comfortably; the practical limit is your browser's memory and paste buffer. For very large CSVs, use the download button instead of copy.

References & standards