How to remove duplicate lines
- Paste your list into the box above, with one item per line.
- Leave Trim spaces ticked unless trailing whitespace is meaningful in your data.
- Tick Ignore capitalisation if Apple and apple should count as the same entry.
- Read the message to see how many duplicates were removed.
- Copy the result, or download it as a text file.
What the options do
- Case folding, so mixed-capitalisation entries collapse into one.
- Whitespace trimming, which catches the invisible duplicates that break naive tools.
- A choice of keeping the first or the last occurrence, which matters for time-ordered data.
- Optional blank line removal in the same pass.
- Optional natural sorting, so item2 lands before item10.
- A count of exactly how many duplicates were removed, so you can sanity-check the result.
Duplicates are almost never harmless
A repeated line in a list is rarely just untidy. Send an email campaign to a list with duplicates and some people receive it twice, which raises complaint rates and can affect deliverability. Import a product feed with duplicate codes and the database either rejects the batch or creates ghost records. Count keywords with repeats and every percentage you calculate is wrong.
Duplicates accumulate naturally whenever data comes from more than one place, two exports merged, a spreadsheet copied twice, a form submitted on a double click. Removing them is usually the first step before anything else can be trusted.
Why trimming matters more than it sounds
The most frustrating duplicates are the ones you cannot see. john@example.com and john@example.com with a trailing space look identical on screen and are different strings to any computer. Data exported from spreadsheets and pasted from web pages is full of them.
That is why trimming is on by default here. Turn it off only when whitespace genuinely carries meaning, indented code, for example, where leading spaces are structural. For everything else, leaving it on prevents the tool from reporting "no duplicates found" on a list that visibly contains them.
Case folding is a judgement call
- Turn it on for email addresses, which are case-insensitive in practice, and for names and keyword lists.
- Leave it off for passwords, API keys, case-sensitive identifiers and any code where capitalisation carries meaning.
Note that with folding enabled, the surviving line keeps its original capitalisation. The comparison ignores case, the output does not change it. If you want consistent capitalisation as well, run the result through the case converter.
First or last?
For a plain list of values it makes no difference. It matters when the lines carry an implicit order. In a chronological log the last occurrence is the most recent state, so keeping the last is right. In a priority-ordered list the first entry is the one that should win.
The tool preserves the original order of whatever survives, so the result reads the same way the input did, minus the repeats. Turn on sorting only when you want alphabetical or numerical order instead, that uses natural comparison, so item2 comes before item10 rather than after it.
Related cleanups
Duplicates are one of several problems in messy list data. Blank lines can be dropped here in the same pass; inconsistent spacing is handled by the whitespace remover; and if the list needs ordering as well, the text sorter gives you more control over how. For repeated words inside a sentence rather than repeated lines, use the duplicate word remover. All of it runs in your browser, so customer lists and internal data never leave your machine.