How to use the duplicate line remover
- Paste your list into the left box, one item per line: email addresses, keywords, URLs, names, log entries, anything.
- The result appears on the right straight away, with the first copy of every line kept in its original position.
- Choose how strict the match should be. By default the tool ignores capital letters and extra spaces at the ends of lines, which is what you want for most lists. Tick Case-sensitive if Apple and apple are different items for you.
Use Show to flip the question: list only the lines that repeat, or only the lines that appear exactly once. The counters show how many lines went in, how many came out and how many were dropped.
How duplicates are detected
Each line is turned into a comparison key. If case is ignored, the key is the lowercase version of the line. If spaces are ignored, leading and trailing spaces and tabs are removed first. Two lines are duplicates when their keys are identical. The tool keeps the first spelling it saw, so with Banana followed later by banana, the result keeps Banana.
Only whole lines are compared. The tool does not look inside lines, so "apple pie" and "apple" are different items, and a word that appears in two lines is not a duplicate. To find repeated words inside text, use the word counter's most-used-words list.
Worked example
Press Try sample to load these ten lines, including one blank line and some different capitals and spaces:
apple, Banana, apple, cherry, banana, " cherry ", (blank), Date, (blank), apple
- Default settings (ignore case, ignore edge spaces, remove blanks): the result is apple, Banana, cherry, Date, which is 4 lines. Six lines were dropped: two blanks and four duplicates (two more apple, banana and cherry).
- Case-sensitive on: Banana and banana now count as different, so the result has 5 lines: apple, Banana, cherry, banana, Date.
- Only duplicated lines: apple, Banana, cherry, each shown once.
- Only lines that appear once: Date.
Common uses
| Job | Setting |
|---|---|
| Clean an email or mailing list | Defaults, then copy the result |
| Dedupe a keyword list from several sources | Defaults, with sorted A to Z |
| Find which URLs in a list are repeated | Show: only the duplicated lines |
| Find entries that exist in only one file (after merging two lists) | Show: only the lines that appear once |
| Compare lines where capitalisation matters, such as codes | Tick case-sensitive |
For comparing two lists, paste both into the left box one after the other. Lines that appear once exist in only one list, assuming neither list has duplicates of its own. Lines shown by the duplicates option are in both.
Why duplicate lines creep into lists
- Merging sources. Combining two exports, such as email lists from a shop and a newsletter, nearly always creates overlaps.
- Copying with slight differences. A trailing space, a different capital letter or a stray tab makes two identical-looking lines different to a computer. The default settings ignore those differences.
- Repeated log or error messages. A program that logs the same warning thousands of times is easier to read once the repeats are removed.
- Keyword research. Tools often return the same phrase under several groups, and duplicates inflate counts and waste time when you review the list by hand.
A note on privacy and large lists
Lists often contain names, email addresses or customer data, so it matters where they are processed. The comparison runs entirely in your browser tab, using a hash lookup, which means the time it takes grows in step with the length of your list and not with its square. A list of 100,000 lines is handled in a fraction of a second on an ordinary laptop. Nothing is uploaded, and closing the tab removes the data from memory.
If you need to keep a record of what was removed, run the tool twice: once with the default setting to get the clean list, and once with Only the lines that are duplicated to see which items repeated. Comparing the two outputs tells you exactly what changed.
Accuracy and limits
Lines are compared exactly after the case and space settings are applied. Different Unicode forms of the same letter, such as an accent typed as one character and as two, count as different unless your device normalises them. Windows (CRLF), Mac and Linux line endings are all understood, and the output uses plain line breaks. Lists of a few hundred thousand lines work on a typical computer; very large files are better handled in a spreadsheet or command-line tool.
Frequently asked questions
Does it keep the original order?
Yes, by default. The first copy of every line stays where it was, and later copies are dropped. You can also sort the result A to Z or Z to A.
Is the comparison case-sensitive?
Not by default. “Apple” and “apple” count as the same line. Tick the case-sensitive box if they should be treated as different.
Can it remove duplicates in Excel or Google Sheets?
Those programs have their own Remove Duplicates command for columns. This tool is for plain text lists, and you can paste a single spreadsheet column into it and copy the result back.
How do I find the lines that appear in both of two lists?
Paste both lists one after the other and choose “Only the lines that are duplicated”. Lines found in both lists appear in the result. This assumes each list has no repeats of its own; remove those first with the default setting.
What happens to empty lines?
By default they are removed. Untick the option to keep them, in which case all blank lines count as the same line and only the first is kept.