How to Find Duplicate Lines
in a Text File Using Windows Tools

How to Check a Text File for Duplicate Lines in Windows

The easiest reusable method is the BAT script in Method 1. Drag a TXT file onto it to create both a duplicate report and a cleaned copy without repeated lines. The original file is not modified. For a quick console-only check, use the following PowerShell command:

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Sort-Object Count -Descending |
Select-Object Count, Name

Replace input.txt with the actual file name. This command only reports duplicates; it does not create a cleaned file. The -Encoding UTF8 option is important in Windows PowerShell 5.1 because a UTF-8 file without a byte order mark may otherwise be decoded as the Windows ANSI code page.

What Counts as a Duplicate Line in a Windows Text File?

A duplicate line is a line whose comparison value appears more than once in the same file. However, the result depends on the comparison rules you choose.

For a strict check, use a case-sensitive comparison and do not trim whitespace. For lists of names, URLs, domains, or identifiers, you may prefer to ignore letter case and accidental spaces at the beginning or end of a line.

Find Duplicate Lines and Create a Clean File with a BAT Script

For repeated checks, use the following find_duplicate_lines.bat script. Drag a text file onto it or enter the full path. The script creates two UTF-8 files in the same folder:

• filename_duplicates.txt — a report containing every repeated line and its total occurrence count.
• filename_without_duplicates.txt — a cleaned copy in which only the first occurrence of each line is kept.

Create find_duplicate_lines.bat

1. Open Notepad.
2. Copy the complete script shown below.
3. Select File → Save As.
4. Enter find_duplicate_lines.bat in the file name field.
5. Select All files as the file type and save the BAT file.

@echo off
setlocal EnableExtensions DisableDelayedExpansion
title Find and Remove Duplicate Lines in a Text File

set "INPUT_FILE=%~1"

if not defined INPUT_FILE (
  echo Drag a text file onto this BAT file, or enter its full path below.
  echo.
  set /p "INPUT_FILE=Text file path: "
)

if not defined INPUT_FILE (
  echo.
  echo No file was selected.
  pause
  exit /b 1
)

for %%I in ("%INPUT_FILE%") do set "INPUT_FILE=%%~fI"

if not exist "%INPUT_FILE%" (
  echo.
  echo File not found:
  echo %INPUT_FILE%
  pause
  exit /b 1
)

for %%I in ("%INPUT_FILE%") do (
  set "DUPLICATE_FILE=%%~dpnI_duplicates.txt"
  set "CLEAN_FILE=%%~dpnI_without_duplicates.txt"
)

echo.
echo Processing:
echo %INPUT_FILE%
echo.

set "FD_SCRIPT_FILE=%~f0"
set "FD_INPUT_FILE=%INPUT_FILE%"
set "FD_DUPLICATE_FILE=%DUPLICATE_FILE%"
set "FD_CLEAN_FILE=%CLEAN_FILE%"

powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "$content=[IO.File]::ReadAllText($env:FD_SCRIPT_FILE); $marker=':'+'POWERSHELL'; $code=$content.Substring($content.IndexOf($marker)+$marker.Length); & ([ScriptBlock]::Create($code)) -Path $env:FD_INPUT_FILE -DuplicatePath $env:FD_DUPLICATE_FILE -CleanPath $env:FD_CLEAN_FILE"

set "RESULT=%ERRORLEVEL%"
echo.

if "%RESULT%"=="0" (
  echo Finished. Two files were created:
  echo Duplicate report:
  echo %DUPLICATE_FILE%
  echo.
  echo File without duplicate lines:
  echo %CLEAN_FILE%
) else if "%RESULT%"=="2" (
  echo Finished. No exact duplicate lines were found.
  echo The cleaned copy was still created:
  echo %CLEAN_FILE%
) else (
  echo The file could not be processed.
)

echo.
pause
exit /b %RESULT%

:POWERSHELL
param(
    [Parameter(Mandatory = $true)]
    [string]$Path,

    [Parameter(Mandatory = $true)]
    [string]$DuplicatePath,

    [Parameter(Mandatory = $true)]
    [string]$CleanPath
)

$ErrorActionPreference = 'Stop'

function Get-TextEncodingInfo {
    param([string]$FilePath)

    $stream = [System.IO.File]::OpenRead($FilePath)

    try {
        $bom = New-Object byte[] 4
        $read = $stream.Read($bom, 0, 4)
    }
    finally {
        $stream.Dispose()
    }

    if ($read -ge 4 -and
        $bom[0] -eq 0x00 -and $bom[1] -eq 0x00 -and
        $bom[2] -eq 0xFE -and $bom[3] -eq 0xFF) {
        return [PSCustomObject]@{
            Encoding = [System.Text.Encoding]::GetEncoding(12001)
            AllowAnsiFallback = $false
        }
    }

    if ($read -ge 4 -and
        $bom[0] -eq 0xFF -and $bom[1] -eq 0xFE -and
        $bom[2] -eq 0x00 -and $bom[3] -eq 0x00) {
        return [PSCustomObject]@{
            Encoding = [System.Text.Encoding]::UTF32
            AllowAnsiFallback = $false
        }
    }

    if ($read -ge 3 -and
        $bom[0] -eq 0xEF -and $bom[1] -eq 0xBB -and
        $bom[2] -eq 0xBF) {
        return [PSCustomObject]@{
            Encoding = New-Object System.Text.UTF8Encoding($true)
            AllowAnsiFallback = $false
        }
    }

    if ($read -ge 2 -and $bom[0] -eq 0xFF -and $bom[1] -eq 0xFE) {
        return [PSCustomObject]@{
            Encoding = [System.Text.Encoding]::Unicode
            AllowAnsiFallback = $false
        }
    }

    if ($read -ge 2 -and $bom[0] -eq 0xFE -and $bom[1] -eq 0xFF) {
        return [PSCustomObject]@{
            Encoding = [System.Text.Encoding]::BigEndianUnicode
            AllowAnsiFallback = $false
        }
    }

    return [PSCustomObject]@{
        Encoding = New-Object System.Text.UTF8Encoding($false, $true)
        AllowAnsiFallback = $true
    }
}

function Read-And-CleanTextFile {
    param(
        [string]$FilePath,
        [System.Text.Encoding]$Encoding,
        [string]$TemporaryCleanPath
    )

    $comparer = [System.StringComparer]::Ordinal
    $counts = New-Object 'System.Collections.Generic.Dictionary[string,int]' ($comparer)
    $seen = New-Object 'System.Collections.Generic.HashSet[string]' ($comparer)
    $utf8WithBom = New-Object System.Text.UTF8Encoding($true)
    $reader = New-Object System.IO.StreamReader($FilePath, $Encoding, $true)
    $writer = New-Object System.IO.StreamWriter($TemporaryCleanPath, $false, $utf8WithBom)
    $totalLines = 0

    try {
        while (($line = $reader.ReadLine()) -ne $null) {
            $totalLines++

            if ($seen.Add($line)) {
                $writer.WriteLine($line)
            }

            $count = 0

            if ($counts.TryGetValue($line, [ref]$count)) {
                $counts[$line] = $count + 1
            }
            else {
                $counts.Add($line, 1)
            }
        }
    }
    finally {
        $reader.Dispose()
        $writer.Dispose()
    }

    return [PSCustomObject]@{
        Counts = $counts
        TotalLines = $totalLines
        UniqueLines = $seen.Count
    }
}

$tempCleanPath = Join-Path ([System.IO.Path]::GetDirectoryName($CleanPath)) ([System.IO.Path]::GetRandomFileName())

try {
    $encodingInfo = Get-TextEncodingInfo -FilePath $Path

    try {
        $data = Read-And-CleanTextFile -FilePath $Path -Encoding $encodingInfo.Encoding -TemporaryCleanPath $tempCleanPath
    }
    catch [System.Text.DecoderFallbackException] {
        if (-not $encodingInfo.AllowAnsiFallback) {
            throw
        }

        if (Test-Path -LiteralPath $tempCleanPath) {
            Remove-Item -LiteralPath $tempCleanPath -Force
        }

        $data = Read-And-CleanTextFile -FilePath $Path -Encoding ([System.Text.Encoding]::Default) -TemporaryCleanPath $tempCleanPath
    }

    if (Test-Path -LiteralPath $CleanPath) {
        Remove-Item -LiteralPath $CleanPath -Force
    }

    Move-Item -LiteralPath $tempCleanPath -Destination $CleanPath

    $duplicates = @(
        $data.Counts.GetEnumerator() |
        Where-Object { $_.Value -gt 1 } |
        Sort-Object Key
    )

    if ($duplicates.Count -eq 0) {
        if (Test-Path -LiteralPath $DuplicatePath) {
            Remove-Item -LiteralPath $DuplicatePath -Force
        }

        Write-Host ("Lines read: {0}" -f $data.TotalLines)
        Write-Host ("Unique lines written: {0}" -f $data.UniqueLines)
        exit 2
    }

    $report = New-Object System.Collections.Generic.List[string]
    $report.Add("Count`tLine")

    foreach ($duplicate in $duplicates) {
        $report.Add(("{0}`t{1}" -f $duplicate.Value, $duplicate.Key))
    }

    $utf8WithBom = New-Object System.Text.UTF8Encoding($true)
    [System.IO.File]::WriteAllLines($DuplicatePath, $report, $utf8WithBom)

    Write-Host ("Lines read: {0}" -f $data.TotalLines)
    Write-Host ("Unique lines written: {0}" -f $data.UniqueLines)
    Write-Host ("Duplicate groups found: {0}" -f $duplicates.Count)
    exit 0
}
catch {
    if (Test-Path -LiteralPath $tempCleanPath) {
        Remove-Item -LiteralPath $tempCleanPath -Force
    }

    Write-Error $_.Exception.Message
    exit 1
}

How to Run the BAT Script

1. Drag the TXT file onto find_duplicate_lines.bat. Alternatively, double-click the BAT file and paste the full path to the text file.
2. Wait while the script reads the file and counts exact line matches.
3. Open filename_duplicates.txt to see repeated lines and their occurrence counts.
4. Open filename_without_duplicates.txt to get the cleaned list with duplicate occurrences removed.

What the Script Does with Line Order

Duplicate Report Example

Count	Line
4	example.com
2	server-01

Comparison Rules Used by the Script

• Comparison is case-sensitive: Windows and windows are treated as different lines.
• Leading and trailing spaces are significant.
• Blank lines are treated like any other line and are reduced to a single blank line in the cleaned output.
• UTF-8 with or without BOM is detected automatically. UTF-16 and UTF-32 are supported when a BOM is present.
• A non-UTF-8 file without BOM falls back to the current Windows ANSI code page when its bytes are not valid UTF-8.
• Both generated files are written as UTF-8 with BOM for reliable display in Windows applications.

How to Show Duplicate Lines and Their Counts with PowerShell

PowerShell reads the file as a collection of lines. -Encoding UTF8 tells Windows PowerShell 5.1 to decode a UTF-8 file correctly, including UTF-8 files without a byte order mark. Group-Object groups equal lines, and Where-Object keeps only groups that contain more than one item.

1. Place the text file in an easy-to-find folder.
2. Open that folder in File Explorer.
3. Click the address bar, type powershell, and press Enter.
4. Run the command below after replacing input.txt with your file name.

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Sort-Object Count -Descending |
Select-Object Count, Name

Example Result

Count Name
----- ----
    4 example.com
    3 192.168.1.10
    2 Windows 11

This result means that example.com appears four times, 192.168.1.10 appears three times, and Windows 11 appears twice.

Use a Full File Path

You can run the same check from any folder by specifying the full path:

Get-Content -LiteralPath "C:\Users\User\Desktop\input.txt" -Encoding UTF8 |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Select-Object Count, Name

-LiteralPath is preferable when a file name contains characters such as square brackets because PowerShell treats the path exactly as written. The examples use -Encoding UTF8; replace it with the correct encoding parameter when the source file is not UTF-8.

How to Check Whether Duplicate Lines Exist and Return Only Yes or No

Use this version when you only need to know whether at least one repeated line exists:

$duplicate = Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Select-Object -First 1

if ($null -ne $duplicate) {
    Write-Host "Duplicate lines found."
} else {
    Write-Host "No duplicate lines found."
}

The command returns only the first duplicate group, although PowerShell still has to read and group the file before it can produce that result. It is useful in a script, scheduled task, or repeatable validation workflow.

One-Line Version

if (Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 | Group-Object -CaseSensitive | Where-Object Count -gt 1 | Select-Object -First 1) { "Duplicate lines found" } else { "No duplicate lines found" }

How to Export Duplicate Lines and Occurrence Counts to CSV

To save the duplicate report instead of displaying it only in the console, export the grouped results to a CSV file:

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Sort-Object Count -Descending |
Select-Object Count, @{Name="Line"; Expression={$_.Name}} |
Export-Csv -LiteralPath ".\duplicate-report.csv" -NoTypeInformation -Encoding UTF8

The output file duplicate-report.csv can be opened in Excel, LibreOffice Calc, Notepad, or another text editor.

Save Each Duplicate Value Only Once

Use the following command when you need a plain TXT file containing one copy of each line that was duplicated:

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
ForEach-Object Name |
Set-Content -LiteralPath ".\duplicate-lines.txt" -Encoding UTF8

How to Ignore Letter Case, Extra Spaces, or Blank Lines

Ignore Letter Case

Group-Object is case-insensitive by default. Remove -CaseSensitive to treat PC, Pc, and pc as the same value:

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
Group-Object |
Where-Object Count -gt 1 |
Select-Object Count, Name

Ignore Spaces at the Beginning and End

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
ForEach-Object { $_.Trim() } |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Select-Object Count, Name

This makes example, example, and example equivalent. It does not remove spaces inside a line.

Ignore Empty and Whitespace-Only Lines

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
Where-Object { $_.Trim().Length -gt 0 } |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Select-Object Count, Name

Ignore Case, Trim Spaces, and Skip Empty Lines

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
ForEach-Object { $_.Trim() } |
Where-Object { $_.Length -gt 0 } |
Group-Object |
Where-Object Count -gt 1 |
Sort-Object Count -Descending |
Select-Object Count, Name

How to Check Duplicate Lines with Command Prompt and the SORT Command

The built-in Windows sort.exe command can arrange identical lines next to each other. This makes duplicates easier to see manually, but SORT does not automatically identify, count, or remove repeated lines.

1. Open the folder containing the text file.
2. Click the File Explorer address bar, type cmd, and press Enter.
3. Run the following command:

sort "input.txt" /o "sorted.txt"

Open sorted.txt in Notepad. Repeated values will appear next to each other, making a visual check possible.

For an automatic check from Command Prompt, call Windows PowerShell from CMD. This example also specifies UTF-8 explicitly so Cyrillic and accented characters are not decoded as ANSI in Windows PowerShell 5.1:

powershell.exe -NoProfile -Command "Get-Content -LiteralPath '.\input.txt' -Encoding UTF8 | Group-Object -CaseSensitive | Where-Object Count -gt 1 | Select-Object Count, Name"

How to Check Large Text Files for Duplicate Lines in Windows

The BAT script in Method 1 is more efficient than the short Group-Object examples because it does not read the complete file into an array and does not alphabetically sort every source line. It reads sequentially, writes first occurrences directly to the cleaned output, and stores only the unique keys and their counters in memory.

The direct PowerShell commands remain convenient, but Group-Object can be noticeably slower and may consume substantial RAM on a large file.

For a large file, use these precautions:

• Close memory-intensive applications before starting the check.
• Test the command on a copy of the file.
• Export only the duplicate report instead of formatting a huge result in the console.
• Use a 64-bit PowerShell process on 64-bit Windows.
• For multi-gigabyte files, consider a streaming script or a database-based workflow instead of Group-Object.

How to Fix Incorrect Characters When Checking Duplicate Lines

If PowerShell displays Cyrillic, accented, or other non-English text as sequences such as РіС..., the source file is usually being decoded with the wrong character encoding. The text itself may still be intact; the problem is how Get-Content interprets the file bytes.

Read a UTF-8 File Correctly

Get-Content -LiteralPath ".\input.txt" -Encoding UTF8 |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Sort-Object Count -Descending |
Select-Object Count, Name

This works for UTF-8 text with or without a byte order mark in Windows PowerShell 5.1. In newer PowerShell versions, UTF-8 is normally the default for many text operations, but specifying it explicitly makes the command unambiguous.

Choose the Parameter That Matches the File

Read a Legacy ANSI File

Get-Content -LiteralPath ".\input.txt" -Encoding Default |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Select-Object Count, Name

Read a UTF-16 Little-Endian File

Get-Content -LiteralPath ".\input.txt" -Encoding Unicode |
Group-Object -CaseSensitive |
Where-Object Count -gt 1 |
Select-Object Count, Name

The reusable find_duplicate_lines.bat script included in this guide recognizes UTF-8 with or without BOM, supports BOM-marked UTF-16 and UTF-32, and falls back to the Windows ANSI code page when a BOM-less byte sequence is not valid UTF-8. It writes both the duplicate report and the cleaned copy as UTF-8 with BOM.

FAQ: Finding Duplicate Lines in TXT Files on Windows