text editor to delete duplicates from pool of large files

[Cmd2002]

2 other options for you would be to install Notepad++ and go CTRL + A

Edit > Line Operations > Remove Duplicates

OR

https://jpm22.github.io/txt/

go to the remove duplicate file option and select Big File

The second method may be better as it gives the option for Case Sensitive

Hope this helps!