Best way to reject tens/hundreds of billions of duplicate candidates?

[undeath]

what you describe can be achieved by running

sort -m masterpwlist newpwlist | uniq -u

where masterpwlist and newpwlist are sorted