Utf32 support

[dvu]

hashcat support mode: 1770 = sha512(utf16le($pass))

How can I add support for this mode?

sha512(utf32le($pass))

Do I need to patch hashcat source code to add a custom mode or it can be achieved by some rules?

[philsmd]

for a -a 0 (wordlist) attack, you could just use -m 1700 = SHA2-512 and use the "--encoding-to utf32le" feature.

This might be all you need to crack a specific hash (list)... of course if you need all the features and attack modes, you are right that this can only be done by adding a new/specific hash mode (new hash algorithm)