hcxtools - solution for capturing wlan traffic and conversion to hashcat formats

[ZerBea]

Hi ciccio17. Thanks, I'm fine. You too?

This is my test environment:
Arch Linux x86 (archlinux-2020.02.01-x86_64) hcxdumptool, hcxtools, hashcat, JtR
$ uname -r
5.5.5-arch1-1

Arch Linux Arm (ArchLinuxARM-rpi-latest) hcxdumptool, hcxtools
$ uname -r
4.19.102-1-ARCH

Regarding Kernel 5.5.5 many issues are still unsolved:
https://bugzilla.kernel.org/show_bug.cgi?id=202541 -> mostly USB3.0 related, mostly on AMD RYZEN systems
as well as most of the ath9k issues
https://bugzilla.kernel.org/buglist.cgi?...earch=ath9
https://bugs.openwrt.org/index.php?do=de...=&closedto=

As of today, I only use/test devices running mt76 and rt2800usb driver. Both chipsets running fine and out of the box on Raspberry PI A, B, A+, B+, Zero and x86 systems. The maintainers of this drivers are doing an excellent good job!
I won't do any more tests and I will not carry out any further optimizations (hcxdumptool) on other drivers unless they are fixed. That includes the Broadcom chipset (Raspberry's onboard SOC) because the native driver doesn't support monitor mode and packet injection.

ath9k_htc log (as of today):

Code:

$ hcxdumptool -v
hcxdumptool 6.0.1 (C) 2020 ZeroBeat

$ hcxdumptool -I
wlan interfaces:
f81a67027a1f wlp39s0f3u3u1u2 (ath9k_htc)

$ sudo hcxdumptool -i wlp39s0f3u3u1u2 --enable_status=127
initialization...

start capturing (stop with ctrl+c)
NMEA 0183 SENTENCE........: N/A
INTERFACE NAME............: wlp39s0f3u3u1u2
INTERFACE HARDWARE MAC....: f81a67027a1f
DRIVER....................: ath9k_htc
DRIVER VERSION............: 5.5.5-arch1-1
DRIVER FIRMWARE VERSION...: 1.4
ERRORMAX..................: 100 errors
FILTERLIST ACCESS POINT...: 0 entries
FILTERLIST CLIENT.........: 0 entries
FILTERMODE................: 0
WEAK CANDIDATE............: 12345678
PREDEFINED ACCESS POINT...: 0 entries
MAC ACCESS POINT BROADCAST: 0084ed5b1df6
MAC ACCESS POINT..........: 0084ed5b1df7 (incremented on every new client)
MAC CLIENT................: e00db95be6ae
REPLAYCOUNT...............: 61893
ANONCE....................: b0b4a53cddc62c8901bc67bd2f044517bbced62f55d432a810d6745be337e79f
SNONCE....................: b2580bcdc99fbb28b2238e634f1309f64ce10c35f601ddb3657794235f29cd24

...
09:45:00  12 INFO ERROR:0 INCOMING:2026 OUTGOING:301 PMKID:32 MP:0 GPS:0 RINGBUFFER:10

$ dmesg
[ 1732.644558] usb 5-3.1.2: Product: USB2.0 WLAN
[ 1732.644560] usb 5-3.1.2: Manufacturer: ATHEROS
[ 1732.644562] usb 5-3.1.2: SerialNumber: 12345
[ 1732.846948] usb 5-3.1.2: ath9k_htc: Firmware ath9k_htc/htc_9271-1.4.0.fw requested
[ 1732.847000] usbcore: registered new interface driver ath9k_htc
[ 1733.143466] usb 5-3.1.2: ath9k_htc: Transferred FW: ath9k_htc/htc_9271-1.4.0.fw, size: 51008
[ 1733.393997] ath9k_htc 5-3.1.2:1.0: ath9k_htc: HTC initialized with 33 credits
[ 1733.630748] ath9k_htc 5-3.1.2:1.0: ath9k_htc: FW Version: 1.4
[ 1733.630751] ath9k_htc 5-3.1.2:1.0: FW RMW support: On
[ 1733.630752] ath: EEPROM regdomain: 0x809c
[ 1733.630753] ath: EEPROM indicates we should expect a country code
[ 1733.630754] ath: doing EEPROM country->regdmn map search
[ 1733.630755] ath: country maps to regdmn code: 0x52
[ 1733.630756] ath: Country alpha2 being used: CN
[ 1733.630757] ath: Regpair used: 0x52
[ 1733.634523] ieee80211 phy0: Atheros AR9271 Rev:1
[ 1733.638888] ath9k_htc 5-3.1.2:1.0 wlp39s0f3u3u1u2: renamed from wlan0
[ 1768.128502] device wlp39s0f3u3u1u2 entered promiscuous mode
...
ath: phy0: Short RX data len, dropping (dlen: 8)
...

I used your last git on kernel 4.19 and is totally a different tool for me
-> hcxdumptool refactoring is in progress. Next version will get a different look. I noticed that the RPI drops packets when the system is busy (real time display). So we have to reduce this output to a minimum.

Latest git head of hashcat and hcxdumptool contain a lot of new features. It is a good idea to test them.

[MrShannon]

Can someone help me find the field order in the .22000 hashlist format?  I'm having a heck of a time tracking down the format description, and after digging through the .c code in hcxtools I am just not convinced that I understand what I am looking at :/

I am trying to extract the MAC Address of Clients and APs that were captured to build macfilters for hcxdumptool.

Before the .22000 format was introduced, I was using hcxpcaptool to convert .pcapng's to the .hccapx and older .16800 hashlist formats, then processing those with wlanhcxinfo or awk to get the mac addresses.  Of course, now that this tool and these formats are deprecated, I want to move to 22000, but here's how I was doing it:

Code:

# these tools/formats are now deprecated!

# Get Client MAC addresses to filter
# Convert the .pcapng to the older .hccapx format (EAPOL)
hcxpcaptool -o TEST.hccapx *.pcapng

# Get the "station" mac addresses (client MACs)
wlanhcxinfo -i TEST.hccapx -s | sort -u > GOTMACSCLI.txt

# Count how many we've got
wc -l GOTMACSCLI.txt
# 30 GOTMACSCLI.txt

# Convert the .pcapng to the old-older 16800 format (PMKID)
hcxpcaptool -z TEST.16800old *.pcapng
awk -F* '{print $2}' TEST.16800old | sort -u > GOTMACSAPS.txt

# Count how many we've got
wc -l GOTMACSAPS.txt
# 45 GOTMACSAPS.txt

Since the .22000 format combines both PMKID and EAPOL captures, I was hoping I could sift the MAC addresses out of the file with tools like awk/grep/sed, but since I don't know the format I am really just making guesses.  Can someone confirm my suspicions below?

Code:

# Convert the .pcapng to the new .22000 format (PMKID+EAPOL)
hcxpcapngtool --prefix="TEST" *.pcapng

Did a quick compare the contents of the GOTMACSCLI.txt and GOTMACSAPS.txt lists by grep'ing for a few examples

• A MAC that was in GOTMACSCLI.txt should be the MAC of a captured Client.  These MACs were in the .22000 list on lines that started with "WPA*02" and were the FIFTH field on that line.
  
• A MAC that was in GOTMACSAPS.txt should be the MAC of a captured Access Point.  These MACs were in the .22000 list on lines that started with "WPA*01" and were the FOURTH field on that line.
  
• Some of the MACs that were in GOTMACSAPS.txt were in multiple lines of the .22000 list that started with both "WPA*01" and "WPA*02", but they were always in in the FOURTH field.  I assume this means the AP was involved in a client EAPOL capture as well as a PMKID capture.
  

Based on that, I assume...

• lines that start with "WPA*02" are EAPOL hashes, so I want to grab the FIFTH field as the MAC of the Client to filter
  
• lines that start with "WPA*01" are PMKID hashes, so I want to grab the FOURTH field as the MAC of the Access Point to filter
  

If that's right, then I can got a little grep/awk/sort to get my MAC addresses.

Code:

# Get Client MAC addresses to filter and Count how many we've got
grep 'WPA\*02' TEST.22000 | awk -F* '{print $5}' | sort -u | wc -l
# 30

# Get Access Point MAC addresses to filter and Count how many we've got
grep 'WPA\*01' TEST.22000 | awk -F* '{print $4}' | sort -u | wc -l
# 44

This looks pretty good, but there is a discrepancy in the number of Access Point MAC addresses.  I've found other differences in the count of hashes when comparing .hccapx with .22000 before, so I am not totally surprised.  That said, this new format seems to have fewer results that the older format.  Maybe that's ok...

Am I doing this right?

[ZerBea]

The format of a 22000 hashline is:

Code:

SIGNATURE*TYPE*PMKID/MIC*MACAP*MACSTA*ESSID*ANONCE*EAPOL*MESSAGEPAIR

    SIGNATURE = "WPA"
    TYPE = 01 for PMKID, 02 for EAPOL, others to follow
    PMKID/MIC = PMKID if TYPE==01, MIC if TYPE==02
    MACAP = MAC of AP
    MACSTA = MAC of CLIENT
    ESSID = ESSID
    ANONCE = ANONCE
    EAPOL = EAPOL (SNONCE is in here as well as all EAPOL data)
    MESSAGEPAIR = Bitmask:

0: MP info (https://hashcat.net/wiki/doku.php?id=hccapx)
1: MP info (https://hashcat.net/wiki/doku.php?id=hccapx)
2: MP info (https://hashcat.net/wiki/doku.php?id=hccapx)
3: x (unused)
4: ap-less attack (set to 1) - no nonce-error-corrections necessary
5: LE router detected (set to 1) - nonce-error-corrections only for LE necessary
6: BE router detected (set to 1) - nonce-error-corrections only for BE necessary
7: not replaycount checked (set to 1) - replaycount not checked, nonce-error-corrections

Your bash commands to count CLIENT and AP MACs are ok. You can use bash commands as well as hcxhashtool to work on 22000 lines. And you can run hcxhashtool to verify the results of your script.
The discrepancy between hccapx converted with hcxpcaptool and 22000 converted with hcxpcapngtool is ok, too, because hcxpcangtool is running a better dupe detection. If you need all possible EAPOL message pair combinations you can use --all to retrieve them.

Also you should know, that hcxdumptool use randomized MACs as well as real MACs from received CLIENTs and received APs. It is a pretty good stealth feature to prevent counter measures against hcxdumptool, but will falsify the result of your count. Also keep in mind that filtering of CLIENTs is mostly useless if the CLIENT use randomized MACs.

Additional hcxdumptool v6.0.2 has an option to run BPF code, which is much faster then the old filter modes.

BTW:
hcxdumptool v6.0.2 add ROGUE to received PMKIDs and/or M1M2 message pairs if they are the result of a "CLIENT-LESS" or AP-LESS attack vector.

[MrShannon]

Thank you for confirming the format / field order in 22000!  That makes me feel a lot better about the method I am using and confidence to remove the scripting I was doing with the older tools and formats.

Also you should know, that hcxdumptool use randomized MACs as well as real MACs from received CLIENTs and received APs. It is a pretty good stealth feature to prevent counter measures against hcxdumptool, but will falsify the result of your count. Also keep in mind that filtering of CLIENTs is mostly useless if the CLIENT use randomized MACs.

I knew that hcxdumptool used random MACs but didn't know it would re-use captured MACs too, that is pretty great!  But I am extracting the MACS for EAPOL and PMKID separately, so I don't think that shouldn't be an issue right?  I'm sorting specifically for captured MACs on EAPOL and then for MACs on PMKID captures, so I should only be getting "genuine" MACs of attacked clients/aps.  To restate:

• The WPA*01 lines are PMKID captures, so the AP MAC address in field 4 would be a genuine attack target, but the fifth field would either be a random one that hcxdumptool made up or be one from an AP that was captured earlier.  I'm not using the 5th field on these lines, so that's ok.
  
• The WPA*02 lines are EAPOL captures, so the Client MAC address in field 5 would be a genuine attack target, but the fourth field would either be a random one that hcxdumptool made up or be one from a client that was captured earlier.  I'm not using the 4th field on these lines, so that's ok too.
  
• There is a possibility that the captured Client (which would appear in WPA*02 field 5) could be generating it's own random MAC addresses (hello Android).  In which case, filtering/ignoring those client MACs previously captured wouldn't do much good because the client would just generate a new MAC on subsequent attacks anyway.  But other static MAC using devices would get filtered as desired.
  

Additional hcxdumptool v6.0.2 has an option to run BPF code, which is much faster then the old filter modes.

I'm currently using 6.0.1 with both the "--filterlist_client" and "-filterlist_ap" switches. I know in older version BPF was used but I wasn't sure if I was formatting that right.  Looking at the changelog for 6.0.2, it looks like I would need to use tcpdump to create the new list. There is a 256 line limit on the current Client and AP filter lists.

1. Any such restrictions or guidance when using BPF?
   
2. Would I combine Client and AP macs into a single BPF?
   

[ZerBea]

Yes, you are right.

The BPF is a nice and fast kernel feature to filter out unwanted packets. You can combine everything (MAC_AP, MAC_CLIENT, rx branch, tx branch). The best way to use BPFC is to protect APs and CLIENTs.

Example to protect AP and CLIENT in tx branch and rx branch:
001122334455 == AP
aabbccddeeff == CLIENT

run hcxdumptool -m to set monitor interface for tcpdump (tcmpdump need this to now how to generate BPFC)
create filtercode as follows:

Code:

$ tcpdump -i <interface> not wlan addr1 00:11:22:33:44:55 and not wlan addr2 00:11:22:33:44:55 and not wlan addr1 aa:bb:cc:dd:ee and not wlan addr2 aa:bb:cc:dd:ee -ddd > protect.bpf

Notice:
You will loose PSKs transmitted in plain, if you use attack filters due to MAC randomization of most of the CLIENTs. The PSK transmitted with MAC x used by the CLIENT will not match to the authentication with MAC z used by the CLIENT
Due to your attack filter z, x is filtered out!!!!

I dropped BPF support in old wlandump-ng, because it was very limited. But now BPFC is restricted to 65535 code blocks. That should be more than enough for every purpose. A good reason to re-add this feature.

-> But I am extracting the MACS for EAPOL and PMKID separately, so I don't think that shouldn't be an issue right?
that could be a problem if you feed hashcat with either EAPOL or PMKID
we can have two (or more - depend on count of the CLIENTs) valid but different PSKs for a combination of MAC and ESSID, if the essid is a common ESSID like "default", "home", "Home", ....
- one PSK belongs to the AP
- one PSK belongs to the CLIENT A
- one PSK belongs to the CLIENT B
- one PSK belongs to the CLIENT C
...
The PSKs will only match if the CLIENT belongs to the AP
This does not necessarily have to be the case.

[MrShannon]

> But now BPFC is restricted to 65535 code blocks.

Wow, that is significant. Right now I am generating filter lists that cuts off at 256 entries after adding the "user defined" set of addresses to ignore. I also add the interfaces of the raspi (wlan0 and wlan1) just to be sure it doesn't attack itself.

I'm wondering about the length of the command being sent to tcpdump.

Let's pretend we have roughly 10% of the maximum number of mac addresses to filter, that's about 6550 mac addresses. Assuming we has to pass "and not wlan addr1 00:11:22:33:44:55" for each mac address to include, that's 37 characters per mac address, equating to more than 242,350 characters on the command line. Can you actually do that?

A cursory web search says the limit is around 5,200 characters, so at 37 characters per mac address, out upper limit is actually around 140 mac addresses. Which is a lot worse than using the mac filter list switches...

Am I misunderstanding how this works?

[ZerBea]

That depend on the environment variables of your system.
$ getconf ARG_MAX
2097152

I haven't tested it exactly , because I have not so much entries.

BTW:
It is allowed to use BPFC and filtermode options (filterlist_ap/filterlist_client) together.

[strike1953]

Hello, question
how to view 22000 format, essid and essid mac
Thank you

[Mem5]

Format 22000 is easily readable, just explained at the top of this thread :

The format of a 22000 hashline is:

Code:

SIGNATURE*TYPE*PMKID/MIC*MACAP*MACSTA*ESSID*ANONCE*EAPOL*MESSAGEPAIR

    SIGNATURE = "WPA"
    TYPE = 01 for PMKID, 02 for EAPOL, others to follow
    PMKID/MIC = PMKID if TYPE==01, MIC if TYPE==02
    MACAP = MAC of AP
    MACSTA = MAC of CLIENT
    ESSID = ESSID
    ANONCE = ANONCE
    EAPOL = EAPOL (SNONCE is in here as well as all EAPOL data)
    MESSAGEPAIR = Bitmask:

[strike1953]

My question is how do I view the essid and its mac from a xxx.22000 file in plain text?