hcxtools - solution for capturing wlan traffic and conversion to hashcat formats

[taxil]

ZerBea thank you for all the work on hcxtools, I wanted to post my experience with your tools in The-Distribution-Which-Does-Not-Handle-OpenCL-Well (Kali) Nethunter on a OnePlus One phone. I have been able to compile hcxtools and hcxdumptool. I am using a Panda PAU05 and TP-Link WN722N wireless card during testing.

First thing I ran into is when trying to run the makemonnb script. I get this:

deactivating NetworkManager and wpa_supplicant
Running in chroot, ignoring request: stop
Running in chroot, ignoring request: stop
activating monitor mode on wlan1
Interface wlan1
ifindex 29
type monitor
wiphy 6


I am not sure how to go about fixing the chroot issue but the card does manage to get into monitor mode.

I am able to capture handshakes with both wireless cards and with wlandump and hcxdumptool. I am getting seg faults after 15min-60min with hcxdumptool but no errors during.

[ZerBea]

Hi taxil.
That is a typical issue of that distribution and the included drivers.
For example:
https://null-byte.wonderhowto.com/forum/...e-0178595/
Device is not set 100% into monitor mode.
The script is optimized for arch LINUX. On other distros you must identify all services which takes access to the interface
and stop them.
First check dmseg output for errors.
Then check device for monitor mode:
sudo iw dev <device> info
A typical status looks like this:
wlp39s0f3u4u1u3 f8:1a:67:07:7d:0e
activating monitor mode on wlp39s0f3u4u1u3
Interface wlp39s0f3u4u1u3
ifindex 3
wdev 0x1
addr f8:1a:67:07:7d:0e
type monitor
wiphy 0
channel 1 (2412 MHz), width: 20 MHz (no HT), center1: 2412 MHz
txpower 20.00 dBm

- interface is set to monitor mode
- we can control tx pwr
- we can control channel set


BTW:
some of the tools are running in background of wpa-sec (https://wpa-sec.stanev.org)

take a look at the stats (as of today):
Last 24h processed handshakes: 26389
Last 24h performance: 293.12K/s
Last 24h submissions: 410
Last 24h founds: 1262

[taxil]

Is anyone able to compile hcxdumptool on macOS? When I enter make I get this:

make: Nothing to be done for `all'.

[ZerBea]

No, macOS is not supported, because hcxdumptool and wlandump-ng are too LINUX specific.
make on macOS is disabled
You can try a VM.

[taxil]

Also here is what I see in dmesg

[10381.689021] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[10381.689125] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[10381.689177] usb usb1: Product: xHCI Host Controller
[10381.689412] usb usb1: Manufacturer: Linux 3.4.113-lineageos-g2c0e9edc xhci-hcd
[10381.689509] usb usb1: SerialNumber: xhci-hcd
[10381.689662] usb usb1: parent xhci-hcd should not be sleeping
[10381.691200] xHCI xhci_add_endpoint called for root hub
[10381.691210] xHCI xhci_check_bandwidth called for root hub
[10381.691486] hub 1-0:1.0: USB hub found
[10381.691593] hub 1-0:1.0: 1 port detected
[10381.692148] xhci-hcd xhci-hcd: xHCI Host Controller
[10381.692211] xhci-hcd xhci-hcd: new USB bus registered, assigned bus number 2
[10381.692300] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[10381.692412] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[10381.692468] usb usb2: Product: xHCI Host Controller
[10381.692843] usb usb2: Manufacturer: Linux 3.4.113-lineageos-g2c0e9edc xhci-hcd
[10381.692894] usb usb2: SerialNumber: xhci-hcd
[10381.693065] usb usb2: parent xhci-hcd should not be sleeping
[10381.693883] xHCI xhci_add_endpoint called for root hub
[10381.693887] xHCI xhci_check_bandwidth called for root hub
[10381.694174] hub 2-0:1.0: USB hub found
[10381.694290] hub 2-0:1.0: 1 port detected
[10382.460783] usb 1-1: new high-speed USB device number 2 using xhci-hcd
[10382.488740] usb 1-1: New USB device found, idVendor=0cf3, idProduct=9271
[10382.488781] usb 1-1: New USB device strings: Mfr=16, Product=32, SerialNumber=48
[10382.488846] usb 1-1: Product: USB2.0 WLAN
[10382.488880] usb 1-1: Manufacturer: ATHEROS
[10382.488944] usb 1-1: SerialNumber: 12345
[10382.503565] usb 1-1: ath9k_htc: Firmware htc_9271.fw requested
[10382.787849] usb 1-1: ath9k_htc: Transferred FW: htc_9271.fw, size: 51272
[10383.033229] ath9k_htc 1-1:1.0: ath9k_htc: HTC initialized with 33 credits
[10383.292559] ath9k_htc 1-1:1.0: ath9k_htc: FW Version: 1.3
[10383.292614] ath: EEPROM regdomain: 0x809c
[10383.292617] ath: EEPROM indicates we should expect a country code
[10383.292621] ath: doing EEPROM country->regdmn map search
[10383.292624] ath: country maps to regdmn code: 0x52
[10383.292627] ath: Country alpha2 being used: CN
[10383.292630] ath: Regpair used: 0x52
[10383.297940] ieee80211 phy7: Atheros AR9271 Rev:1
[10383.298394] cfg80211: Calling CRDA for country: CN
[10383.300250] cfg80211: Current regulatory domain intersected:
[10383.300316] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[10383.300392] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm)
[10383.300487] cfg80211: (5170000 KHz - 5250000 KHz @ 80000 KHz), (600 mBi, 2300 mBm)
[10383.300529] cfg80211: (5250000 KHz - 5330000 KHz @ 80000 KHz), (600 mBi, 2300 mBm)
[10383.300603] cfg80211: (5735000 KHz - 5835000 KHz @ 80000 KHz), (600 mBi, 3000 mBm)
[10383.300683] cfg80211: (57240000 KHz - 59400000 KHz @ 2160000 KHz), (N/A, 2800 mBm)
[10383.300787] cfg80211: (59400000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 4000 mBm)
[10383.302426] Registered led device: ath9k_htc-phy7
[10384.300496] init: Starting service 'su_daemon'...
[10384.318380] init: Service 'su_daemon' (pid 26083) exited with status 0
[10384.318436] init: Service 'su_daemon' (pid 26083) killing any children in process group
[10384.318578] init: Untracked pid 26085 exited with status 1
[10385.515171] msm_thermaltore_cc_enabled Core control disabled
[10385.520364] msm_thermaltore_cc_enabled Core control enabled
[10385.521442] msm_thermaltore_cc_enabled Core control disabled
[10385.533830] msm_thermaltore_cc_enabled Core control enabled

When I run makemonnb I see:
[10441.201188] [11:26:02.014539] [0000012396713AA2] [MC_Th] wlan: [E :HDP] hdd_tx_rx_pkt_cnt_stat_timer_handler: Disable split scan
[10443.748652] init: Starting service 'su_daemon'...
[10443.779344] init: Untracked pid 26221 exited with status 1
[10443.779590] init: Service 'su_daemon' (pid 26219) exited with status 0
[10443.779759] init: Service 'su_daemon' (pid 26219) killing any children in process group
[10444.143353] [11:26:04.956707] [0000012399CF3132] [wpa_s] wlan: [E :HDD] wlan_hdd_get_frame_logs: Frame Logging not init!
[10448.785875] init: Starting service 'su_daemon'...
[10448.854015] init: Service 'su_daemon' (pid 26224) exited with status 0
[10448.855960] init: Service 'su_daemon' (pid 26224) killing any children in process group
[10448.859087] init: Untracked pid 26226 exited with status 1

sudo iw dev wlan1 info gives me:
Interface wlan1
ifindex 30
type monitor
wiphy 7

[ZerBea]

Your firmware is old.

[46356.910951] usb 5-4.1.3: Manufacturer: ATHEROS
[46356.910952] usb 5-4.1.3: SerialNumber: 12345
[46356.922044] usb 5-4.1.3: ath9k_htc: Firmware ath9k_htc/htc_9271-1.4.0.fw requested
[46357.202532] usb 5-4.1.3: ath9k_htc: Transferred FW: ath9k_htc/htc_9271-1.4.0.fw, size: 51008
[46357.452903] ath9k_htc 5-4.1.3:1.0: ath9k_htc: HTC initialized with 33 credits
[46357.678909] ath9k_htc 5-4.1.3:1.0: ath9k_htc: FW Version: 1.4
[46357.678912] ath9k_htc 5-4.1.3:1.0: FW RMW support: On
[46357.678913] ath: EEPROM regdomain: 0x809c
[46357.678914] ath: EEPROM indicates we should expect a country code
[46357.678915] ath: doing EEPROM country->regdmn map search
[46357.678915] ath: country maps to regdmn code: 0x52
[46357.678916] ath: Country alpha2 being used: CN
[46357.678916] ath: Regpair used: 0x52
[46357.683281] ieee80211 phy1: Atheros AR9271 Rev:1
[46357.684834] ath9k_htc 5-4.1.3:1.0 wlp39s0f3u4u1u3: renamed from wlan0
[46402.096342] device wlp39s0f3u4u1u3 entered promiscuous mode


Your wireless subsystem doesn't allow all possible channels (on 2.4 GHz only 1 to 13).
$ iw reg get
global
country 98: DFS-FCC
(2402 - 2482 @ 40), (N/A, 20), (N/A)
(5170 - 5250 @ 80), (N/A, 23), (N/A), AUTO-BW
(5250 - 5330 @ 80), (N/A, 23), (0 ms), DFS, AUTO-BW
(5735 - 5835 @ 80), (N/A, 30), (N/A)

hcxdumptool default scanlist:
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
34, 36, 38, 40, 42, 44, 46, 48, 52, 56, 58, 60, 62, 64,
100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 147, 149, 151, 153, 155, 157,
161, 165, 167, 169, 184, 188, 192, 196, 200, 204, 208, 212, 216,
So you will run into trouble, if the scan reached a "not allowed channel".
You can try a custom scanlist (-c 1,2,3,4,5,6,7,8,9,10,11)

And the important part of dmesg:
[46402.096342] device wlp39s0f3u4u1u3 entered promiscuous mode
your interface didn't enter promiscuous mode

If everything is fine, hcxdumptool show this status:
$ sudo hcxdumptool -i wlp39s0f3u4u7 -o test.hccapx -s

start capturing (stop with ctrl+c)
INTERFACE: wlp39s0f3u4u7
MAC_AP...: 00259d61542c (rogue access point)
MAC_STA..: f0a225dd6912 (rogue client)
INFO.....: cha=3, rcv=76, err=0

cha (current channel) should change
rcv (received packets) should increase
err (error) should be 0

If this doesn't happen, your system is misconfigured or your driver isn't working as expected.

[taxil]

Anyone else having trouble with the blacklist files? I am getting "reading blacklist entry failed error." It seems that none of the blacklist file is working as my client laptop is being deauthed even though it is included in blacklist. Attached is part of my blacklist file.

[ZerBea]

Hi taxil.
tested your blacklist:
$ sudo hcxdumptool -i wlp39s0f3u4u5 -o test.pcap -s -B blacklistO.txt

start capturing (stop with ctrl+c)
INTERFACE: wlp39s0f3u4u5
MAC_AP...: e80410a4b6d2 (rogue access point)
MAC_STA..: fcc233d8e21e (rogue client)
INFO.....: cha=7, rcv=598, err=0

and it seems to be ok.

Are you shure that the client is deauthenticated /disassociated or
does the client try to connect to the rogue access point.

If hcxdumptool retrieved a handshake for a network, it will stop sending deauthentications / disassociations.

There is no need to add all bssid+client+ssid pairs to the blacklist.
Only one combination is enough to stop deauthentications/disassociations on this network.

The client stores all attempts to connect to an ap and tries it again and again and again.

Does your client use randomized macs?

We can't stop the client trying to connect us because of "MAC randomization".
We use a randomized mac and most of the clients use a randomized mac.

start capturing (stop with ctrl+c)
INTERFACE: wlp39s0f3u4u5
MAC_AP...: 00182534639c (rogue access point)
MAC_STA..: fcc2333c3cf1 (rogue client)

Take a look at this example:
We start hcxdumptool against an Android 6 test client

$ sudo hcxdumptool -i wlp39s0f3u4u5 -o test.pcap -s -t 60 -c 11
start capturing (stop with ctrl+c)
INTERFACE: wlp39s0f3u4u5
MAC_AP...: 00269f8920cf (rogue access point)
MAC_STA..: fcc23386b99d (rogue client)
[08:38:55] 00269f8920d0:f072cea7edfd:Testnetwork [HANDSHAKE]
terminated...

we used a randomized mac and the client used a randomized mac.
the client connected to us and we retrieved a handshake.
we stopped hcxtumptool.

Now we started hcxdumptool again.

$ sudo hcxdumptool -i wlp39s0f3u4u5 -o test.pcap -s -t 60 -c 11
start capturing (stop with ctrl+c)
INTERFACE: wlp39s0f3u4u5
MAC_AP...: 24bf747e299a (rogue access point)
MAC_STA..: fcc233144966 (rogue client)
[08:39:55] 00269f8920d0:a68e3357e491:Testnetwork [HANDSHAKE]
terminated...

as you can see, the client used a different mac
and tried to connect us with his the stored informations from the last attempt to connect us

Now we restart the client (turn off / turn on cell phone)
$ sudo hcxdumptool -i wlp39s0f3u4u5 -o test.pcap -s -t 60 -c 11
start capturing (stop with ctrl+c)
INTERFACE: wlp39s0f3u4u5
MAC_AP...: 0418b6d009f8 (rogue access point)
MAC_STA..: f0a2250bbd7b (rogue client)
[08:48:01] 0418b6d009f9:c35f72a6f9d3:Testnetwork [HANDSHAKE]

now we have complete new mac addresses.

If you use hcxdumptool in an allready discovered area (stationary at home), use Option -D and -t 15 to get only the new clients.
If you discover a new area run hcxdumptool -t 5 for a while to get handshakes from all AP's with connect clients in range.

Please keep in mind:
hcxtools are analysis tools. That means, we want the client to do something that he normally doesn't do (give us the content of his NVRAM for example).
Therefore we must be a little bit aggressive.

[taxil]

My goal today is to create one master 100% crackable hccapx from all of the pcap files from the last week created from hcxdumptool.

I have created a processing script that has two ways of attempting to doing this:


~/cap/Combined/tools/hcxtools/wlancap2hcx -p output/merged.pcap *.pcap
cat *.pcap > output/merged_cat.pcap

#old method using wlancap
~/cap/Combined/tools/hcxtools/wlancap2hcx -Z -D -o output/current_wlan_p.hccapx output/merged.pcap
~/cap/Combined/tools/hcxtools/wlancap2hcx -Z -D -o output/current_wlan_cat.hccapx output/merged_cat.pcap

#newer hcxpcaptool
~/cap/Combined/tools/hcxtools/hcxpcaptool -o output/current_new_p.hccapx output/merged.pcap
~/cap/Combined/tools/hcxtools/hcxpcaptool -o output/current_new_cat.hccapx output/merged_cat.pcap


Results:

---
15K May  2 18:02 current_new_cat.hccapx
3.6M May  2 18:02 current_new_p.hccapx
188K May  2 18:02 current_wlan_cat.hccapx
251K May  2 18:02 current_wlan_p.hccapx
22M May  2 18:02 merged.pcap
60M May  2 18:02 merged_cat.pcap
---


MBP:~/cap/Combined$ ./tools/hcxtools/wlanhcxinfo -i output/current_wlan_p.hccapx 
total hashes read from file.......: 653
handshakes from clients...........: 393
little endinan router detected....: 0
big endinan router detected.......: 0
zeroed ESSID......................: 0
802.1x Version 2001...............: 481
802.1x Version 2004...............: 172
WPA1 RC4 Cipher, HMAC-MD5.........: 3
WPA2 AES Cipher, HMAC-SHA1........: 648
WPA2 AES Cipher, AES-128-CMAC.....: 0
group key flag set................: 1
message pair M12E2................: 539 (27 not replaycount checked)
message pair M14E4................: 14 (8 not replaycount checked)
message pair M32E2................: 100 (43 not replaycount checked)
message pair M32E3................: 0 (0 not replaycount checked)
message pair M34E3................: 0 (0 not replaycount checked)
message pair M34E4................: 0 (0 not replaycount checked)
nonce-error-corrections is working on that file

MBP:~/cap/Combined$ ./tools/hcxtools/wlanhcxinfo -i output/current_wlan_cat.hccapx 
total hashes read from file.......: 489
handshakes from clients...........: 276
little endinan router detected....: 0
big endinan router detected.......: 0
zeroed ESSID......................: 0
802.1x Version 2001...............: 360
802.1x Version 2004...............: 129
WPA1 RC4 Cipher, HMAC-MD5.........: 0
WPA2 AES Cipher, HMAC-SHA1........: 487
WPA2 AES Cipher, AES-128-CMAC.....: 0
group key flag set................: 1
message pair M12E2................: 402 (26 not replaycount checked)
message pair M14E4................: 14 (8 not replaycount checked)
message pair M32E2................: 73 (36 not replaycount checked)
message pair M32E3................: 0 (0 not replaycount checked)
message pair M34E3................: 0 (0 not replaycount checked)
message pair M34E4................: 0 (0 not replaycount checked)
nonce-error-corrections is working on that file

MBP:~/cap/Combined$ ./tools/hcxtools/wlanhcxinfo -i output/current_new_p.hccapx 
total hashes read from file.......: 9530
handshakes from clients...........: 8867
little endinan router detected....: 97
big endinan router detected.......: 0
zeroed ESSID......................: 0
802.1x Version 2001...............: 6486
802.1x Version 2004...............: 3044
WPA1 RC4 Cipher, HMAC-MD5.........: 2
WPA2 AES Cipher, HMAC-SHA1........: 9528
WPA2 AES Cipher, AES-128-CMAC.....: 0
group key flag set................: 0
message pair M12E2................: 9200 (29 not replaycount checked)
message pair M14E4................: 240 (118 not replaycount checked)
message pair M32E2................: 90 (2 not replaycount checked)
message pair M32E3................: 0 (0 not replaycount checked)
message pair M34E3................: 0 (0 not replaycount checked)
message pair M34E4................: 0 (0 not replaycount checked)
nonce-error-corrections is working on that file

MBP:~/cap/Combined$ ./tools/hcxtools/wlanhcxinfo -i output/current_new_cat.hccapx
total hashes read from file.......: 39
handshakes from clients...........: 26
little endinan router detected....: 0
big endinan router detected.......: 0
zeroed ESSID......................: 0
802.1x Version 2001...............: 23
802.1x Version 2004...............: 16
WPA1 RC4 Cipher, HMAC-MD5.........: 0
WPA2 AES Cipher, HMAC-SHA1........: 39
WPA2 AES Cipher, AES-128-CMAC.....: 0
group key flag set................: 0
message pair M12E2................: 35 (1 not replaycount checked)
message pair M14E4................: 0 (0 not replaycount checked)
message pair M32E2................: 4 (0 not replaycount checked)
message pair M32E3................: 0 (0 not replaycount checked)
message pair M34E3................: 0 (0 not replaycount checked)
message pair M34E4................: 0 (0 not replaycount checked)
nonce-error-corrections is working on that file

---



My questions:
1. What is the better merge method? I know I have many duplicate handshakes
2. Something seems very wrong with the hcxpcaptool method I am using, any thoughts on way I have so few handshakes in the current_new_cat.hccapx?
3. I want the hccapx file to be as clean as possible, if I feed a cracked wordlist into Hashcat and --remove, I do not want to see for example, ATT722 in the potfile and still in the Hashcat processed hccapx file.

[ZerBea]

There ara several ways to do this. But keep in mind, that there isn't a 100% solution to create a 100% crackable hccapx file. A big problem is a packetloss during capturing. This will not happen on ap-less handshakes, because hcxtools requests missing packets.
So it's a good idea to create a hccapx file only from clients:
wlanhcx2ssid -i all.hccapx -w apless.hccapx
wlanhcx2ssid -i apless.hccapx -N aplesscleaned.hccapx

aplesscleaned.hccapx now contains one handshake each mac_ap,mac_sta, ESSID combination from clients
you can run nonce-error-corrections=0 on that file.
all handshakes are 100% crackable, but may not contain the correct PSK for a network
- a clients typed 12345678 to get acces to a networks which isn't his own
- a clients made a typo passwore instead of password

or if you want also handshakes from regular APs:
wlanhcx2ssid -i all.hccapx -r rcchecked.hccapx
wlanhcx2ssid -i rcchecked.hccapx -N rccheckedcleaned.hccapx

merged caps can lead to unwanted results or uncrackable handshakes (using a PSK) in that case if the AP changed his ESSID. Nevertheless this handshakes are crackable using a PMK.

I prefer 2 hccapx files:
archiv_best.hccapx (created by hcxpcaptool -o for usage with hashmode -m 2500 only)
archiv_raw.hccapx (created by hcxpcaptool -O for usage with hashmode -m 2501 only)
Now it's time to strip the ones to be checked:
for example:
wlanhcx2ssid -i archiv_best.hccapx -X Home
hashcat -m 2500 Home.hccapx hashes.org-2018.txt

So, it doesn't make sense to run hashcat on "one big hccapx". You will wast GPU time. But it's a good idea to create some big hccapx files and use them as archive.
The retrieve the ones you like to crack from that archiv and run hashcat on them.

It's also a good idea to use separate potfiles for 2500 and 2501 and analyze this potfiles. You will get many infos about the used keyspace and weak points from this 2 files.
It also make sense to create ESSID, USERNAME and IDENTITY files (hcxpcaptool -E -U -I).
Cat them together with your founds and run princeproccessor against you hccapx files.

BTW:
wpa-sec (https://wpa-sec.stanev.org/?stats) retrieved several hundred PSKs a day using this method (as of today):
Last 24h founds: 307

Please note, that hcxtools are not designed to crack single networks. Goal is to break the system by running massiv attacks against all(!) reachable clients (prefered, because clients are much much more vulnerable than ap's) and ap's.