hashcat Forum › Support › hashcat v
NVAPI_ERROR

Threaded Mode
NVAPI_ERROR
vitiura Offline
Junior Member
**
Posts: 5
Threads: 1
Joined: Mar 2022
#1
03-11-2022, 06:21 PM
Hello everyone!
I try to start hashcat (wordlist brutforce) on my rig (win server 2019 with 10 of RTX 3080ti) and it crashes after about 10-20 minutes with unknown error.

I tried:
1. Upgrade/Downgrade nvidia drivers;
2. Downgrade hashcat version (all down to 6.2.0);
2. Expand swap file up to 120 GB;
3. Use different mods (-w option) doesn't impact;
4. Use different wordlists (100 and 400 GB);
5. Use different hashtypes;

Any other conditions:
- Windows server 2019
- RAM 64GB
- Intel core i7 7700k
- NVIDIA EVGA RTX 3080ti (10 devices)
- ETH miner works stable
- Any of 7 cards works stable. The problem start with 8th device
- Temp max 60 C
- Don't use any boost. Just default settings.



Thank you for any help!

hashcat output below:

[s]tatus [p]ause [b]ypass [c]heckpoint [f]inish [q]uit => NvAPI_GPU_GetPerfPoliciesStatus(): NVAPI_ERROR

nuLaunchKernel(): unknown error  UcuLaunchKernel(): unknown errorcuMemcpyHtoDAsync(): unknown error known Error

cvmlDeviceGetUtilizationRates(): UnvmlDeviceGetUtilizationRates(): Unknown Error


nvmlDeviceGetUtilizationRates(): Unknown Error

nvmlDeviceGetUtilizationRates(): Unknown ErrornvmlDeviceGetUtilizationRates(): U


nvmlDeviceGetUtilizationRates(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

cuMemcpyDtoHAsync(): unknown error

cuMemcpyDtoHAsync(): unknown error

nvmlDeviceGetFanSpeed(): Unknown Error

Session..........: hashcat
Status...........: Error
Hash.Mode........: 20 (md5($salt.$pass))
Hash.Target......: C:\hashcatgui\hashcat\hashes\test.txt
Time.Started.....: Wed Mar 09 11:46:21 2022 (13 mins, 15 secs)
Time.Estimated...: Wed Mar 09 12:24:39 2022 (25 mins, 3 secs)
Kernel.Feature...: Optimized Kernel
Guess.Base.......: File (D:\files\list.txt)
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:  5354.7 kH/s (0.36ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#2.........:  5396.1 kH/s (0.32ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#3.........:  5472.6 kH/s (0.29ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#4.........:  5436.7 kH/s (0.38ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#5.........:  5002.4 kH/s (0.60ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#6.........:  5283.0 kH/s (0.66ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#7.........:  5344.5 kH/s (0.56ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#8.........:  5120.5 kH/s (0.38ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#9.........:  5220.3 kH/s (0.48ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#10.........:  4995.7 kH/s (0.57ms) @ Accel:1024 Loops:1 Thr:32 Vec:1
Speed.#*.........: 52626.5 kH/s
Recovered........: 0/3 (0.00%) Digests, 0/3 (0.00%) Salts
Progress.........: 41625845760/120741743394 (34.48%)
Rejected.........: 0/41625845760 (0.00%)
Restore.Point....: 13738967040/40247247798 (34.14%)
Restore.Sub.#1...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#2...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#3...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#4...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#5...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#6...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#7...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#8...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#9...: Salt:2 Amplifier:0-1 Iteration:0-1
Restore.Sub.#10...: Salt:2 Amplifier:0-1 Iteration:0-1
Candidate.Engine.: Device Generator
Candidates.#1....:  ->
Candidates.#2....:  ->
Candidates.#3....:  ->
Candidates.#4....:  ->
Candidates.#5....:  ->
Candidates.#6....:  ->
Candidates.#7....:  ->
Candidates.#8....:  ->
Candidates.#9....:  ->
Candidates.#10....:  ->
Hardware.Mon.#1..: Temp: 52c Core:1950MHz Mem:9242MHz Bus:1
Hardware.Mon.#2..: Temp: 51c Core:1859MHz Mem:9242MHz Bus:1
Hardware.Mon.#3..: Temp: 51c Core:1948MHz Mem:9242MHz Bus:1
Hardware.Mon.#4..: Temp: 54c Core:1979MHz Mem:9242MHz Bus:1
Hardware.Mon.#5..: Temp: 51c Core:1933MHz Mem:9242MHz Bus:1
Hardware.Mon.#6..: Temp: 46c Core:1933MHz Mem:9242MHz Bus:1
Hardware.Mon.#7..: Temp: 50c Core:1964MHz Mem:9242MHz Bus:1
Hardware.Mon.#8..: Temp: 51c Core:1933MHz Mem:9242MHz Bus:1
Hardware.Mon.#9..: Temp: 49c Core:1799MHz Mem:9492MHz Bus:1
Hardware.Mon.#10..: Temp: 54c Core:1999MHz Mem:9242MHz Bus:1
cuMemFree(): unknown error

cuMemFree(): unknown error

cuMemFree(): unknown error

cuMemFree(): unknown error

cuMemFree(): unknown error

cuMemFree(): unknown error

cuMemFree(): unknown error
Find
Reply
slyexe Offline
Senior Member
****
Posts: 374
Threads: 0
Joined: Nov 2017
#2
03-13-2022, 04:37 AM
I would suspect it has an issue with the amount of memory being allocated based on the error appearing "cuMemFree(): unknown error".

The typical setup for such a large system is to have VRAM = RAM so in this scenario you would need a total of 120GB of RAM. I don't suspect a swap file would work as intended in this situation so either increase the physical RAM or disable cards.
Find
Reply
vitiura Offline
Junior Member
**
Posts: 5
Threads: 1
Joined: Mar 2022
#3
03-14-2022, 11:38 AM
(03-13-2022, 04:37 AM)slyexe Wrote: I would suspect it has an issue with the amount of memory being allocated based on the error appearing "cuMemFree(): unknown error".

The typical setup for such a large system is to have VRAM =  RAM so in this scenario you would need a total of 120GB of RAM. I don't suspect a swap file would work as intended in this situation so either increase the physical RAM or disable cards.

Thank you for support! 
I was also inclined to this opinion. But when Hashcat works, according to the task manager, the RAM load of the entire system is no more than 30%. Do you think that this is not a true indicator?
Find
Reply
Snoopy Offline
Posting Freak
*****
Posts: 893
Threads: 15
Joined: Sep 2017
#4
03-14-2022, 07:39 PM (This post was last modified: 03-14-2022, 07:45 PM by Snoopy.)
can you post the output of hashcat -I

EDIT: was missing this part - Any of 7 cards works stable. The problem start with 8th device
-> did you used -D 2 -d 1,2,3,4,5,6,7 to use only 7 cards? if not see comment below

yeah like slyexe was mentioning, seems to be a problem with your system ram, you could use options -D 2 -d 1,2,3,4 ... to tell hashcat just to use the first x cards and see if this failure disappears

never the less, plz show output of hashcat -I
Find
Reply
vitiura Offline
Junior Member
**
Posts: 5
Threads: 1
Joined: Mar 2022
#5
03-14-2022, 11:40 PM
(03-14-2022, 07:39 PM)Snoopy Wrote: EDIT: was missing this part - Any of 7 cards works stable. The problem start with 8th device
-> did you used -D 2 -d 1,2,3,4,5,6,7 to use only 7 cards?

Yes, I tried to explain that any combination with 7 CUDA devices works good and the problem is not with one of my devices.
-D 2 -d 1,2,3,4,5,6,7 and -D 2 -d 4,5,6,7,8,9,10 lead to the same result (it works). But even if I add another one CUDA device it will crash in 10-15 minutes of work.


Hashcat -I output: 

CUDA Info:
==========

CUDA.Version.: 11.5

Backend Device ID #1 (Alias: #11)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:01:00.0

Backend Device ID #2 (Alias: #12)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:02:00.0

Backend Device ID #3 (Alias: #13)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:03:00.0

Backend Device ID #4 (Alias: #14)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:04:00.0

Backend Device ID #5 (Alias: #15)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:08:00.0

Backend Device ID #6 (Alias: #16)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:09:00.0

Backend Device ID #7 (Alias: #17)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:0b:00.0

Backend Device ID #8 (Alias: #18)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:0c:00.0

Backend Device ID #9 (Alias: #19)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:0d:00.0

Backend Device ID #10 (Alias: #20)
  Name...........: NVIDIA GeForce RTX 3080 Ti
  Processor(s)...: 80
  Clock..........: 1800
  Memory.Total...: 12287 MB
  Memory.Free....: 10054 MB
  Local.Memory...: 99 KB
  PCI.Addr.BDFe..: 0000:0e:00.0

OpenCL Info:
============

OpenCL Platform ID #1
  Vendor..: NVIDIA Corporation
  Name....: NVIDIA CUDA
  Version.: OpenCL 3.0 CUDA 11.5.56

  Backend Device ID #11 (Alias: #1)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 01:00.0

  Backend Device ID #12 (Alias: #2)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 02:00.0

  Backend Device ID #13 (Alias: #3)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 03:00.0

  Backend Device ID #14 (Alias: #4)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 04:00.0

  Backend Device ID #15 (Alias: #5)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 08:00.0

  Backend Device ID #16 (Alias: #6)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 09:00.0

  Backend Device ID #17 (Alias: #7)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 0b:00.0

  Backend Device ID #18 (Alias: #8)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 0c:00.0

  Backend Device ID #19 (Alias: #9)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 0d:00.0

  Backend Device ID #20 (Alias: #10)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: NVIDIA GeForce RTX 3080 Ti
    Version........: OpenCL 3.0 CUDA
    Processor(s)...: 80
    Clock..........: 1800
    Memory.Total...: 12287 MB (limited to 3071 MB allocatable in one block)
    Memory.Free....: 10624 MB
    Local.Memory...: 48 KB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 496.13
    PCI.Addr.BDF...: 0e:00.0

OpenCL Platform ID #2
  Vendor..: Intel(R) Corporation
  Name....: Intel(R) OpenCL
  Version.: OpenCL 2.1

  Backend Device ID #21
    Type...........: GPU
    Vendor.ID......: 8
    Vendor.........: Intel(R) Corporation
    Name...........: Intel(R) HD Graphics 630
    Version........: OpenCL 2.1
    Processor(s)...: 24
    Clock..........: 1150
    Memory.Total...: 26082 MB (limited to 1023 MB allocatable in one block)
    Memory.Free....: 12992 MB
    Local.Memory...: 64 KB
    OpenCL.Version.: OpenCL C 2.0
    Driver.Version.: 21.20.16.4534

  Backend Device ID #22
    Type...........: CPU
    Vendor.ID......: 8
    Vendor.........: Intel(R) Corporation
    Name...........: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
    Version........: OpenCL 2.1 (Build 359)
    Processor(s)...: 8
    Clock..........: 4200
    Memory.Total...: 65227 MB (limited to 8153 MB allocatable in one block)
    Memory.Free....: 32581 MB
    Local.Memory...: 32 KB
    OpenCL.Version.: OpenCL C 2.0
    Driver.Version.: 6.6.0.359
Find
Reply
Snoopy Offline
Posting Freak
*****
Posts: 893
Threads: 15
Joined: Sep 2017
#6
03-15-2022, 07:26 PM
well like slyexe mentioned, you dont have enough ram to use all cards, so upgrade your ram or stick to 7 cards (i would assume that even 7 cards will/could throw an error, depending on the attack
Find
Reply
vitiura Offline
Junior Member
**
Posts: 5
Threads: 1
Joined: Mar 2022
#7
03-16-2022, 11:06 AM (This post was last modified: 03-16-2022, 02:50 PM by vitiura.)
(03-15-2022, 07:26 PM)Snoopy Wrote: well like slyexe mentioned, you dont have enough ram to use all cards, so upgrade your ram or stick to 7 cards (i would assume that even 7 cards will/could throw an error, depending on the attack

Thank you for your answer.
I almost completely agree with you, but I have some doubts. On another 4x1080ti rig (11 GB VRAM each) Hashcat works great with any type of attack in nightmare mode. At the same time, the motherboard has only 8 GB of RAM. VRAM/RAM ratio is 4!!!

UPD: I've solved problem! The error gone when I removed the -O flag from arguments (Enable optimized kernels). Honestly, I do not know if there will be any limits with this, but it works! Hope this post can help somebody.
Find
Reply
slyexe Offline
Senior Member
****
Posts: 374
Threads: 0
Joined: Nov 2017
#8
03-17-2022, 05:20 AM
Glad you were able to solve the issue. I was going to reply back but was out of town for work. I was going to suggest trying to run 2 attacks simultaneously dividing the cards into 2 separate instances. But if you have resolved the problem I would say this would be irrelevant at this point in time.
Find
Reply
MidnightHunter Offline
Junior Member
**
Posts: 3
Threads: 1
Joined: Mar 2022
#9
03-25-2022, 03:47 AM
(03-16-2022, 11:06 AM)vitiura Wrote:
(03-15-2022, 07:26 PM)Snoopy Wrote: well like slyexe mentioned, you dont have enough ram to use all cards, so upgrade your ram or stick to 7 cards (i would assume that even 7 cards will/could throw an error, depending on the attack

Thank you for your answer.
I almost completely agree with you, but I have some doubts. On another 4x1080ti rig (11 GB VRAM each) Hashcat works great with any type of attack in nightmare mode. At the same time, the motherboard has only 8 GB of RAM. VRAM/RAM ratio is 4!!!

UPD: I've solved problem! The error gone when I removed the -O flag from arguments (Enable optimized kernels). Honestly, I do not know if there will be any limits with this, but it works! Hope this post can help somebody.

Hello my friend!

I have the same problem with my set (8 RTX3090 The-Distribution-Which-Does-Not-Handle-OpenCL-Well (Kali)) Tried to solve the problem in the same ways, but have no result

If I understand you correctly, you solve it, by deleting argument -O? But it turns off work with optimized Kernel, and speed drops by three times..
Find
Reply
vitiura Offline
Junior Member
**
Posts: 5
Threads: 1
Joined: Mar 2022
#10
04-01-2022, 01:05 PM
[/quote]

If I understand you correctly, you solve it, by deleting argument -O? But it turns off work with optimized Kernel, and speed drops by three times..
[/quote]

Hey! I think not always by three times, but I agree that it's not the most graceful solution, but it works. If you have any easy ways to fix problem, I'll be glad to watch it Smile
Find
Reply