there is no such additional cache for storing cracked hashes except the potfile.
Maybe you are confusing the startup time with the OpenCL kernel compilation time, which only happens once for each kernel and those files are cached within the kernels/ directory.
I would highly recommend that you are not comparing with very low runtimes (less than a dozens of minutes), because otherwise some minor factors like the initialization/startup time can't be ignored (e.g. the more devices you have, the more devices need to be initialized etc, all of this actually small times are not really good to take into account and are negligible if you let the benchmarks/tests run longer)
BTW: there is also the hashcat.dictstat2 file (dictionary cache) that could take some time to generate for each different dictionary... this is also something that should be very obvious to notice, because hashcat tells you what it is doing when it is generating the dictstat etc
Maybe you are confusing the startup time with the OpenCL kernel compilation time, which only happens once for each kernel and those files are cached within the kernels/ directory.
I would highly recommend that you are not comparing with very low runtimes (less than a dozens of minutes), because otherwise some minor factors like the initialization/startup time can't be ignored (e.g. the more devices you have, the more devices need to be initialized etc, all of this actually small times are not really good to take into account and are negligible if you let the benchmarks/tests run longer)
BTW: there is also the hashcat.dictstat2 file (dictionary cache) that could take some time to generate for each different dictionary... this is also something that should be very obvious to notice, because hashcat tells you what it is doing when it is generating the dictstat etc