Integer overflow detected while brute forcing

[kjs]

Indeed ?a x 11 BF is not feasible therefore it doesn't matter if it overflows or not which is why hashcat works as it works. You need to try more clever attacks based on wordlists.

Shouldn't the software evaluate any potential issues that could lead to overflows and return an error message in the first place?

This error message is from hashcat. It detected the overflow and reported it to the user. Not sure how the software can do it better than that.

Integer overflow detected in keyspace of mask: ?a?a?a?a?a?a?a?a?a?a?a

Yes, but can't you detect that an overflow would occur based on the input mask and the CPU architecture?