I'm overlooking something when implementing new algorithm: please help

[John Doe]

Just a guess:

With a key 0xAAAAAAAAAAAAAAAA you do not have to care about Little endian/Big endian, because the key has always the same byte value. Try again with 0xBBBBBBBBBBBBBBBB and update your Ciphertext and if we're on the right track it will still work. 

0xBBBBBBBBBBBBBBBB  works.

After that, try the key 0xAAAAAAAABCAAAAAA instead of 0xAAAAAAAAAAAAAABC with the original Ciphertext. If it still works it's for sure LE/BE handling.

Works, bingo! Quick fix:

Code:

u32x convertEndian(u32x num)
{
   u32x b0,b1,b2,b3;
   b0= (num & 0x000000FF)>>0;
   b1= (num & 0x0000FF00)>>8;
   b2= (num & 0x00FF0000)>>16;
   b3= (num & 0xFF000000)>>24;
   num= (b0<<24) | (b1<<16) | (b2<<8) | (b3<<0) ;
   return num;
}

However I see some endian related functions in e.g. inc_common.cl. What would be the best / fastest / cleanest option for endian switching for a 64 bits value that's stored in two 32-bits values (u32x key[2])?

Also note what epixoip said or, as an alternative for developer, use the

int force_jit_compilation = -1;

and set it to:

int force_jit_compilation = XXX;

Where XXX is your new hash mode. This will also enable you to use printf() on some OpenCL runtime from within the kernel.

Very useful tip thanks!