known mask structure
#1
The strings that are being hashed (SHA256) in my case follow this rule:

xxxxxxxxxxxx|1:1:1

In words: It is twelve random hex characters (0-9,ABCDEF) followed by a pipe, followed by a timestamp of the format h:m:s. Hours range from 0-23, mins and secs from 0-59. There is no padding for numbers less than 10 (ie, what would normally be written as 13:07:01 is represented here as 13:7:1, meaning there could be one OR two digits).

To get the hex chars, I can set the custom-charset 1 to ?dabcdef, but how do I define a mask that contains a pipe, colons, and different numbers of digits for the timestamp?

Using Windows 7 and CudaHashcat64
#2
Read here: http://hashcat.net/wiki/doku.php?id=mask_attack
You could easily do this w/ a combination of custom charsets ( http://hashcat.net/wiki/doku.php?id=mask...m_charsets ), some masks and the .hcmask file feature http://hashcat.net/wiki/doku.php?id=mask...mask_files

For instance your .hcmask file could look something like this (not tested but you may get the idea):

Code:
$ cat my.hcmask
?dABCDEF,?1?1?1?1?1?1?1?1?1?1?1?1|?d:?d:?d
?dABCDEF,12345,?1?1?1?1?1?1?1?1?1?1?1?1|?d:?d:?2?d
?dABCDEF,12345,?1?1?1?1?1?1?1?1?1?1?1?1|?d:?2?d:?d
?dABCDEF,12345,?1?1?1?1?1?1?1?1?1?1?1?1|?d:?2?d:?2?d
?dABCDEF,?1?1?1?1?1?1?1?1?1?1?1?1|1?d:?d:?d
?dABCDEF,12345,?1?1?1?1?1?1?1?1?1?1?1?1|1?d:?d:?2?d
?dABCDEF,12345,?1?1?1?1?1?1?1?1?1?1?1?1|1?d:?2?d:?d
?dABCDEF,12345,?1?1?1?1?1?1?1?1?1?1?1?1|1?d:?2?d:?2?d
?dABCDEF,0123,?1?1?1?1?1?1?1?1?1?1?1?1|2?2:?d:?d
?dABCDEF,0123,12345,?1?1?1?1?1?1?1?1?1?1?1?1|2?2:?d:?3?d
?dABCDEF,0123,12345,?1?1?1?1?1?1?1?1?1?1?1?1|2?2:?3?d:?d
?dABCDEF,0123,12345,?1?1?1?1?1?1?1?1?1?1?1?1|2?2:?3?d:?3?d

You need several lines because you have several limitations...=> several different masks

Everything "character" that should be literally in your mask you just include directly in the mask, as shown above.

Note: you said that the mask is upper-case hex 0-9A-F but than used ?dabcdef...
If it is lowercase hex then you need to change the first custom charset to ?dabcdef

You can use this mask file as you would use any other mask w/ -a 3 my.hcmask

Furthermore, note that the keyspace is quite huge, having 12 random hex chars isn't (depending on the hash algo) that easy/fast to bruteforce