Packages for Linux distributions
#1
Anyone here experienced with creating and maintaining Linux packages?
#2
I am, but you knew that.

The problem with packaging hashcat right now, as dropdead and I were discussing, is that hashcat doesn't exactly adhere to the FHS. Basically the hashcat directory is a working directory, and that's rather contradictory to the FHS.

Ideally before we go down the road of Linux packaging, we would re-organize the file structure, such that binaries are in /usr/bin; kernels, rule files, etc. are in /usr/share/hashcat; and user files (pot, restore, induct) are in the user's home directory, e.g. ~/.hashcat).

Otherwise, we basically just have to stuff everything in /usr/share/hashcat and have the user run from that directory as root.
#3
Yes, that's correct. I just wanted to start the discussion. I think what we need is a list of paths we'd need to change to make it more compatible.

Btw. this _maybe_ has influence on the windows port, too, but it's not a must. For windows I think people like it as it is (do everything in the installation directory).
#4
Ok, here's an example of a package manifest for oclHashcat 64bit (truncated in places where repetitive)

Code:
/usr/bin/oclHashcat64.bin
/usr/share/docs/oclHashcat
/usr/share/docs/oclHashcat/changes.txt
/usr/share/docs/oclHashcat/contact.txt
/usr/share/oclHashcat/charsets
/usr/share/oclHashcat/charsets/combined
/usr/share/oclHashcat/charsets/combined/Bulgarian.hcchr
/usr/share/oclHashcat/charsets/combined/Castilian.hcchr
/usr/share/oclHashcat/charsets/special
/usr/share/oclHashcat/charsets/special/Castilian
/usr/share/oclHashcat/charsets/special/Castilian/es-ES_ISO-8859-1-special.hcchr
/usr/share/oclHashcat/charsets/special/Castilian/es-ES_ISO-8859-15-special.hcchr
/usr/share/oclHashcat/charsets/standard
/usr/share/oclHashcat/charsets/standard/Bulgarian
/usr/share/oclHashcat/charsets/standard/Bulgarian/bg_ISO-8859-5.hcchr
/usr/share/oclHashcat/charsets/standard/Bulgarian/bg_KOI8-R.hcchr
/usr/share/oclHashcat/examples/example.dict
/usr/share/oclHashcat/examples/example0.hash
/usr/share/oclHashcat/examples/example400.hash
/usr/share/oclHashcat/examples/example500.hash
/usr/share/oclHashcat/examples/oclExample0.sh
/usr/share/oclHashcat/examples/oclExample400.sh
/usr/share/oclHashcat/examples/oclExample500.sh
/usr/share/oclHashcat/extra
/usr/share/oclHashcat/extra/rules_optimize
/usr/share/oclHashcat/extra/rules_optimize/rules_optimize.bin
/usr/share/oclHashcat/extra/tab_completion
/usr/share/oclHashcat/extra/tab_completion/install
/usr/share/oclHashcat/extra/tab_completion/oclHashcat64.sh
/usr/share/oclHashcat/hashcat.hcstat
/usr/share/oclHashcat/kernels
/usr/share/oclHashcat/kernels/4098
/usr/share/oclHashcat/kernels/4098/amp_a0_v1.llvmir
/usr/share/oclHashcat/kernels/4098/m00000_a0.VLIW1.llvmir
/usr/share/oclHashcat/masks
/usr/share/oclHashcat/masks/rockyou-1-60.hcmask
/usr/share/oclHashcat/masks/rockyou-2-1800.hcmask
/usr/share/oclHashcat/rules
/usr/share/oclHashcat/rules/Incisive-leetspeak.rule
/usr/share/oclHashcat/rules/InsidePro-HashManager.rule
/usr/share/oclHashcat/rules/hybrid
/usr/share/oclHashcat/rules/hybrid/append_d.rule
/usr/share/oclHashcat/rules/hybrid/append_ds.rule

And then there's some things that should go along with this,

1. Any file that hashcat writes to (potfile, session files, etc.) should go in ~/.hashcat. I think both hashcat & oclHashcat can share this directory.

2. Should also have ~/.hashcat/rules, ~/.hashcat/masks, etc. for local/user-created files.

3. Default internal search path for rules, masks, charsets, etc. including /usr/share/oclHashcat and ~/.hashcat to help keep command lines short. Can either always search the internal paths first, followed by cwd, or if we detect a file path (e.g. preceding / or ./) skip searching path/cwd.
#5
I think we should distinguish this task in 2 separate goals:
1. what devs of hashcat/oclHashcat need to change/do to facilitate creation of packages.
2. what the people need to do/know whose goal is to prepare the packages (aka maintainers).

Maybe we should discuss this in 2+ different forum threads in the future (such that there exists a general "packaging" thread, a "package maintainer" thread and an "allow packaging" - only for changes in hashcat/oclHashcat source code - thread).

Here below I only discuss what hashcat and oclHashcat need to change (not what the distro maintainers need to do, i.e. not where THEY - maintainers - need to put the files).
Therefore, my discussion here is only about where oclHashcat/hashcat should try to find the files (and which one it should prefer if there are multiple options or files with same name in different locations).

RULES:
1. if the binaries are being run on a non-unix system (e.g. windows. But should we limit it to Linux? what about osx ?), only use [cwd] for every path (as it was before)
2. always prefer current working directory [cwd] if the binary is directly executed from the current directory (e.g. "./oclHashcat64.bin")
3. if the user just runs "hashcat-cli64.bin" / "oclHashcat64.bin" etc (i.e. by using the PATH variable), do not search in [cwd]
4. developers should prefer to use the non-packaged version (i.e. everything is relative to [cwd]), this also implies that kernels (e.g. when disabling BINARY_KERNEL) should only be in [cwd]
5. for temporary files always prefer the /tmp/hashcat/ folder if it exists

6. a not so important but additional feature we could add is to help the user (directly by modifying the hashcat/oclHashcat source code?) to find .rule/.hcmask files etc for instance at ~/.hashcat/ with some kind of inbuilt tab-completion (or wouldn't it be better to modify /extra/tab_completion/oclHashcat64.sh to do so?) ???
7. (what about paths on OSX with cpu hashcat?) ???

TYPE OF FOLDERS:

profile_folder:
- location: [cwd] or (~/.hashcat/)  (or should we use ~/.hashcat/oclHashcat/ and ~/.hashcat/hashcat/)
- list of files:
 1. [session].pot
 2. [session].log
 3. [session].restore (and [session].restore.new)
 4. [session].outfiles (DIR)
 5. [session].induct   (DIR)
 6. [session].induct/loopback.[TIMESTAMP]_[RANDOM_NUMBER]
 7. *.dictstat
 8. *.hcstat
 9. kernels/4098/*.kernel ???

packaged_file_folder:
- location: [cwd] or /usr/share/hashcat/
- list of files:
 1. kernels/4098/*.llvmir
 2. kernels/4318/*.cubin

temporary_file_folder:
- location: /tmp/hashcat/ (preferred!?) or [cwd]
- list of files:
 1. temp files used for --remove ([hashfile].old and [hashfile].new)

There might be the need to adapt this guide a little bit. Any suggestion is welcome.
#6
I agree with #1-4.

For #5, I'd much rather prefer to see ~/.hashcat/tmp over /tmp/hashcat, if for no other reason than to better enable multiple users to share the same system.

I don't see a reason at this point why hashcat & oclHashcat can't share a working directory, would certainly be a lot more convenient.
#7


Suggestions from epix:
  • I disagree partially, we need to distinguish between hashcat and oclHashcat on /usr/share/doc/ because of colliding filenames in the docs/* folder
  • I disagree to have ~/.hashcat/rules or ~/.hashcat/masks, at least not automatically generated. Of course a user can create this by his own or symlink to /usr/share/hashcat/rules 
  • I disagree to Default internal search path.  I can see it's benefits, but the price for the increased complexity is too high for me.


Suggestions from phil:
  • I agree, windows stays as it is
  • I like the wording profile folder
  • I agree with epix for #5 the tmp should be part of the profile folder, but I don't see the need for such a folder at all (see structure below)
  • Let's call us the packaged_file_folder = install folder
  • 1.8 (the .hcstat) should be part of install folder
  • 1.9 = yes
  • We should not try to find out if the binary was called using ./ or via PATH, there should be a different way of finding out whatever type of information this would give us


Some more thoughts:
  • Make it simple!
  • Make it easy for package maintainer
  • We have 3 "types" of folder, 2 of them were named already "profile folder" and "install folder" and I'll add another one called "session folder"
  • Here's some structure I have in mind now:


Shared, those files go in /usr/share/hashcat for both hashcat and oclHashcat:
  • /usr/share/hashcat/hashcat.hcstat
  • /usr/share/hashcat/charsets/*
  • /usr/share/hashcat/kernels/*
  • /usr/share/hashcat/masks/*
  • /usr/share/hashcat/rules/*


Not shared:

hashcat:
  • /usr/bin/*.bin 
  • /usr/share/doc/hashcat/ (stuff from docs/*)
oclHashcat:
  • /usr/bin/*.bin 
  • /usr/share/doc/oclHashcat/ (stuff from docs/* and example* files)
  • /usr/share/doc/oclHashcat/extra/* 


The following files are for testing. I'd say we don't copy them at all, they are _only_ part of the source tree:

hashcat:
  • examples/*
oclHashcat:
  • test/* 


Finally we need to discuss the profile folder. My suggestion:
  • ~/.hashcat/$session/hashcat.log
  • ~/.hashcat/$session/hashcat.restore
  • ~/.hashcat/$session/hashcat.restore.new
  • ~/.hashcat/$session/outfiles/
  • ~/.hashcat/$session/induct/
  • ~/.hashcat/kernels/4098/*.kernel
  • ~/.hashcat/hashcat.pot
  • ~/.hashcat/hashcat.dictstat
#8
(12-09-2015, 12:06 AM)atom Wrote: I disagree partially, we need to distinguish between hashcat and oclHashcat on /usr/share/doc/ because of colliding filenames in the docs/* folder

Actually I didn't suggest that anything in /usr/share should be shared between hashcat and oclHashcat. I said I see no reason that the working directory (i.e., ~/.hashcat) cannot be shared between the two cats. Paths in /usr/share absolutely should be unique. 

(12-09-2015, 12:06 AM)atom Wrote: Shared, those files go in /usr/share/hashcat for both hashcat and oclHashcat:

And this ties in with what I just wrote: these must not (and actually cannot) be shared at all unless you plan to break all of these things out of hashcat/oclHashcat and create an entirely separate hashcat-common package or something. You cannot have two packages that write to the same file path.
#9
Indeed well hashcat-common would be nice, but it's too much at once I think.

So here's the updated suggestion:



hashcat:
  • /usr/bin/*.bin
  • /usr/share/doc/hashcat/ (stuff from docs/*)
  • /usr/share/hashcat/hashcat.hcstat
  • /usr/share/hashcat/charsets/*
  • /usr/share/hashcat/kernels/*
  • /usr/share/hashcat/masks/*
  • /usr/share/hashcat/rules/*

oclHashcat (also oclHashcat in the folder names for cudaHashcat):
  • /usr/bin/*.bin
  • /usr/share/doc/oclHashcat/ (stuff from docs/* and example* files)
  • /usr/share/doc/oclHashcat/extra/*
  • /usr/share/oclHashcat/hashcat.hcstat
  • /usr/share/oclHashcat/charsets/*
  • /usr/share/oclHashcat/kernels/*
  • /usr/share/oclHashcat/masks/*
  • /usr/share/oclHashcat/rules/*
#10
That list looks good