Hashing Performance

which hashing algo to use?

tested on athena

1. list off all alsos that open ssl has.

md5
mdc2  
rmd160
sha   
sha1  
sha224
sha256
sha384
sha512

2. create a shell script that times the hashing of a file 100 times for each method, then shuffle that file

sample:

/usr/bin/time openssl dgst -md5 ../smartOS-latest-USB.img.bz2 2>out-md5-15
/usr/bin/time openssl dgst -sha ../smartOS-latest-USB.img.bz2 2>out-sha-16
/usr/bin/time openssl dgst -sha384 ../smartOS-latest-USB.img.bz2 2>out-sha384-17
/usr/bin/time openssl dgst -sha224 ../smartOS-latest-USB.img.bz2 2>out-sha224-18
/usr/bin/time openssl dgst -mdc2 ../smartOS-latest-USB.img.bz2 2>out-mdc2-19
/usr/bin/time openssl dgst -sha256 ../smartOS-latest-USB.img.bz2 2>out-sha256-20
/usr/bin/time openssl dgst -sha256 ../smartOS-latest-USB.img.bz2 2>out-sha256-21
/usr/bin/time openssl dgst -sha ../smartOS-latest-USB.img.bz2 2>out-sha-23
/usr/bin/time openssl dgst -sha384 ../smartOS-latest-USB.img.bz2 2>out-sha384-24
/usr/bin/time openssl dgst -md5 ../smartOS-latest-USB.img.bz2 2>out-md5-25
/usr/bin/time openssl dgst -sha512 ../smartOS-latest-USB.img.bz2 2>out-sha512-26
/usr/bin/time openssl dgst -sha224 ../smartOS-latest-USB.img.bz2 2>out-sha224-27
/usr/bin/time openssl dgst -mdc2 ../smartOS-latest-USB.img.bz2 2>out-mdc2-28
/usr/bin/time openssl dgst -sha1 ../smartOS-latest-USB.img.bz2 2>out-sha1-29

3. for each method calculate the mean and stddev

created a script to gets stats (it's still in flux):

stats.pl - this doesn't work. Ack!

#!/usr/bin/perl

use Statistics::Basic::StdDev;

use Statistics::Basic qw(:all);

use Tie::File;

tie @array, 'Tie::File', "just-sha512" or die "failed to tie file ";

my $median = median(@array);
my $mean   = mean(@array);
my $variance = variance( @array );
my $stddev   = stddev(   @array );

#print "Median: $median\n";
#print "Mean $mean\n";
#print "Variance $variance\n";
#print "Stddev $stddev\n";

# from gnuplot docs:
# whisker plot: x box_min whisker_min whisker_high box_high
#
my $box_min       = $mean - $stddev;
my $box_high      = $mean + $stddev;
my $whisker_min   = $mean + ( 2 * stddev ) ;
my $whisker_high  = $mean - ( 2 * stddev ) ;

print "$mean $box_min $whisker_min $whisker_high $box_high\n";

doallstats.sh

echo -n "md5 "
grep user out-* | grep out-md5 | awk '{print $3}' | cut -b 3-7  | perl stats.pl
echo -n "mdc2 "
grep user out-* | grep out-mdc2 | awk '{print $3}' | cut -b 3-7  | perl stats.pl
echo -n "rmd160 "
grep user out-* | grep out-rmd160 | awk '{print $3}' | cut -b 3-7  | perl stats.pl
echo -n "sha "
grep user out-* | grep out-sha | awk '{print $3}' | cut -b 3-7  | perl stats.pl
echo -n "sha1 "
grep user out-* | grep out-sha1 | awk '{print $3}' | cut -b 3-7  | perl stats.pl
echo -n "sha224 "
grep user out-* | grep out-sha224 | awk '{print $3}' | cut -b 3-7  | perl stats.pl
echo -n "sha256 "
grep user out-* | grep out-sha256 | awk '{print $3}' | cut -b 3-7  | perl stats.pl
echo -n "sha384 "
grep user out-* | grep out-sha384 | awk '{print $3}' | cut -b 3-7  | perl stats.pl
echo -n "sha512 "
grep user out-* | grep out-sha512 | awk '{print $3}' | cut -b 3-7  | perl stats.pl

lets jsut do average for now:

for i in `cat digest.list `; do echo -n $i " "; grep user out-* | grep out-$i | awk '{print $3}' | cut -b 3-7  | awk '{sum+=$1} END { print "Average = ",sum/NR}'; done

md5  Average =  0.7401 <------THIS is the fastest
mdc2  Average =  46.8681
rmd160  Average =  2.2449
sha  Average =  4.06665
sha1  Average =  1.3751
sha224  Average =  3.6005
sha256  Average =  3.6019
sha384  Average =  6.7991
sha512  Average =  6.8885

4. send that to gnuplot:

hash_candles

this