Stupid Shell Tricks

From Federal Burro of Information
Jump to navigationJump to search


Shell invocation

/bin/sh 

-u - treat the use of unset variables as errors.
-x - show me execution.

PS for swap

ps -eo pcpu,pid,pmem,rss,vsz,comm --sort=-rss

PS1

export PS1='[\u@\h \t \w]\$ '


from: https://github.com/stntngo/nil/blob/master/bash_profile 

parse_git_branch() {
       git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/'
}
export PS1="\u \[\033[32m\]\${PS1X}\[\033[33m\]\$(parse_git_branch)\[\033[00m\]⚡"

Command line Tools

Diff

alias diffss='diff --width=180 --suppress-common-lines --side-by-side'


pwgen

pwgen -B -c -n -y
  • unabiguous
  • 1 capital
  • 1 number
  • 1 special char

Printing /proc/<pid>/environ

cat /proc/<pid>/environ | xargs --null --max-args=1 echo

cat /proc/(pidof process)/environ | tr '\0' '^M'

cat /proc/(pidof process)/environ | tr '\0' '<control-v><enter>'

HTTP response codes - filter

tail -f /var/log/httpd/access | awk '$9 !=200'

the print is implied, $9 happens to be where the http response code is in my log: 200 means OK, so it's show me the NOT OK stuff.

Grep failed wiki createaccount from apache log for iptables deny rule

ips=$(tail -200 access.log | grep title=Special:CreateAccount | awk '{print $1}' |sort | uniq | grep -v 142.126.179.34 | tr "\n" "," | sed 's/,$//')

/sbin/iptables -I INPUT 1 -s ${ips} -j DROP

or 

/sbin/iptables -R INPUT 1 -s ${ips} -j DROP

Disk usage report

du -x --max-depth=1 / | sort  -rn

Date

shell var of NOW

NOW=`date +%a.%d.%b.%Y`

result: 

Mon.04.Jul.2016

for DNS serial numbers: 

NOW=`date "+%Y%m%d%H%M%S"`

result: 

20160704174515

iso styleee! 

$ date --iso-8601=seconds
2017-06-06T11:14:05-0400

Javascript 

date "+%Y-%m-%dT%H:%M:%S.000Z"

some unix dates can't do millis, so just set to 000

shell var for shadowLastChange (ldap)

echo $((`date --utc --date "$1" +%s`/86400))


Days since the UNIX epoch

Return the number of days since the UNIX epoch using perl: 

perl -e 'printf qq{%d\n},time/86400'

Convert /etc/shadow lastchg to date

Convert the lastchg field in /etc/shadow to a date using GNU date: 

date -d "1 January 1970 + lastchg days"

Set timezone

set it system wide: 

sudo timedatectl set-timezone Europe/Brussels

for your shell 

export TZ=timezone


Links

fake_tomcat.sh

ARGV="$@"
if [ "x$ARGV" = "x" ] ; then
        echo usage: all start, stop, reload, abort, flush, or check
        exit
fi

case $# in
0)      echo 'Usage: ./snapshot <CPE name> (ie, ./snapshot YCDECUBC)' 1>&2; exit 2
esac

trap 'echo "";exit 3' 2 15
trap 'echo fake_tomcat.sh caught 1 HUP \-\> ok bye\! ; exit 3' 1
trap 'echo fake_tomcat.sh caught 3 QUIT \-\> ok bye\! ; exit 3' 3
trap 'echo fake_tomcat.sh caught 9 KILL \-\> ok bye\! ; exit 3' 9
trap 'echo fake_tomcat.sh caught 15 TERM \-\> ok bye\! ; exit 3' 15

TMPFILE=`mktemp /tmp/$0.XXXXXX` || exit 1

To move/duplicate filesystems I have a favorite way to do it locally:
# cd $filesystem_to_duplicate
# find . -print | cpio -pvdm /mnt

...where /mnt is the new filesystem/slice.
To duplicate/move across the network do it like this:
# cd $filesystem_to_duplicate
# tar cf - . | ssh otherhost "cd /$new_filesystem ; tar xf -"

function waitfor {
        if [ $# -lt 1 ] ; then
                echo "nothing to wait for"
        else
                echo "ok I'll wait"
                echo Still running = 1
                STILL_RUNNING=1
                while [ $STILL_RUNNING -gt 0 ]
                        do
                        STILL_RUNNING=`ps -auwwwx | grep $1 | grep -v grep | wc -l`
                        echo STILL_RUNNING = $STILL_RUNNING
                        sleep 1
                        echo waiting...
                        done
        fi
        echo $1
}
| tr '\n' ','


TimerOn()
{
  sleep $TIMELIMIT && kill -s 14 $$ &
  # Waits 3 seconds, then sends sigalarm to script.
}

Int14Vector()
{
  answer="TIMEOUT"
  PrintAnswer
  exit 14
}

trap Int14Vector 14

While loops for Fun and Profit

this script runs until you stop it. It collects file handle usage on a server putting the results in a file in the form: 

<timestamp> <total allocated> <free> <maxpossible>

the last three field are from the /proc fs: 

3391    969     52427
|	 |       |
|	 |       |
|        |       maximum open file descriptors
|       total free allocated file descriptors
total allocated file descriptors
(the number of file descriptors allocated since boot)

scripts: 

while true;
do
 echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/s4.t55.file-nr.2008040300;
 cat /proc/sys/fs/file-nr >> /home/dathornton/s4.t55.file-nr.2008040300;
 sleep 5;
done

You MUST MUST MUST put the sleep in there or "Bad Things Will Happen"(tm), it will loop too fast and you device could crash.

or in one line: 

while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/servername.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/servername.file-nr.2008040300; sleep 5; done

while loop over multi column input

tl;dr: 

k get cm -A | grep ggr | awk '{print $1 , $2}' | while read ns cm; do echo ns is $ns; echo cm is $cm ; done <&0

You are looking for something in a kubernetes configmap ( cm ). You query all configmaps for all names spaces 

k get cm -A

and grep out a term. You don't want all those columns, you want the first two: namespace and configmap name. 

grep ggr | awk '{print $1 , $2}'

now comes the fun part, you do while loop with a read, and by sending it &0 ( the stdin file descriptor ) , you can read the pipe you were given. 

while read ns cm;
do
echo ns is $ns;
echo cm is $cm ;
done <&0

until loops

until PGPASSWORD=$POSTGRES_PASSWORD psql -h "$host" -U "postgres" -c '\q'; do
  >&2 echo "Postgres is unavailable - sleeping"
  sleep 1
done

Traps

#!/bin/bash
# traptest.sh

trap "echo Booh!" SIGINT SIGTERM
trap "echo Kill" SIGKILL
echo "pid is $$"

while: # This is the same as "while true".
do
 sleep 5 # This script is not really doing anything.
done

sort

Sorting Hostnames

service<instance>.location<instance>.fart.gas.bum

sort -t . -k2.2,1.1n -k1n

Sorting Ip Addresses

By Last three octets: 

sort -t . -k 2,2n -k 3,3n -k 4,4n serverlist| more

epoch

#!/bin/sh
date -d "1970-01-01 UTC $1 seconds"

bash

disable bell

echo "set bell-style none" >> ~/.inputrc

timestamps in history

export HISTTIMEFORMAT='%F %T '

Awk

show me lines that don't have that in field 2 

awk ' $2 !~ "[A-Za-z]" {print $0}'

who me lines that have less than 2 field: 

awk ' NF < 2 {print $0}'

or shorter: 

awk 'NF<2'

gimme field 2 - end (squash the first field then strip the leading space.) 

awk '{$1=""; sub(/^space:*/,""); print}'


"Crontab last Saturday of the month" problem

  • Client had a problem where they wanted a script run on a server at 11 pm on the last Saturday of the month
  • the crontab that was originally devised was:

#0 11 1-6 * 6 /home/smsadmin/CPU_util/runall.sh *snip*

  • this ended up running at 11am from the 1st of the month to the 6th of the month, *as well as* every Saturday, NOT on Saturday as long as it was only the 1st to the 6th (this is somewhat unintuitive).
  • to work around this, we wrote a quick wrapper 1-liner script that returned true if the same day next week had a different month than this month:

0 11 * * 6 [ $(date +\%m) != $(date +\%m -d "next week") ] && <rest of the command to run if the test passed>

find

find recent large stuff on / 

find / -xdev -mtime -1 -size +10M | xargs ls -lad

Which process is on which cpu?

ps -eo psr,pid,tid,nlwp,tty,comm

or sorted by processor: 

ps -eo psr,pid,tid,nlwp,tty,comm | sort -n

ps doesn't seem to want to sort on processor. These don't work: 

ps --sort=psr -eo psr,pid,tid,nlwp,tty,comm
ps --sort psr -eo psr,pid,tid,nlwp,tty,comm
ps kpsr -eo psr,pid,tid,nlwp,tty,comm

how many processes on which cpu? 

ps h -eo psr | sort | uniq -c | awk '{printf "%4s %4s\n", $2 ,$1}' | sort -n

Sed

grep out one variable with sed

cat /var/log/zimbra.log | sed -n 's/.*client=//p' | sort |uniq -c|sort -rn | head -30

remove terminal colours

sed 's/\x1b\[[0-9;]*m//g'


set / unset / empty

   +----------------------+------------+-----------------------+-----------------------+
   |   if VARIABLE is:    |    set     |         empty         |        unset          |
   +----------------------+------------+-----------------------+-----------------------+
 - |  ${VARIABLE-default} | $VARIABLE  |          ""           |       "default"       |
 = |  ${VARIABLE=default} | $VARIABLE  |          ""           | $(VARIABLE="default") |
 ? |  ${VARIABLE?default} | $VARIABLE  |          ""           |       exit 127        |
 + |  ${VARIABLE+default} | "default"  |       "default"       |          ""           |
   +----------------------+------------+-----------------------+-----------------------+
:- | ${VARIABLE:-default} | $VARIABLE  |       "default"       |       "default"       |
:= | ${VARIABLE:=default} | $VARIABLE  | $(VARIABLE="default") | $(VARIABLE="default") |
:? | ${VARIABLE:?default} | $VARIABLE  |       exit 127        |       exit 127        |
:+ | ${VARIABLE:+default} | "default"  |          ""           |          ""           |
   +----------------------+------------+-----------------------+-----------------------+

ref: https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash

parallel processes , ghetto style

{
  xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF'
sleep 1; echo 1
sleep 2; echo 2
sleep 3; echo 3
echo 4
EOF
} &

intercepting options with bash

Something runs a program, it's hard coded it to run it a certain way, you want to run it a different way, say with different otions.

swap the binary out, and put a small shell script in the way:

in this exmaple the original runs chrome drvier with --port=XX and --log-path=/dev/null.

You want to keep the port, but change the logging.

mv /usr/bin/chromedriver /usr/bin/chromedriver.binary

heredoc user:user 755 /usr/bin/chromedriver <<< 

#!/bin/bash

re="--port=([0-9]*?) "
if [[ $@ =~ $re ]]; then
 port=${BASH_REMATCH[1]}
 /usr/bin/chromedriver.binary --port=${port} --log-path=/tmp/chromedriver.log --log-level=INFO --append-log
else
 echo "no port specified"
 exit 1
fi
Retrieved from "https://wiki.quadratic.net/index.php?title=Stupid_Shell_Tricks&oldid=31497"