Stupid Shell Tricks: Difference between revisions

From Federal Burro of Information
Jump to navigationJump to search
Line 167: Line 167:
  | |      |
  | |      |
  |      |      maximum open file descriptors
  |      |      maximum open file descriptors
  |       total free allocated file descriptors
  |       total free allocated file descriptors
  total allocated file descriptors
  total allocated file descriptors
  (the number of file descriptors allocated since boot)
  (the number of file descriptors allocated since boot)

Revision as of 18:26, 2 May 2019

Shell invocation

/bin/sh

-u - treat the use of unset variables as errors.
-x - show me execution.

PS for swap

ps -eo pcpu,pid,pmem,rss,vsz,comm --sort=-rss

PS1

export PS1='[\u@\h \t \w]\$ '

Command line Tools

Diff

alias diffss='diff --width=180 --suppress-common-lines --side-by-side'


pwgen

pwgen -B -c -n -y
  • unabiguous
  • 1 capital
  • 1 number
  • 1 special char

Printing /proc/<pid>/environ

cat /proc/<pid>/environ | xargs --null --max-args=1 echo
cat /proc/(pidof process)/environ | tr '\0' '^M'
cat /proc/(pidof process)/environ | tr '\0' '<control-v><enter>'

HTTP response codes - filter

tail -f /var/log/httpd/access | awk '$9 !=200'

the print is implied, $9 happens to be where the http response code is in my log: 200 means OK, so it's show me the NOT OK stuff.

Disk usage report

du -x --max-depth=1 / | sort  -rn

shell var of NOW

NOW=`date +%a.%d.%b.%Y`

result:

Mon.04.Jul.2016

for DNS serial numbers:

NOW=`date "+%Y%m%d%H%M%S"`

result:

20160704174515

iso styleee!

$ date --iso-8601=seconds
2017-06-06T11:14:05-0400

shell var for shadowLastChange (ldap)

echo $((`date --utc --date "$1" +%s`/86400))


Days since the UNIX epoch

Return the number of days since the UNIX epoch using perl:

$ perl -e 'printf qq{%d\n},time/86400'

Convert /etc/shadow lastchg to date

Convert the lastchg field in /etc/shadow to a date using GNU date:

$ date -d "1 January 1970 + lastchg days"

Links

fake_tomcat.sh

ARGV="$@"
if [ "x$ARGV" = "x" ] ; then
        echo usage: all start, stop, reload, abort, flush, or check
        exit
fi

case $# in
0)      echo 'Usage: ./snapshot <CPE name> (ie, ./snapshot YCDECUBC)' 1>&2; exit 2
esac

trap 'echo "";exit 3' 2 15
trap 'echo fake_tomcat.sh caught 1 HUP \-\> ok bye\! ; exit 3' 1
trap 'echo fake_tomcat.sh caught 3 QUIT \-\> ok bye\! ; exit 3' 3
trap 'echo fake_tomcat.sh caught 9 KILL \-\> ok bye\! ; exit 3' 9
trap 'echo fake_tomcat.sh caught 15 TERM \-\> ok bye\! ; exit 3' 15

TMPFILE=`mktemp /tmp/$0.XXXXXX` || exit 1

To move/duplicate filesystems I have a favorite way to do it locally:
# cd $filesystem_to_duplicate
# find . -print | cpio -pvdm /mnt

...where /mnt is the new filesystem/slice.
To duplicate/move across the network do it like this:
# cd $filesystem_to_duplicate
# tar cf - . | ssh otherhost "cd /$new_filesystem ; tar xf -"

function waitfor {
        if [ $# -lt 1 ] ; then
                echo "nothing to wait for"
        else
                echo "ok I'll wait"
                echo Still running = 1
                STILL_RUNNING=1
                while [ $STILL_RUNNING -gt 0 ]
                        do
                        STILL_RUNNING=`ps -auwwwx | grep $1 | grep -v grep | wc -l`
                        echo STILL_RUNNING = $STILL_RUNNING
                        sleep 1
                        echo waiting...
                        done
        fi
        echo $1
}
| tr '\n' ','


TimerOn()
{
  sleep $TIMELIMIT && kill -s 14 $$ &
  # Waits 3 seconds, then sends sigalarm to script.
}

Int14Vector()
{
  answer="TIMEOUT"
  PrintAnswer
  exit 14
}

trap Int14Vector 14

While loops for Fun and Profit

this script runs until you stop it. It collects file handle usage on a server putting the results in a file in the form:

<timestamp> <total allocated> <free> <maxpossible>

the last three field are from the /proc fs:

3391    969     52427
|	 |       |
|	 |       |
|       |       maximum open file descriptors
|       total free allocated file descriptors
total allocated file descriptors
(the number of file descriptors allocated since boot)

scripts:

while true;
do
 echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/s4.t55.file-nr.2008040300;
 cat /proc/sys/fs/file-nr >> /home/dathornton/s4.t55.file-nr.2008040300;
 sleep 5;
done

You MUST MUST MUST put the sleep in there or "Bad Things Will Happen"(tm), it will loop too fast and you device could crash.

or in one line:

while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/servername.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/servername.file-nr.2008040300; sleep 5; done

until loops

until PGPASSWORD=$POSTGRES_PASSWORD psql -h "$host" -U "postgres" -c '\q'; do
  >&2 echo "Postgres is unavailable - sleeping"
  sleep 1
done

Traps

#!/bin/bash
# traptest.sh

trap "echo Booh!" SIGINT SIGTERM
trap "echo Kill" SIGKILL
echo "pid is $$"

while: # This is the same as "while true".
do
 sleep 5 # This script is not really doing anything.
done

sort

Sorting Hostnames

service<instance>.location<instance>.fart.gas.bum
sort -t . -k2.2,1.1n -k1n

Sorting Ip Addresses

By Last three octets:

sort -t . -k 2,2n -k 3,3n -k 4,4n serverlist| more

epoch

#!/bin/sh
date -d "1970-01-01 UTC $1 seconds"

bash

disable bell

echo "set bell-style none" >> ~/.inputrc

timestamps in history

export HISTTIMEFORMAT='%F %T '

Awk

show me lines that don't have that in field 2

awk ' $2 !~ "[A-Za-z]" {print $0}'

who me lines that have less than 2 field:

awk ' NF < 2 {print $0}'

or shorter:

awk 'NF<2'

gimme field 2 - end (squash the first field then strip the leading space.)

awk '{$1=""; sub(/^space:*/,""); print}' 


"Crontab last Saturday of the month" problem

  • Client had a problem where they wanted a script run on a server at 11 pm on the last Saturday of the month
  • the crontab that was originally devised was:

#0 11 1-6 * 6 /home/smsadmin/CPU_util/runall.sh *snip*

  • this ended up running at 11am from the 1st of the month to the 6th of the month, *as well as* every Saturday, NOT on Saturday as long as it was only the 1st to the 6th (this is somewhat unintuitive).
  • to work around this, we wrote a quick wrapper 1-liner script that returned true if the same day next week had a different month than this month:

0 11 * * 6 [ $(date +\%m) != $(date +\%m -d "next week") ] && <rest of the command to run if the test passed>

find

find recent large stuff on /

find / -xdev -mtime -1 -size +10M | xargs ls -lad

Which process is on which cpu?

ps -eo psr,pid,tid,nlwp,tty,comm

or sorted by processor:

ps -eo psr,pid,tid,nlwp,tty,comm | sort -n

ps doesn't seem to want to sort on processor. These don't work:

ps --sort=psr -eo psr,pid,tid,nlwp,tty,comm
ps --sort psr -eo psr,pid,tid,nlwp,tty,comm
ps kpsr -eo psr,pid,tid,nlwp,tty,comm

how many processes on which cpu?

ps h -eo psr | sort | uniq -c | awk '{printf "%4s %4s\n", $2 ,$1}' | sort -n

Sed

grep out one variable with sed

cat /var/log/zimbra.log | sed -n 's/.*client=//p' | sort |uniq -c|sort -rn | head -30

remove terminal colours

sed 's/\x1b\[[0-9;]*m//g'

set / unset / empty

   +----------------------+------------+-----------------------+-----------------------+
   |   if VARIABLE is:    |    set     |         empty         |        unset          |
   +----------------------+------------+-----------------------+-----------------------+
 - |  ${VARIABLE-default} | $VARIABLE  |          ""           |       "default"       |
 = |  ${VARIABLE=default} | $VARIABLE  |          ""           | $(VARIABLE="default") |
 ? |  ${VARIABLE?default} | $VARIABLE  |          ""           |       exit 127        |
 + |  ${VARIABLE+default} | "default"  |       "default"       |          ""           |
   +----------------------+------------+-----------------------+-----------------------+
:- | ${VARIABLE:-default} | $VARIABLE  |       "default"       |       "default"       |
:= | ${VARIABLE:=default} | $VARIABLE  | $(VARIABLE="default") | $(VARIABLE="default") |
:? | ${VARIABLE:?default} | $VARIABLE  |       exit 127        |       exit 127        |
:+ | ${VARIABLE:+default} | "default"  |          ""           |          ""           |
   +----------------------+------------+-----------------------+-----------------------+

ref: https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash

parallel processes , ghetto style

{
  xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF'
sleep 1; echo 1
sleep 2; echo 2
sleep 3; echo 3
echo 4
EOF
} &