Stupid Shell Tricks: Difference between revisions

From Federal Burro of Information
Jump to navigationJump to search
 
(46 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[Category:Script]]
== Shell invocation ==
== Shell invocation ==


Line 5: Line 7:
  -x - show me execution.
  -x - show me execution.


== pwgen ==
== PS for swap ==
 
ps -eo pcpu,pid,pmem,rss,vsz,comm --sort=-rss
 
== PS1 ==
 
export PS1='[\u@\h \t \w]\$ '
 
 
from: https://github.com/stntngo/nil/blob/master/bash_profile
 
parse_git_branch() {
        git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/'
}
export PS1="\u \[\033[32m\]\${PS1X}\[\033[33m\]\$(parse_git_branch)\[\033[00m\]⚡"
 
== Command line Tools ==
 
=== Diff ===
 
alias diffss='diff --width=180 --suppress-common-lines --side-by-side'
 
 
=== pwgen ===


  pwgen -B -c -n -y
  pwgen -B -c -n -y
Line 13: Line 38:
* 1 number
* 1 number
* 1 special char
* 1 special char
== Printing /proc/<pid>/environ ==
cat /proc/<pid>/environ | xargs --null --max-args=1 echo
cat /proc/(pidof process)/environ | tr '\0' '^M'
cat /proc/(pidof process)/environ | tr '\0' '<control-v><enter>'


== HTTP response codes - filter ==
== HTTP response codes - filter ==
Line 19: Line 52:


the print is implied, $9 happens to be where the http response code is in my log: 200 means OK, so it's show me the NOT OK stuff.
the print is implied, $9 happens to be where the http response code is in my log: 200 means OK, so it's show me the NOT OK stuff.
== Grep failed wiki createaccount from apache log for iptables deny rule ==
ips=$(tail -200 access.log | grep title=Special:CreateAccount | awk '{print $1}' |sort | uniq | grep -v 142.126.179.34 | tr "\n" "," | sed 's/,$//')
/sbin/iptables -I INPUT 1 -s ${ips} -j DROP
or
/sbin/iptables -R INPUT 1 -s ${ips} -j DROP


== Disk usage report ==
== Disk usage report ==
Line 24: Line 67:
  du -x --max-depth=1 / | sort  -rn
  du -x --max-depth=1 / | sort  -rn


== shell var of NOW ==
== Date ==
 
=== shell var of NOW ===


  NOW=`date +%a.%d.%b.%Y`
  NOW=`date +%a.%d.%b.%Y`
result:
Mon.04.Jul.2016
for DNS serial numbers:
NOW=`date "+%Y%m%d%H%M%S"`
result:
20160704174515
iso styleee!
$ date --iso-8601=seconds
2017-06-06T11:14:05-0400
Javascript
date "+%Y-%m-%dT%H:%M:%S.000Z"
some unix dates can't do millis, so just set to 000
=== shell var for shadowLastChange (ldap) ===
echo $((`date --utc --date "$1" +%s`/86400))
=== Days since the UNIX epoch ===
Return the number of days since the UNIX epoch using ''perl'':
perl -e 'printf qq{%d\n},time/86400'
=== Convert ''/etc/shadow'' ''lastchg'' to date ===
Convert the ''lastchg'' field in ''/etc/shadow'' to a date using GNU ''date'':
date -d "1 January 1970 + ''lastchg'' days"
=== Set timezone ===
set it system wide:
sudo timedatectl set-timezone Europe/Brussels
for your shell
export TZ=timezone
=== Links ===
* [http://en.wikipedia.org/wiki/Unix_time Unix time]
* [http://www.sunmanagers.org/pipermail/summaries/2002-November/002707.html SUMMARY - Convert lastchg field in shadow file to date]


== fake_tomcat.sh ==
== fake_tomcat.sh ==
Line 94: Line 194:
</pre>
</pre>


=While loops for Fun and Profit=
==While loops for Fun and Profit==


this script runs until you stop it. It collects file handle usage on a server putting the results in a file in the form:
this script runs until you stop it. It collects file handle usage on a server putting the results in a file in the form:
Line 105: Line 205:
  | |      |
  | |      |
  | |      |
  | |      |
  |       |      maximum open file descriptors
  |       |      maximum open file descriptors
  |       total free allocated file descriptors
  |       total free allocated file descriptors
  total allocated file descriptors
  total allocated file descriptors
  (the number of file descriptors allocated since boot)
  (the number of file descriptors allocated since boot)
Line 119: Line 219:
  done
  done


You MUST MUST MUST put the sleep in there or "Bad Things Will Happen"(tm).
You MUST MUST MUST put the sleep in there or "Bad Things Will Happen"(tm), it will loop too fast and you device could crash.


or in one line:
or in one line:


  while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/servername.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/servername.file-nr.2008040300; sleep 5; done
  while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/servername.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/servername.file-nr.2008040300; sleep 5; done
=== while loop over multi column input ===
tl;dr:
k get cm -A | grep ggr | awk '{print $1 , $2}' | while read ns cm; do echo ns is $ns; echo cm is $cm ; done <&0
You are looking for something in a kubernetes configmap ( cm ).
You query all configmaps for all names spaces
k get cm -A
and grep out a term. You don't want all those columns, you want the first two: namespace and configmap name.
grep ggr | awk '{print $1 , $2}'
now comes the fun part, you do while loop with a read, and by sending it &0 ( the stdin file descriptor ) , you can read the pipe you were given.
while read ns cm;
do
echo ns is $ns;
echo cm is $cm ;
done <&0
=== until loops ===
until PGPASSWORD=$POSTGRES_PASSWORD psql -h "$host" -U "postgres" -c '\q'; do
  >&2 echo "Postgres is unavailable - sleeping"
  sleep 1
done


==Traps==
==Traps==
Line 141: Line 271:
</pre>
</pre>


== Sorting Hostnames ==
=== sort ===
==== Sorting Hostnames ====


  service<instance>.location<instance>.fart.gas.bum
  service<instance>.location<instance>.fart.gas.bum
Line 147: Line 278:
  sort -t . -k2.2,1.1n -k1n
  sort -t . -k2.2,1.1n -k1n


== Sorting Ip Addresses ==
==== Sorting Ip Addresses ====


By Last three octets:
By Last three octets:
Line 158: Line 289:
  date -d "1970-01-01 UTC $1 seconds"
  date -d "1970-01-01 UTC $1 seconds"


== disable bash bell ==
== bash ==
=== disable bell ===


  echo "set bell-style none" >> ~/.inputrc
  echo "set bell-style none" >> ~/.inputrc


=== timestamps in history ===
export HISTTIMEFORMAT='%F %T '
== Awk ==
show me lines that don't have that in field 2
awk ' $2 !~ "[A-Za-z]" {print $0}'
who me lines that have less than 2 field:
awk ' NF < 2 {print $0}'
or shorter:
awk 'NF<2'
gimme field 2 - end (squash the first field then strip the leading space.)
awk '{$1=""; sub(/^[[:space:]]*/,""); print}'
=="Crontab last Saturday of the month" problem==
*Client had a problem where they wanted a script run on a server at 11 pm on the last Saturday of the month
*the crontab that was originally devised was:
#0 11 1-6 * 6 /home/smsadmin/CPU_util/runall.sh *snip*</pre>
*this ended up running at 11am from the 1st of the month to the 6th of the month, *as well as* every Saturday, '''NOT''' on Saturday as long as it was only the 1st to the 6th (this is somewhat unintuitive).
*to work around this, we wrote a quick wrapper 1-liner script that returned true if the same day next week had a different month than this month:
0 11 * * 6 [ $(date +\%m) != $(date +\%m -d "next week") ] && <rest of the command to run if the test passed></pre>
== find ==
find recent large stuff on /
find / -xdev -mtime -1 -size +10M | xargs ls -lad


[[Category:Computers]]
[[Category:Computers]]
== Which process is on which cpu? ==
ps -eo psr,pid,tid,nlwp,tty,comm
or sorted by processor:
ps -eo psr,pid,tid,nlwp,tty,comm | sort -n
ps doesn't seem to want to sort on processor. These don't work:
ps --sort=psr -eo psr,pid,tid,nlwp,tty,comm
ps --sort psr -eo psr,pid,tid,nlwp,tty,comm
ps kpsr -eo psr,pid,tid,nlwp,tty,comm
how many processes on which cpu?
ps h -eo psr | sort | uniq -c | awk '{printf "%4s %4s\n", $2 ,$1}' | sort -n
== Sed ==
=== grep out one variable with sed ===
cat /var/log/zimbra.log | sed -n 's/.*client=//p' | sort |uniq -c|sort -rn | head -30
=== remove terminal colours ===
sed 's/\x1b\[[0-9;]*m//g'
== set / unset / empty ==
<pre>
  +----------------------+------------+-----------------------+-----------------------+
  |  if VARIABLE is:    |    set    |        empty        |        unset          |
  +----------------------+------------+-----------------------+-----------------------+
- |  ${VARIABLE-default} | $VARIABLE  |          ""          |      "default"      |
= |  ${VARIABLE=default} | $VARIABLE  |          ""          | $(VARIABLE="default") |
? |  ${VARIABLE?default} | $VARIABLE  |          ""          |      exit 127        |
+ |  ${VARIABLE+default} | "default"  |      "default"      |          ""          |
  +----------------------+------------+-----------------------+-----------------------+
:- | ${VARIABLE:-default} | $VARIABLE  |      "default"      |      "default"      |
:= | ${VARIABLE:=default} | $VARIABLE  | $(VARIABLE="default") | $(VARIABLE="default") |
:? | ${VARIABLE:?default} | $VARIABLE  |      exit 127        |      exit 127        |
:+ | ${VARIABLE:+default} | "default"  |          ""          |          ""          |
  +----------------------+------------+-----------------------+-----------------------+
</pre>
ref: https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash
== parallel processes , ghetto style ==
<pre>
{
  xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF'
sleep 1; echo 1
sleep 2; echo 2
sleep 3; echo 3
echo 4
EOF
} &
</pre>
== intercepting options with bash ==
Something runs a program, it's hard coded it to run it a certain way, you want to run it a different way, say with different otions.
swap the binary out, and put a small shell script in the way:
in this exmaple the original runs chrome drvier with --port=XX and --log-path=/dev/null.
You want to keep the port, but change the logging.
mv /usr/bin/chromedriver /usr/bin/chromedriver.binary
heredoc user:user 755 /usr/bin/chromedriver <<<
<pre>
#!/bin/bash
re="--port=([0-9]*?) "
if [[ $@ =~ $re ]]; then
port=${BASH_REMATCH[1]}
/usr/bin/chromedriver.binary --port=${port} --log-path=/tmp/chromedriver.log --log-level=INFO --append-log
else
echo "no port specified"
exit 1
fi
</pre>

Latest revision as of 19:00, 20 November 2022


Shell invocation

/bin/sh

-u - treat the use of unset variables as errors.
-x - show me execution.

PS for swap

ps -eo pcpu,pid,pmem,rss,vsz,comm --sort=-rss

PS1

export PS1='[\u@\h \t \w]\$ '


from: https://github.com/stntngo/nil/blob/master/bash_profile

parse_git_branch() {
       git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/'
}
export PS1="\u \[\033[32m\]\${PS1X}\[\033[33m\]\$(parse_git_branch)\[\033[00m\]⚡"

Command line Tools

Diff

alias diffss='diff --width=180 --suppress-common-lines --side-by-side'


pwgen

pwgen -B -c -n -y
  • unabiguous
  • 1 capital
  • 1 number
  • 1 special char

Printing /proc/<pid>/environ

cat /proc/<pid>/environ | xargs --null --max-args=1 echo
cat /proc/(pidof process)/environ | tr '\0' '^M'
cat /proc/(pidof process)/environ | tr '\0' '<control-v><enter>'

HTTP response codes - filter

tail -f /var/log/httpd/access | awk '$9 !=200'

the print is implied, $9 happens to be where the http response code is in my log: 200 means OK, so it's show me the NOT OK stuff.

Grep failed wiki createaccount from apache log for iptables deny rule

ips=$(tail -200 access.log | grep title=Special:CreateAccount | awk '{print $1}' |sort | uniq | grep -v 142.126.179.34 | tr "\n" "," | sed 's/,$//')
/sbin/iptables -I INPUT 1 -s ${ips} -j DROP

or

/sbin/iptables -R INPUT 1 -s ${ips} -j DROP

Disk usage report

du -x --max-depth=1 / | sort  -rn

Date

shell var of NOW

NOW=`date +%a.%d.%b.%Y`

result:

Mon.04.Jul.2016

for DNS serial numbers:

NOW=`date "+%Y%m%d%H%M%S"`

result:

20160704174515

iso styleee!

$ date --iso-8601=seconds
2017-06-06T11:14:05-0400

Javascript

date "+%Y-%m-%dT%H:%M:%S.000Z"

some unix dates can't do millis, so just set to 000

shell var for shadowLastChange (ldap)

echo $((`date --utc --date "$1" +%s`/86400))


Days since the UNIX epoch

Return the number of days since the UNIX epoch using perl:

perl -e 'printf qq{%d\n},time/86400'

Convert /etc/shadow lastchg to date

Convert the lastchg field in /etc/shadow to a date using GNU date:

date -d "1 January 1970 + lastchg days"

Set timezone

set it system wide:

sudo timedatectl set-timezone Europe/Brussels

for your shell

export TZ=timezone


Links

fake_tomcat.sh

ARGV="$@"
if [ "x$ARGV" = "x" ] ; then
        echo usage: all start, stop, reload, abort, flush, or check
        exit
fi

case $# in
0)      echo 'Usage: ./snapshot <CPE name> (ie, ./snapshot YCDECUBC)' 1>&2; exit 2
esac

trap 'echo "";exit 3' 2 15
trap 'echo fake_tomcat.sh caught 1 HUP \-\> ok bye\! ; exit 3' 1
trap 'echo fake_tomcat.sh caught 3 QUIT \-\> ok bye\! ; exit 3' 3
trap 'echo fake_tomcat.sh caught 9 KILL \-\> ok bye\! ; exit 3' 9
trap 'echo fake_tomcat.sh caught 15 TERM \-\> ok bye\! ; exit 3' 15

TMPFILE=`mktemp /tmp/$0.XXXXXX` || exit 1

To move/duplicate filesystems I have a favorite way to do it locally:
# cd $filesystem_to_duplicate
# find . -print | cpio -pvdm /mnt

...where /mnt is the new filesystem/slice.
To duplicate/move across the network do it like this:
# cd $filesystem_to_duplicate
# tar cf - . | ssh otherhost "cd /$new_filesystem ; tar xf -"

function waitfor {
        if [ $# -lt 1 ] ; then
                echo "nothing to wait for"
        else
                echo "ok I'll wait"
                echo Still running = 1
                STILL_RUNNING=1
                while [ $STILL_RUNNING -gt 0 ]
                        do
                        STILL_RUNNING=`ps -auwwwx | grep $1 | grep -v grep | wc -l`
                        echo STILL_RUNNING = $STILL_RUNNING
                        sleep 1
                        echo waiting...
                        done
        fi
        echo $1
}
| tr '\n' ','


TimerOn()
{
  sleep $TIMELIMIT && kill -s 14 $$ &
  # Waits 3 seconds, then sends sigalarm to script.
}

Int14Vector()
{
  answer="TIMEOUT"
  PrintAnswer
  exit 14
}

trap Int14Vector 14

While loops for Fun and Profit

this script runs until you stop it. It collects file handle usage on a server putting the results in a file in the form:

<timestamp> <total allocated> <free> <maxpossible>

the last three field are from the /proc fs:

3391    969     52427
|	 |       |
|	 |       |
|        |       maximum open file descriptors
|       total free allocated file descriptors
total allocated file descriptors
(the number of file descriptors allocated since boot)

scripts:

while true;
do
 echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/s4.t55.file-nr.2008040300;
 cat /proc/sys/fs/file-nr >> /home/dathornton/s4.t55.file-nr.2008040300;
 sleep 5;
done

You MUST MUST MUST put the sleep in there or "Bad Things Will Happen"(tm), it will loop too fast and you device could crash.

or in one line:

while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/servername.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/servername.file-nr.2008040300; sleep 5; done

while loop over multi column input

tl;dr:

k get cm -A | grep ggr | awk '{print $1 , $2}' | while read ns cm; do echo ns is $ns; echo cm is $cm ; done <&0

You are looking for something in a kubernetes configmap ( cm ). You query all configmaps for all names spaces

k get cm -A

and grep out a term. You don't want all those columns, you want the first two: namespace and configmap name.

grep ggr | awk '{print $1 , $2}'

now comes the fun part, you do while loop with a read, and by sending it &0 ( the stdin file descriptor ) , you can read the pipe you were given.

while read ns cm;
do
echo ns is $ns;
echo cm is $cm ;
done <&0

until loops

until PGPASSWORD=$POSTGRES_PASSWORD psql -h "$host" -U "postgres" -c '\q'; do
  >&2 echo "Postgres is unavailable - sleeping"
  sleep 1
done

Traps

#!/bin/bash
# traptest.sh

trap "echo Booh!" SIGINT SIGTERM
trap "echo Kill" SIGKILL
echo "pid is $$"

while: # This is the same as "while true".
do
 sleep 5 # This script is not really doing anything.
done

sort

Sorting Hostnames

service<instance>.location<instance>.fart.gas.bum
sort -t . -k2.2,1.1n -k1n

Sorting Ip Addresses

By Last three octets:

sort -t . -k 2,2n -k 3,3n -k 4,4n serverlist| more

epoch

#!/bin/sh
date -d "1970-01-01 UTC $1 seconds"

bash

disable bell

echo "set bell-style none" >> ~/.inputrc

timestamps in history

export HISTTIMEFORMAT='%F %T '

Awk

show me lines that don't have that in field 2

awk ' $2 !~ "[A-Za-z]" {print $0}'

who me lines that have less than 2 field:

awk ' NF < 2 {print $0}'

or shorter:

awk 'NF<2'

gimme field 2 - end (squash the first field then strip the leading space.)

awk '{$1=""; sub(/^space:*/,""); print}' 


"Crontab last Saturday of the month" problem

  • Client had a problem where they wanted a script run on a server at 11 pm on the last Saturday of the month
  • the crontab that was originally devised was:

#0 11 1-6 * 6 /home/smsadmin/CPU_util/runall.sh *snip*

  • this ended up running at 11am from the 1st of the month to the 6th of the month, *as well as* every Saturday, NOT on Saturday as long as it was only the 1st to the 6th (this is somewhat unintuitive).
  • to work around this, we wrote a quick wrapper 1-liner script that returned true if the same day next week had a different month than this month:

0 11 * * 6 [ $(date +\%m) != $(date +\%m -d "next week") ] && <rest of the command to run if the test passed>

find

find recent large stuff on /

find / -xdev -mtime -1 -size +10M | xargs ls -lad

Which process is on which cpu?

ps -eo psr,pid,tid,nlwp,tty,comm

or sorted by processor:

ps -eo psr,pid,tid,nlwp,tty,comm | sort -n

ps doesn't seem to want to sort on processor. These don't work:

ps --sort=psr -eo psr,pid,tid,nlwp,tty,comm
ps --sort psr -eo psr,pid,tid,nlwp,tty,comm
ps kpsr -eo psr,pid,tid,nlwp,tty,comm

how many processes on which cpu?

ps h -eo psr | sort | uniq -c | awk '{printf "%4s %4s\n", $2 ,$1}' | sort -n

Sed

grep out one variable with sed

cat /var/log/zimbra.log | sed -n 's/.*client=//p' | sort |uniq -c|sort -rn | head -30

remove terminal colours

sed 's/\x1b\[[0-9;]*m//g'


set / unset / empty

   +----------------------+------------+-----------------------+-----------------------+
   |   if VARIABLE is:    |    set     |         empty         |        unset          |
   +----------------------+------------+-----------------------+-----------------------+
 - |  ${VARIABLE-default} | $VARIABLE  |          ""           |       "default"       |
 = |  ${VARIABLE=default} | $VARIABLE  |          ""           | $(VARIABLE="default") |
 ? |  ${VARIABLE?default} | $VARIABLE  |          ""           |       exit 127        |
 + |  ${VARIABLE+default} | "default"  |       "default"       |          ""           |
   +----------------------+------------+-----------------------+-----------------------+
:- | ${VARIABLE:-default} | $VARIABLE  |       "default"       |       "default"       |
:= | ${VARIABLE:=default} | $VARIABLE  | $(VARIABLE="default") | $(VARIABLE="default") |
:? | ${VARIABLE:?default} | $VARIABLE  |       exit 127        |       exit 127        |
:+ | ${VARIABLE:+default} | "default"  |          ""           |          ""           |
   +----------------------+------------+-----------------------+-----------------------+

ref: https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash

parallel processes , ghetto style

{
  xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF'
sleep 1; echo 1
sleep 2; echo 2
sleep 3; echo 3
echo 4
EOF
} &

intercepting options with bash

Something runs a program, it's hard coded it to run it a certain way, you want to run it a different way, say with different otions.

swap the binary out, and put a small shell script in the way:

in this exmaple the original runs chrome drvier with --port=XX and --log-path=/dev/null.

You want to keep the port, but change the logging.

mv /usr/bin/chromedriver /usr/bin/chromedriver.binary

heredoc user:user 755 /usr/bin/chromedriver <<<

#!/bin/bash

re="--port=([0-9]*?) "
if [[ $@ =~ $re ]]; then
 port=${BASH_REMATCH[1]}
 /usr/bin/chromedriver.binary --port=${port} --log-path=/tmp/chromedriver.log --log-level=INFO --append-log
else
 echo "no port specified"
 exit 1
fi