Stupid Shell Tricks: Difference between revisions
No edit summary |
|||
(54 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
[[Category:Script]] | |||
== Shell invocation == | |||
/bin/sh | |||
-u - treat the use of unset variables as errors. | |||
-x - show me execution. | |||
== PS for swap == | |||
ps -eo pcpu,pid,pmem,rss,vsz,comm --sort=-rss | |||
== PS1 == | |||
export PS1='[\u@\h \t \w]\$ ' | |||
from: https://github.com/stntngo/nil/blob/master/bash_profile | |||
parse_git_branch() { | |||
git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/' | |||
} | |||
export PS1="\u \[\033[32m\]\${PS1X}\[\033[33m\]\$(parse_git_branch)\[\033[00m\]⚡" | |||
== Command line Tools == | |||
=== Diff === | |||
alias diffss='diff --width=180 --suppress-common-lines --side-by-side' | |||
=== pwgen === | |||
pwgen -B -c -n -y | |||
* unabiguous | |||
* 1 capital | |||
* 1 number | |||
* 1 special char | |||
== Printing /proc/<pid>/environ == | |||
cat /proc/<pid>/environ | xargs --null --max-args=1 echo | |||
cat /proc/(pidof process)/environ | tr '\0' '^M' | |||
cat /proc/(pidof process)/environ | tr '\0' '<control-v><enter>' | |||
== HTTP response codes - filter == | |||
tail -f /var/log/httpd/access | awk '$9 !=200' | |||
the print is implied, $9 happens to be where the http response code is in my log: 200 means OK, so it's show me the NOT OK stuff. | |||
== Grep failed wiki createaccount from apache log for iptables deny rule == | |||
ips=$(tail -200 access.log | grep title=Special:CreateAccount | awk '{print $1}' |sort | uniq | grep -v 142.126.179.34 | tr "\n" "," | sed 's/,$//') | |||
/sbin/iptables -I INPUT 1 -s ${ips} -j DROP | |||
or | |||
/sbin/iptables -R INPUT 1 -s ${ips} -j DROP | |||
== Disk usage report == | |||
du -x --max-depth=1 / | sort -rn | du -x --max-depth=1 / | sort -rn | ||
== Date == | |||
=== shell var of NOW === | |||
NOW=`date +%a.%d.%b.%Y` | NOW=`date +%a.%d.%b.%Y` | ||
result: | |||
Mon.04.Jul.2016 | |||
for DNS serial numbers: | |||
NOW=`date "+%Y%m%d%H%M%S"` | |||
result: | |||
20160704174515 | |||
iso styleee! | |||
$ date --iso-8601=seconds | |||
2017-06-06T11:14:05-0400 | |||
Javascript | |||
date "+%Y-%m-%dT%H:%M:%S.000Z" | |||
some unix dates can't do millis, so just set to 000 | |||
=== shell var for shadowLastChange (ldap) === | |||
echo $((`date --utc --date "$1" +%s`/86400)) | |||
=== Days since the UNIX epoch === | |||
Return the number of days since the UNIX epoch using ''perl'': | |||
perl -e 'printf qq{%d\n},time/86400' | |||
=== Convert ''/etc/shadow'' ''lastchg'' to date === | |||
Convert the ''lastchg'' field in ''/etc/shadow'' to a date using GNU ''date'': | |||
date -d "1 January 1970 + ''lastchg'' days" | |||
=== Set timezone === | |||
set it system wide: | |||
sudo timedatectl set-timezone Europe/Brussels | |||
for your shell | |||
export TZ=timezone | |||
=== Links === | |||
* [http://en.wikipedia.org/wiki/Unix_time Unix time] | |||
* [http://www.sunmanagers.org/pipermail/summaries/2002-November/002707.html SUMMARY - Convert lastchg field in shadow file to date] | |||
== fake_tomcat.sh == | |||
<pre> | <pre> | ||
Line 67: | Line 194: | ||
</pre> | </pre> | ||
=While loops for Fun and Profit= | ==While loops for Fun and Profit== | ||
this script runs until you stop it. It collects file handle usage on a server putting the results in a file in the form: | this script runs until you stop it. It collects file handle usage on a server putting the results in a file in the form: | ||
Line 78: | Line 205: | ||
| | | | | | | | ||
| | | | | | | | ||
| | | | maximum open file descriptors | ||
| | | total free allocated file descriptors | ||
total allocated file descriptors | total allocated file descriptors | ||
(the number of file descriptors allocated since boot) | (the number of file descriptors allocated since boot) | ||
Line 92: | Line 219: | ||
done | done | ||
You MUST MUST MUST put the sleep in there or "Bad Things Will Happen"(tm). | You MUST MUST MUST put the sleep in there or "Bad Things Will Happen"(tm), it will loop too fast and you device could crash. | ||
or in one line: | or in one line: | ||
Line 98: | Line 225: | ||
while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/servername.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/servername.file-nr.2008040300; sleep 5; done | while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/servername.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/servername.file-nr.2008040300; sleep 5; done | ||
= | === while loop over multi column input === | ||
tl;dr: | |||
k get cm -A | grep ggr | awk '{print $1 , $2}' | while read ns cm; do echo ns is $ns; echo cm is $cm ; done <&0 | |||
You are looking for something in a kubernetes configmap ( cm ). | |||
You query all configmaps for all names spaces | |||
k get cm -A | |||
and grep out a term. You don't want all those columns, you want the first two: namespace and configmap name. | |||
grep ggr | awk '{print $1 , $2}' | |||
now comes the fun part, you do while loop with a read, and by sending it &0 ( the stdin file descriptor ) , you can read the pipe you were given. | |||
while read ns cm; | |||
do | |||
echo ns is $ns; | |||
echo cm is $cm ; | |||
done <&0 | |||
=== until loops === | |||
until PGPASSWORD=$POSTGRES_PASSWORD psql -h "$host" -U "postgres" -c '\q'; do | |||
>&2 echo "Postgres is unavailable - sleeping" | |||
sleep 1 | |||
done | |||
==Traps== | |||
<pre> | |||
#!/bin/bash | #!/bin/bash | ||
# traptest.sh | # traptest.sh | ||
Line 107: | Line 265: | ||
echo "pid is $$" | echo "pid is $$" | ||
while : | while: # This is the same as "while true". | ||
do | do | ||
sleep 5 # This script is not really doing anything. | |||
done | done | ||
</pre> | |||
=== sort === | |||
==== Sorting Hostnames ==== | |||
service<instance>.location<instance>.fart.gas.bum | service<instance>.location<instance>.fart.gas.bum | ||
Line 118: | Line 278: | ||
sort -t . -k2.2,1.1n -k1n | sort -t . -k2.2,1.1n -k1n | ||
==== Sorting Ip Addresses ==== | |||
By Last three octets: | |||
sort -t . -k 2,2n -k 3,3n -k 4,4n serverlist| more | |||
== epoch == | |||
#!/bin/sh | |||
date -d "1970-01-01 UTC $1 seconds" | |||
== bash == | |||
=== disable bell === | |||
echo "set bell-style none" >> ~/.inputrc | |||
=== timestamps in history === | |||
export HISTTIMEFORMAT='%F %T ' | |||
== Awk == | |||
show me lines that don't have that in field 2 | |||
awk ' $2 !~ "[A-Za-z]" {print $0}' | |||
who me lines that have less than 2 field: | |||
awk ' NF < 2 {print $0}' | |||
or shorter: | |||
awk 'NF<2' | |||
gimme field 2 - end (squash the first field then strip the leading space.) | |||
awk '{$1=""; sub(/^[[:space:]]*/,""); print}' | |||
=="Crontab last Saturday of the month" problem== | |||
*Client had a problem where they wanted a script run on a server at 11 pm on the last Saturday of the month | |||
*the crontab that was originally devised was: | |||
#0 11 1-6 * 6 /home/smsadmin/CPU_util/runall.sh *snip*</pre> | |||
*this ended up running at 11am from the 1st of the month to the 6th of the month, *as well as* every Saturday, '''NOT''' on Saturday as long as it was only the 1st to the 6th (this is somewhat unintuitive). | |||
*to work around this, we wrote a quick wrapper 1-liner script that returned true if the same day next week had a different month than this month: | |||
0 11 * * 6 [ $(date +\%m) != $(date +\%m -d "next week") ] && <rest of the command to run if the test passed></pre> | |||
== find == | |||
find recent large stuff on / | |||
find / -xdev -mtime -1 -size +10M | xargs ls -lad | |||
[[Category:Computers]] | [[Category:Computers]] | ||
== Which process is on which cpu? == | |||
ps -eo psr,pid,tid,nlwp,tty,comm | |||
or sorted by processor: | |||
ps -eo psr,pid,tid,nlwp,tty,comm | sort -n | |||
ps doesn't seem to want to sort on processor. These don't work: | |||
ps --sort=psr -eo psr,pid,tid,nlwp,tty,comm | |||
ps --sort psr -eo psr,pid,tid,nlwp,tty,comm | |||
ps kpsr -eo psr,pid,tid,nlwp,tty,comm | |||
how many processes on which cpu? | |||
ps h -eo psr | sort | uniq -c | awk '{printf "%4s %4s\n", $2 ,$1}' | sort -n | |||
== Sed == | |||
=== grep out one variable with sed === | |||
cat /var/log/zimbra.log | sed -n 's/.*client=//p' | sort |uniq -c|sort -rn | head -30 | |||
=== remove terminal colours === | |||
sed 's/\x1b\[[0-9;]*m//g' | |||
== set / unset / empty == | |||
<pre> | |||
+----------------------+------------+-----------------------+-----------------------+ | |||
| if VARIABLE is: | set | empty | unset | | |||
+----------------------+------------+-----------------------+-----------------------+ | |||
- | ${VARIABLE-default} | $VARIABLE | "" | "default" | | |||
= | ${VARIABLE=default} | $VARIABLE | "" | $(VARIABLE="default") | | |||
? | ${VARIABLE?default} | $VARIABLE | "" | exit 127 | | |||
+ | ${VARIABLE+default} | "default" | "default" | "" | | |||
+----------------------+------------+-----------------------+-----------------------+ | |||
:- | ${VARIABLE:-default} | $VARIABLE | "default" | "default" | | |||
:= | ${VARIABLE:=default} | $VARIABLE | $(VARIABLE="default") | $(VARIABLE="default") | | |||
:? | ${VARIABLE:?default} | $VARIABLE | exit 127 | exit 127 | | |||
:+ | ${VARIABLE:+default} | "default" | "" | "" | | |||
+----------------------+------------+-----------------------+-----------------------+ | |||
</pre> | |||
ref: https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash | |||
== parallel processes , ghetto style == | |||
<pre> | |||
{ | |||
xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF' | |||
sleep 1; echo 1 | |||
sleep 2; echo 2 | |||
sleep 3; echo 3 | |||
echo 4 | |||
EOF | |||
} & | |||
</pre> | |||
== intercepting options with bash == | |||
Something runs a program, it's hard coded it to run it a certain way, you want to run it a different way, say with different otions. | |||
swap the binary out, and put a small shell script in the way: | |||
in this exmaple the original runs chrome drvier with --port=XX and --log-path=/dev/null. | |||
You want to keep the port, but change the logging. | |||
mv /usr/bin/chromedriver /usr/bin/chromedriver.binary | |||
heredoc user:user 755 /usr/bin/chromedriver <<< | |||
<pre> | |||
#!/bin/bash | |||
re="--port=([0-9]*?) " | |||
if [[ $@ =~ $re ]]; then | |||
port=${BASH_REMATCH[1]} | |||
/usr/bin/chromedriver.binary --port=${port} --log-path=/tmp/chromedriver.log --log-level=INFO --append-log | |||
else | |||
echo "no port specified" | |||
exit 1 | |||
fi | |||
</pre> |
Latest revision as of 19:00, 20 November 2022
Shell invocation
/bin/sh
-u - treat the use of unset variables as errors. -x - show me execution.
PS for swap
ps -eo pcpu,pid,pmem,rss,vsz,comm --sort=-rss
PS1
export PS1='[\u@\h \t \w]\$ '
from: https://github.com/stntngo/nil/blob/master/bash_profile
parse_git_branch() { git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/' } export PS1="\u \[\033[32m\]\${PS1X}\[\033[33m\]\$(parse_git_branch)\[\033[00m\]⚡"
Command line Tools
Diff
alias diffss='diff --width=180 --suppress-common-lines --side-by-side'
pwgen
pwgen -B -c -n -y
- unabiguous
- 1 capital
- 1 number
- 1 special char
Printing /proc/<pid>/environ
cat /proc/<pid>/environ | xargs --null --max-args=1 echo
cat /proc/(pidof process)/environ | tr '\0' '^M'
cat /proc/(pidof process)/environ | tr '\0' '<control-v><enter>'
HTTP response codes - filter
tail -f /var/log/httpd/access | awk '$9 !=200'
the print is implied, $9 happens to be where the http response code is in my log: 200 means OK, so it's show me the NOT OK stuff.
Grep failed wiki createaccount from apache log for iptables deny rule
ips=$(tail -200 access.log | grep title=Special:CreateAccount | awk '{print $1}' |sort | uniq | grep -v 142.126.179.34 | tr "\n" "," | sed 's/,$//')
/sbin/iptables -I INPUT 1 -s ${ips} -j DROP
or
/sbin/iptables -R INPUT 1 -s ${ips} -j DROP
Disk usage report
du -x --max-depth=1 / | sort -rn
Date
shell var of NOW
NOW=`date +%a.%d.%b.%Y`
result:
Mon.04.Jul.2016
for DNS serial numbers:
NOW=`date "+%Y%m%d%H%M%S"`
result:
20160704174515
iso styleee!
$ date --iso-8601=seconds 2017-06-06T11:14:05-0400
Javascript
date "+%Y-%m-%dT%H:%M:%S.000Z"
some unix dates can't do millis, so just set to 000
shell var for shadowLastChange (ldap)
echo $((`date --utc --date "$1" +%s`/86400))
Days since the UNIX epoch
Return the number of days since the UNIX epoch using perl:
perl -e 'printf qq{%d\n},time/86400'
Convert /etc/shadow lastchg to date
Convert the lastchg field in /etc/shadow to a date using GNU date:
date -d "1 January 1970 + lastchg days"
Set timezone
set it system wide:
sudo timedatectl set-timezone Europe/Brussels
for your shell
export TZ=timezone
Links
fake_tomcat.sh
ARGV="$@" if [ "x$ARGV" = "x" ] ; then echo usage: all start, stop, reload, abort, flush, or check exit fi case $# in 0) echo 'Usage: ./snapshot <CPE name> (ie, ./snapshot YCDECUBC)' 1>&2; exit 2 esac trap 'echo "";exit 3' 2 15 trap 'echo fake_tomcat.sh caught 1 HUP \-\> ok bye\! ; exit 3' 1 trap 'echo fake_tomcat.sh caught 3 QUIT \-\> ok bye\! ; exit 3' 3 trap 'echo fake_tomcat.sh caught 9 KILL \-\> ok bye\! ; exit 3' 9 trap 'echo fake_tomcat.sh caught 15 TERM \-\> ok bye\! ; exit 3' 15 TMPFILE=`mktemp /tmp/$0.XXXXXX` || exit 1 To move/duplicate filesystems I have a favorite way to do it locally: # cd $filesystem_to_duplicate # find . -print | cpio -pvdm /mnt ...where /mnt is the new filesystem/slice. To duplicate/move across the network do it like this: # cd $filesystem_to_duplicate # tar cf - . | ssh otherhost "cd /$new_filesystem ; tar xf -" function waitfor { if [ $# -lt 1 ] ; then echo "nothing to wait for" else echo "ok I'll wait" echo Still running = 1 STILL_RUNNING=1 while [ $STILL_RUNNING -gt 0 ] do STILL_RUNNING=`ps -auwwwx | grep $1 | grep -v grep | wc -l` echo STILL_RUNNING = $STILL_RUNNING sleep 1 echo waiting... done fi echo $1 } | tr '\n' ',' TimerOn() { sleep $TIMELIMIT && kill -s 14 $$ & # Waits 3 seconds, then sends sigalarm to script. } Int14Vector() { answer="TIMEOUT" PrintAnswer exit 14 } trap Int14Vector 14
While loops for Fun and Profit
this script runs until you stop it. It collects file handle usage on a server putting the results in a file in the form:
<timestamp> <total allocated> <free> <maxpossible>
the last three field are from the /proc fs:
3391 969 52427 | | | | | | | | maximum open file descriptors | total free allocated file descriptors total allocated file descriptors (the number of file descriptors allocated since boot)
scripts:
while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/s4.t55.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/s4.t55.file-nr.2008040300; sleep 5; done
You MUST MUST MUST put the sleep in there or "Bad Things Will Happen"(tm), it will loop too fast and you device could crash.
or in one line:
while true; do echo `date +%s` | awk 'BEGIN{ORS=""}{print $0 " "}' >> /home/dathornton/servername.file-nr.2008040300; cat /proc/sys/fs/file-nr >> /home/dathornton/servername.file-nr.2008040300; sleep 5; done
while loop over multi column input
tl;dr:
k get cm -A | grep ggr | awk '{print $1 , $2}' | while read ns cm; do echo ns is $ns; echo cm is $cm ; done <&0
You are looking for something in a kubernetes configmap ( cm ). You query all configmaps for all names spaces
k get cm -A
and grep out a term. You don't want all those columns, you want the first two: namespace and configmap name.
grep ggr | awk '{print $1 , $2}'
now comes the fun part, you do while loop with a read, and by sending it &0 ( the stdin file descriptor ) , you can read the pipe you were given.
while read ns cm; do echo ns is $ns; echo cm is $cm ; done <&0
until loops
until PGPASSWORD=$POSTGRES_PASSWORD psql -h "$host" -U "postgres" -c '\q'; do >&2 echo "Postgres is unavailable - sleeping" sleep 1 done
Traps
#!/bin/bash # traptest.sh trap "echo Booh!" SIGINT SIGTERM trap "echo Kill" SIGKILL echo "pid is $$" while: # This is the same as "while true". do sleep 5 # This script is not really doing anything. done
sort
Sorting Hostnames
service<instance>.location<instance>.fart.gas.bum
sort -t . -k2.2,1.1n -k1n
Sorting Ip Addresses
By Last three octets:
sort -t . -k 2,2n -k 3,3n -k 4,4n serverlist| more
epoch
#!/bin/sh date -d "1970-01-01 UTC $1 seconds"
bash
disable bell
echo "set bell-style none" >> ~/.inputrc
timestamps in history
export HISTTIMEFORMAT='%F %T '
Awk
show me lines that don't have that in field 2
awk ' $2 !~ "[A-Za-z]" {print $0}'
who me lines that have less than 2 field:
awk ' NF < 2 {print $0}'
or shorter:
awk 'NF<2'
gimme field 2 - end (squash the first field then strip the leading space.)
awk '{$1=""; sub(/^space:*/,""); print}'
"Crontab last Saturday of the month" problem
- Client had a problem where they wanted a script run on a server at 11 pm on the last Saturday of the month
- the crontab that was originally devised was:
#0 11 1-6 * 6 /home/smsadmin/CPU_util/runall.sh *snip*
- this ended up running at 11am from the 1st of the month to the 6th of the month, *as well as* every Saturday, NOT on Saturday as long as it was only the 1st to the 6th (this is somewhat unintuitive).
- to work around this, we wrote a quick wrapper 1-liner script that returned true if the same day next week had a different month than this month:
0 11 * * 6 [ $(date +\%m) != $(date +\%m -d "next week") ] && <rest of the command to run if the test passed>
find
find recent large stuff on /
find / -xdev -mtime -1 -size +10M | xargs ls -lad
Which process is on which cpu?
ps -eo psr,pid,tid,nlwp,tty,comm
or sorted by processor:
ps -eo psr,pid,tid,nlwp,tty,comm | sort -n
ps doesn't seem to want to sort on processor. These don't work:
ps --sort=psr -eo psr,pid,tid,nlwp,tty,comm ps --sort psr -eo psr,pid,tid,nlwp,tty,comm ps kpsr -eo psr,pid,tid,nlwp,tty,comm
how many processes on which cpu?
ps h -eo psr | sort | uniq -c | awk '{printf "%4s %4s\n", $2 ,$1}' | sort -n
Sed
grep out one variable with sed
cat /var/log/zimbra.log | sed -n 's/.*client=//p' | sort |uniq -c|sort -rn | head -30
remove terminal colours
sed 's/\x1b\[[0-9;]*m//g'
set / unset / empty
+----------------------+------------+-----------------------+-----------------------+ | if VARIABLE is: | set | empty | unset | +----------------------+------------+-----------------------+-----------------------+ - | ${VARIABLE-default} | $VARIABLE | "" | "default" | = | ${VARIABLE=default} | $VARIABLE | "" | $(VARIABLE="default") | ? | ${VARIABLE?default} | $VARIABLE | "" | exit 127 | + | ${VARIABLE+default} | "default" | "default" | "" | +----------------------+------------+-----------------------+-----------------------+ :- | ${VARIABLE:-default} | $VARIABLE | "default" | "default" | := | ${VARIABLE:=default} | $VARIABLE | $(VARIABLE="default") | $(VARIABLE="default") | :? | ${VARIABLE:?default} | $VARIABLE | exit 127 | exit 127 | :+ | ${VARIABLE:+default} | "default" | "" | "" | +----------------------+------------+-----------------------+-----------------------+
ref: https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash
parallel processes , ghetto style
{ xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF' sleep 1; echo 1 sleep 2; echo 2 sleep 3; echo 3 echo 4 EOF } &
intercepting options with bash
Something runs a program, it's hard coded it to run it a certain way, you want to run it a different way, say with different otions.
swap the binary out, and put a small shell script in the way:
in this exmaple the original runs chrome drvier with --port=XX and --log-path=/dev/null.
You want to keep the port, but change the logging.
mv /usr/bin/chromedriver /usr/bin/chromedriver.binary
heredoc user:user 755 /usr/bin/chromedriver <<<
#!/bin/bash re="--port=([0-9]*?) " if [[ $@ =~ $re ]]; then port=${BASH_REMATCH[1]} /usr/bin/chromedriver.binary --port=${port} --log-path=/tmp/chromedriver.log --log-level=INFO --append-log else echo "no port specified" exit 1 fi