BASH magic

  • strict warning: Non-static method view::load() should not be called statically in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/views.module on line 906.
  • strict warning: Declaration of views_handler_argument::init() should be compatible with views_handler::init(&$view, $options) in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/handlers/views_handler_argument.inc on line 744.
  • strict warning: Declaration of views_handler_filter::options_validate() should be compatible with views_handler::options_validate($form, &$form_state) in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/handlers/views_handler_filter.inc on line 607.
  • strict warning: Declaration of views_handler_filter::options_submit() should be compatible with views_handler::options_submit($form, &$form_state) in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/handlers/views_handler_filter.inc on line 607.
  • strict warning: Declaration of views_handler_filter_boolean_operator::value_validate() should be compatible with views_handler_filter::value_validate($form, &$form_state) in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/handlers/views_handler_filter_boolean_operator.inc on line 159.
Leeland's picture

== Table of Contents ==

__TOC__

== My .bashrc.user file right now ==
:

export SVN_EDITOR=/usr/bin/vim

# make bash autocomplete with up arrow
bind '"\e[A":history-search-backward'
bind '"\e[B":history-search-forward'

# a simple function that will check for headers of HTTP server.
#
# invoke: headers  []
#                 port number defaults to 80 if not provided.
headers() {
    server=$1; port=${2:-80}
    exec 5<> /dev/tcp/$server/$port
    echo -e "HEAD / HTTP/1.0\nHost: ${server}\n\n" >&5
    cat <5
    exec 5<&-
}

# a quick way to check if a port is open or not. You can always use netcat or telnet,
# but I find this easier
#
# invoke: testPort   []
#                  protocol can be either tcp or udp and defaults to tcp.
testPort() {
    server=$1; port=$2; proto=${3:-tcp}
    exec 5<>/dev/$proto/$server/$port
    (( $? == 0 )) && exec 5<&-
}

# Look if the parent is konsole
dcop konsole-$PPID &> /dev/null
if [ $? -eq 0 ]; then
    #OK we are on konsole
    PROMPT_COMMAND=touch_konsole
    export INKONSOLE=$PPID
    alias vi='kvi'
    alias ssh='kssh'
else
    export INKONSOLE=0
fi

function touch_konsole {
    # Lil'optimization: only change if PWD change
    if [ "x$PWD_LAST" != "x$PWD" ]; then
        #PRETTY_PWD=`echo $PWD | sed "s\\^$HOME\\~\\ "`
        PRETTY_PWD=`echo $PWD | sed "s,^$HOME,~,; s,\(/[^/]*\)/.\+\(/.\+/.\+\)$,\1/..\2,"`
        # You can have a the user name too
        PRETTY_PWD="${USER}:${PRETTY_PWD}"
        dcop $DCOP_OPTIONS $KONSOLE_DCOP_SESSION renameSession "$PRETTY_PWD"
        if [ $? -ne 0 ]; then # our great imagination forces us to make comprobations
            # If you know other situations for this, or you have a fix, please, mail it to thc{at}wanadoo{dot}es
            echo "DCOP call failed. Situations causing this are:"
            echo " -There are lost .DCOPserver_* files in your home, please remove them. This can happen when hostname change."
            echo " -If you called 'su(1)', there are various DCOP sessions for the user you come from."
            echo
            echo "DCOP calls to Konsole are disabled"
            unset PROMPT_COMMAND
        fi
        PWD_LAST=$PWD
    fi
}


# Look if we have Konsole enviroment
if [ "x$KONSOLE_DCOP_SESSION" != "x" ]; then
    #OK we are on konsole
    export PROMPT_COMMAND=touch_konsole
    export DCOP_OPTIONS=""
    if [ "x$USER" = "xroot" ]; then # when root we must use imagination
        # Let's search directly into the parent
        # Is there a '=' char in the user name? come here an fix this.
        export PPID_USER=`strings /proc/$PPID/environ | grep ^USER= | tr = '\n' | tail -n 1`
        if [ "x$PPID_USER" != "xroot" ]; then
            # Oh! a user "suing" in a konsole?... mi parent is not root's konsole
            export DCOP_OPTIONS="--user $PPID_USER --all-sessions"
            # Here you can do other crazy things like the X auth for root...
            # or salutate you
            # export XAUTHORITY="/home/${PPID_USER}/.Xauthority"
            echo "Hello ${PPID_USER}, you got root"
        fi
    fi
fi

== Scripting References ==

# [http://www.faqs.org/docs/abs/HTML/ Advanced Bash-Scripting Guide] A very excellent textbook / self-study manual / reference / source of knowledge on shell scripting techniques. (An alternate location in case the first goes offline [http://tldp.org/LDP/abs/html/index.html http://tldp.org/LDP/abs/html/index.html].) Some favorite sections are:
#* [http://www.faqs.org/docs/abs/HTML/dosbatch.html Converting DOS Batch Files to Shell Scripts]
#* [http://www.faqs.org/docs/abs/HTML/here-docs.html Here Documents]
# Jeremy Mates (a great guy and one heck of a shell hacker) has a great page on shell tricks at http://sial.org/howto/shell/

== Command Line Limits and Parallel Operations ==

=== Finding the command line limits ===

: All shells have a limit for the command line length. Each UNIX / Linux / BSD system has a limit on how many bytes can be used for the command line argument and environment variables. When you start a new process or type a command these limitations are applied and you will see an error message as follows on screen:

:: Argument list too long

: The following command will expose the current command line length limitations

::

$ getconf ARG_MAX

: However, when a command is executed the current environment is pre-pended to the actual command. Therefore, to insure you are not exceeding the real limit your command length you need to use the following command to compute the current actual input command line length limitation

::

$ echo $(( $(getconf ARG_MAX) - $(env | wc -c) ))

=== Overcoming the command line limits ===

: Knowing the command line limits is one step. The next step is overcoming them when needed. There are two ways:

:* Use either the find or xargs commands; or
:* Use the shell for / while loop

: Examples

:* find command example to get rid of "argument list too long" error

$ find /nas/data/accounting/ -type f -exec ls -l {} \;
$ find /nas/data/accounting/ -type f -exec /bin/rm -f {} \;

:* xargs command example to get rid of "argument list too long" error

$ echo /nas/data/accounting/* | xargs ls -l
$ echo /nas/data/accounting/* | xargs /bin/rm -f

:* while loop example to get rid of "argument list too long" error

ls -1 /nas/data/accounting/ | \
 while read file; do 
   mv /nas/data/accounting/$file /local/disk/ ;
 done

:* Alternatively, you can combine above methods:

find /nas/data/accounting/ -type f |
   while read file
   do
     mv /nas/data/accounting/$file /local/disk/
   done

== Reading Java-style Properties Files with Shell ==

=== Method 1 - Scan through the file to get an individual property ===

This works well if you just want to read a specific property in a Java-style properties file. Here's what it does:

:# Remove comments (lines that start with '#')
:# Grep for the property
:# Get the last value.
:# Strip off the property name and the '='

sed '/^\#/d' myprops.properties | grep 'someproperty' | tail -n 1 | sed 's/^.*=//'

It can also be handy to strip off the leading and trailing blanks with this sed expression:

s/^[[:space:]]*//;s/[[:space:]]*$//

Which makes the whole thing look like this:

sed '/^\#/d' myprops.properties | grep 'someproperty' | tail -n 1 | sed 's/^.*=//;s/^[[:space:]]*//;s/[[:space:]]*$//'

Shell scripts can use this technique to set environment variables, for example:

JAVA_HOME=`sed '/^\#/d' build.properties | grep 'jdk.home' | tail -n 1 | sed 's/^.*=//;s/^[[:space:]]*//;s/[[:space:]]*$//'`

=== Method 2 - Convert the properties file to a shell script, then run it ===

This method is good for when you want to read the whole properties file. However, it only works if the property names are valid shell variable names. For example, property names with a '.' in them will cause this to break. Here are the original pseudo code steps that inspired this method.

The steps are:

:# Create a temp file.
:# Stream the properties file while:
:## Removing the whitespace around the equal sign (=)
:## Converting it to Unix LF format (unless you are sure the file was produced on Linux).
:## Escaping any double quotes in the string
:## Double quoting all the values, to preserve spaces.
:# Source in the file. Shell should ignore Java properties file comments automatically.
:# Erase the temp file.

Here is the code:

TEMPFILE=$(mktemp)

cat somefile.properties | \
dos2unix | \
sed -re 's,[[:space:]]*=[[:space:]]*,=,;s/"/\\"/g; s/=(.*)$/="\1"/g' >$TEMPFILE

source $TEMPFILE
rm $TEMPFILE

=== Method 3 reloaded - Use AWK ===

This method is the same as the previous one except we use an inline awk script to replace any non-alpha-digit-underscore characters in the property names with underscore. This solves the problem the previous method has when it processes properties that have '.' in the name, for example.

Some clever person could probably figure out how to do the same thing with sed, but awk works.

Steps:

:# Create a temporary file.
:# Stream the properties file to stdout and (optionally) filter out dos style linefeeds.
:# Split each line using "=" as the field separator. If there are exactly two fields, then substitute the non-identifier characters with underscore in the first one, and print the second field with quotes around it.
:# If the line doesn't have an =, or seems to have more than two values, just print it out.
:# Source in the temporary file and delete it.

TEMPFILE=$(mktemp)
cat somefile.properties | dos2unix |
awk '
 BEGIN { FS="="; } \
 /^\#/ { print; } \
 !/^\#/ { if (NF == 2) { n = $1; gsub(/[^A-Za-z0-9_]/,"_",n); print n "=\"" $2 "\""; } else { print; } }' \
>$TEMPFILE
source $TEMPFILE
rm $TEMPFILE

=== Putting It Into Play ===

PROP2SH="$HOME/bin/prop2sh.sh"
PROPERTIES="$HOME/homeSettings.properties"

# convert the properties to a shell script that can be sourced
TMP_PROPS=/var/tmp/${PLATFORM_PROPERTIES_FILENAME}.$$
rm -f $TMP_PROPS
cat $PROPERTIES | $PROP2SH > $TMP_PROPS
[ $? -ne 0 ] && echo "Error: unable to convert properties to shell script" $$ return 1

# source the parsed properties
set -v
. $TMP_PROPS
set +v
rm -f $TMP_PROPS

unset TMP_PROPS PROPERTIES PROP2SH

The file "$HOME/bin/prop2sh.sh":

#!/bin/sh
#
# Filter to convert a .properties file to a .sh file that can be sourced
#   converts . to _ in property names
#
# Use this script in a pipe:
#   cat something.properties | prop2sh > something.sh
#
# Note: although Java properties can contain forward references, this does
#       not work for shell variables.  Therefore, if you use variables in
#       the value portion of a property, ensure that variable is defined
#       earlier in the file.
#       E.g:
#          some.property=this and ${another.property}
#          another.property=another value
#          ==> WILL NOT WORK -- reverse the lines

function change_var_name
{
    var="$*"

    # replace '.' with '_'
    var=${var//./_}

    echo "$var"
}
function change_var_value
{
    value="$*"

    # exit if there are no variables with periods
    echo $value | grep -qv '\${.*\..*}' && echo "$value" && return

    # parse out the prefix, the variable and the suffix
    prefix=${value%%\${*}
    var_suffix=${value#*\${}
    var=${var_suffix%%\}*}
    suffix=${var_suffix#*\}}

    # recursively change the suffix
    suffix=`change_var_value "$suffix"`

    # reconstruct the value
    var=`change_var_name $var`
    value="${prefix}\${$var}${suffix}"

    # return the value
    echo "$value"
}
while read LINE
do
    # simply output the line if it's blank or a comment
    echo $LINE | grep -q ^# && echo "$LINE" && continue
    echo $LINE | grep -q ^$ && echo "$LINE" && continue

    # parse out the property name and value
    RAWNAME=${LINE%%=*}
    RAWVALUE=${LINE#*=}

    # change the name and value appropriately
    NAME=`change_var_name $RAWNAME`
    VALUE=`change_var_value "$RAWVALUE"`

    # strip surrounding quotes (to be added)
    VALUE=${VALUE#\"}
    VALUE=${VALUE%\"}

    # output the .sh compatible line
    echo "export $NAME=\"$VALUE\""

done

== Fun with Variables ==

What programming language would be complete without variables? Naturally bash provides shell variables which are generally set/assigned a simple equals statement using following syntax with no white space around the equals sign:
VAR=value

=== Expand Variables In A File ===

Need to somehow expand the "shell variables" in a file to create a new "expanded file"? Try this:

eval "cat <<− E
$(paste /dev/null file)
E"

Note: Using the $(paste /dev/null file) is important because it prevents a line with E in the input file from ending the external cat prematurely.

=== Default Values for Variables ===

If value is not given, the variable is assigned the null string. In shell program it is quite useful to provide default value for variables. For example consider rsync.sh script:
#!/bin/bash
RSRC=$1
LOCAL=$2
rsync -avz -e 'ssh ' user@myserver:$RSRC $LOCAL

This script can be run as follows:
$ ./rsync.sh /var/www .
$ ./rsync.sh /home/vivek /home/vivek

It will sync remote /home/vivek directory with local /home/vivek directory. But if you need to supply default values for a variable you can write as follows:

#!/bin/bash
RSRC=$1
LOCAL=$2
: ${RSRC:="/var/www"}
: ${LOCAL:="/disk2/backup/remote/hot"}
rsync -avz -e 'ssh ' user@myserver:$RSRC $LOCAL

: ${RSRC:="/var/www"} ==> this means if the variable RSRC is not already set, set the variable to /var/www. You can also write same statement with following code:

if [ -z "$RSRC" ]
then
RSRC="/var/www"
fi

You can also execute a command and set the value to returned value (output). For example if the variable NOW is not already set, execute command date and set the variable to the todays date using date +"%m-%d-%Y":

#!/bin/bash
NOW=$1
....
.....
: ${NOW:=$(date +"%m-%d-%Y")}
....

=== Set A Variable From A Here Document ===

It is possible to set a variable from the output of a here document. This is actually a devious form of command substitution.

variable=$(cat <set -- *' will take the files matching the * glob and assign them to $* (the input parameters for a function), (ie. $* = $1 $2 $3 $4 ....)

set -- *; echo $*

=== basename and dirname ===

Many times, scripters will use external commands like ''basename'' and ''dirname'' and ''tr'' because they don't realize they can instead use ksh/bash builtins. An added bonus is the builtins are faster and require less system resources because no sub-process is spawned. Here is how to get ''basename'' and ''dirname'' using the shell builtins:

basename=${file##*/}
dirname=${file%/*}

However in shell scripts it is often necessary to convert a relative path name, i.e. "../usr/../lib/somefile" to an absolute path name starting with a slash "/", i.e. "/lib/somefile". The following code fragment does exactly this:

DIR=`dirname "$relpath"`
BASE=`basename "$relpath"`
ABS_PATH="`cd \"$DIR\" 2>/dev/null && pwd || echo \"$DIR\"`/$BASE"

==== Finding an Anchor Directory ====

Sometimes it it handy to let things run from where ever they may be. But you want to be able to reference a tree structure based on the script's location. This is a little tricky in a shell. But so long as you can guarantee the name of the "top" of your tree section name you can do this.

The "TopDIR" would be the name of the directory you are trying to get a lock on.

if [ -z "$SCRIPT_HOME" ] ; then
SCRIPT_HOME="$(cd ${0%/*} ; pwd)"
SCRIPT_HOME="${SCRIPT_HOME%%TopDIR/*}TopDIR"
fi

== Fun With Paths ==

=== Replace Path Item ===

Here is a cute function for doing PATH control. I consider it a more intelligent way to make sure something is on the PATH.

# this code replaces an old path item with a new one
# //$ This function will be deprecated and replaced with scripts in $SO_PLATFORM_HOME/bin
function ReplacePathItem
{
    OLDITEM="$1"
    NEWITEM="$2"

    case "$PATH" in
        *$OLDITEM*)
            PATH=`echo $PATH | sed "s;$OLDITEM;$NEWITEM;g"`
            ;;
        *$NEWITEM*)
            ;;
        :*)
            PATH=${NEWITEM}$PATH
            ;;
        "")
            PATH=$NEWITEM
            ;;
        *)
            PATH=$NEWITEM:$PATH
            ;;
    esac
}

OLD_JAVA_HOME=$JAVA_HOME

JAVA_HOME=/My/Alternate/Java/Home

if [ ! -z "$OLD_JAVA_HOME" ]
then
    ReplacePathItem "$OLD_JAVA_HOME/bin" "$JAVA_HOME/bin"
else
    export PATH=$JAVA_HOME/bin:$PATH
fi

unset OLD_JAVA_HOME
unset -f ReplacePathItem

=== Clean Up The Path ===

You might want to not have the error messages under normal operations as those can be annoying as well as screw up other scripts.

# Clear out duplicate or bad PATH dirs
NEWPATH=""

for ITEM in `echo $PATH | tr ':' ' '`; do
    [ ! -d $ITEM ] && echo "$ITEM from PATH is not a directory skipping" && continue
    [ ! -r $ITEM ] && echo "$ITEM from PATH is not readable skipping" && continue
    case "$NEWPATH" in
        ?(*:)$ITEM?(:*))
            echo "duplicate PATH entry for $ITEM"
            ;;
        *:)
            NEWPATH=${ITEM}$NEWPATH
            ;;
        "")
            NEWPATH=$ITEM
            ;;
        *)
            NEWPATH=$NEWPATH:$ITEM
            ;;
    esac
done
PATH=$NEWPATH

unset NEWPATH

== The Wonders of Set ==

set -a and set -e also do some useful things. set -a (AFAIR) will export all variables to sub-shells and child processes. set -e will die on the first error received when you're running multiple commands at once (eg. cd rob; ls; touch readme; rm readme etc.)

One useful note is that you may use $@ to preserve each as a separate parameter (AFAIR), one early mistake I made was not to double-quote it:
main "$@" is the proper way to pass the current args on to a function in order to preserve them.

== Command History Expansion ==

Setting bash command history expansion to use the up arrow like MATLAB. This is one of those simple once you know it but annoying to figure out settings. Via google, I found two solutions:

* Solution 1: Add the following lines to “.inputrc” in your home directory:
:

"\e[A": history-search-backward
"\e[B": history-search-forward

* Solution 2: Add the following lines to “.profile” or “.bashrc”:
:

# make bash autocomplete with up arrow
bind '"\e[A":history-search-backward'
bind '"\e[B":history-search-forward'

== Tab Command Completion ==

OK so there are is the whole auto command complete thing. Should the tab key list the available commands or cycle through the options? The jury in my head is still out on which behavior I prefer. I wish they would hurry up and decide so I can get used to it.

Again via Google, I found two solutions:

* Solution 1: Add the following lines to “.inputrc” in your home directory:
:

TAB: menu-complete

* Solution 2: Add the following lines to “.profile” or “.bashrc”:
:

# make tab cycle through commands instead of listing
bind '"\t":menu-complete' 

== Arithmetic Evaluations ==

Bash can perform arithmetic evaluations. They are much easier to using expr tool. Below are some examples. Note the absence of $ prefix for variable names.

* increment value of variable 'count' by one
:

((count++))

* set total = total+current
:

((total+=current))

* ternary expression
:

((current>max?max=current:max=max))

== Commenting Out ==

One line of shell code can be "commented out" using the "#" character. Sometimes however it would be nice to "comment out" more than one line of code, like the C "/* */" comments.

One way to comment out multiple lines is to use the ":" command (that returns "true") and the "here document" in non-interpreted mode. It looks like this:

: << '--END-COMMENT--'
your comment here
--END-COMMENT--

This way, there is no restriction with single quotes. Note that any delimiter can be used in place of "--END-COMMENT--".

=== Controlling Output On The Fly ===

You can quickly comment or uncomment commands by using a variable. This could be triggered by a command line argument if you wanted. The basic approach is this:

# Comment out one of the two VAR lines below depending on
# whether you want comments turned on or off
#
# The VAR line below, when uncommented, will comment out all commands
# with $VAR in front of them
#VAR=": #"; export VAR
#
# The VAR line below, when uncommented, will allow execution of all
# commands with $VAR in fron of them.
VAR=""; export VAR
#
#-----MAIN
$VAR date
--MORE CODE--

== Setting default values for variables ==

In shell scripts it's often useful to provide default values for script variables, i.e.

if [ -z "$Host" ]
then
Host=`uname -n`
fi

For this kind of assignment the shell has a shorthand:

: ${Host:=`uname -n`}

This means: if the variable "Host" is not already set, execute the command "uname -n" and set the variable to the returned value.

== tr replacement ==

Many times, scripters will use the external command ''tr'' because they don't realize they can instead use ksh/bash builtins. An added bonus is the builtins are faster and require less system resources because no sub-process is spawned. Here is how to get ''tr'' using the shell builtins:

$ word="MiXeD"
$ # replaced: word=$(echo $word | tr [A-Z] [a-z])
$ typeset -l word
$ echo $word
mixed
$ # replaced: word=$(echo $word | tr [a-z] [A-Z])
$ typeset -u word
$ echo $word
MIXED

== Debugging ==

The easiest way to enable shell debugging is via the -x option to the shell, which enables command tracing. The -x can either be specified on the command line, or on the first line of the shell script.

$ cat somescript
#!/bin/sh
echo foo
$ sh -x somescript
+ echo foo
foo

$ perl -i -ple 's/sh/sh -x/' somescript
$ cat somescript
#!/bin/sh -x
echo foo
$ chmod +x somescript
$ ./somescript
+ echo foo
foo

Sometimes the following flags are useful, too:

set -v # print out info on what would be executed instead of executing
set -e # terminate the script at first error
set -u # unset variables are fatal errors

== Neat bash Tricks ==

* Replace 'orig' in previous command with 'repl' and execute the new command
:

^orig^repl^

* Copy 'filename' as 'filename.bak'
:

cp filename{,.bak}

* Create dir1/subdir1 dir1/subdir2 dir2/subdir1 dir2/subdir2 dir3/subdir1 dir3/subdir2
:

mkdir -p dir{1,2,3}/subdir{1,2}

BODY {
counter-reset: section; /* Create a section counter scope */
}
H2:before { /* negate the TOC H2 header to prevent accidental increment */
counter-increment: section; /* Add 1 to section */
content: "Section " counter(section) ". ";
}
#toc H2:before { /* put in the TOC H2 header formating */
counter-increment: section 0; /* stop section from incrementing here */
content: " "; /* Remove the section notation from the contents header */
}
H2 {
counter-reset: subsection1; /* Set subsection1 to 0 */
color: #800517 ; /* Firebrick */
}
#toc H2 { /* make the TOC H2 header formating right */
color: black ;
}
H3:before {
counter-increment: subsection1;
content: counter(section) "." counter(subsection1) " ";
}
H3 {
counter-reset: subsection2; /* Set subsection2 to 0 */
color: #306754 ; /* Medium Sea Green */
}
H4:before {
counter-increment: subsection2;
content: counter(section) "." counter(subsection1) "." counter(subsection2) " ";
}
H4 {
counter-reset: subsection3; /* Set subsection to 0 */
color: #AF7817; /* Dark Goldenrod */
}
H5:before {
counter-increment: subsection3;
content: counter(section) "." counter(subsection1) "." counter(subsection2) "." counter(subsection3) " ";
}

Thread Slivers eBook at Amazon