All of bash history revisited: load time solved plus directory history

  • strict warning: Non-static method view::load() should not be called statically in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/views.module on line 906.
  • strict warning: Declaration of views_handler_argument::init() should be compatible with views_handler::init(&$view, $options) in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/handlers/views_handler_argument.inc on line 744.
  • strict warning: Declaration of views_handler_filter::options_validate() should be compatible with views_handler::options_validate($form, &$form_state) in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/handlers/views_handler_filter.inc on line 607.
  • strict warning: Declaration of views_handler_filter::options_submit() should be compatible with views_handler::options_submit($form, &$form_state) in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/handlers/views_handler_filter.inc on line 607.
  • strict warning: Declaration of views_handler_filter_boolean_operator::value_validate() should be compatible with views_handler_filter::value_validate($form, &$form_state) in /hermes/walnaweb12a/b57/moo.greydragoncom/nodsw/sites/all/modules/views/handlers/views_handler_filter_boolean_operator.inc on line 159.
Leeland's picture
Imagine logging into a box after a nice long three day weekend and asking yourself "self, what was I doing last Thursday?" or "what did I do on this server when I was here nearly a month ago?" What if you could answer that in such detail as what you did, where you were when you did it and you could get back there with only a few key strokes? Check this out (opening a new ssh session to my production box I haven't been on for weeks):
Using username "a-lartra".
Authenticating with public key "imported-openssh-key" from agent
Last login: Tue Mar  6 09:17:07 2012 from ---.---.---.---
[a-lartra@jira002 ~]$ hh 30
[2012-02-21 15:04:15] ~~~ /home/a-lartra ~~~ find /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64 -name \*certs
[2012-02-21 15:04:23] ~~~ /home/a-lartra ~~~ find /usr/lib/jvm/ -name \*certs
[2012-02-21 15:05:03] ~~~ /home/a-lartra ~~~ cd SSL
[2012-02-21 15:05:11] ~~~ /home/a-lartra/SSL ~~~ vi SSLPoke.java
[2012-02-21 15:05:53] ~~~ /home/a-lartra/SSL ~~~ vi loadCerts.sh
[2012-02-21 15:07:03] ~~~ /home/a-lartra/SSL ~~~ chmod a+x loadCerts.sh
[2012-02-21 15:07:07] ~~~ /home/a-lartra/SSL ~~~ ./loadCerts.sh
[2012-02-21 15:07:46] ~~~ /home/a-lartra/SSL ~~~ openssl s_client -showcerts -connect jira.corp.com:8443
[2012-02-21 15:09:13] ~~~ /home/a-lartra/SSL ~~~ vi *.cert
[2012-02-21 15:09:50] ~~~ /home/a-lartra/SSL ~~~ cat loadCerts.sh
[2012-02-21 15:09:56] ~~~ /home/a-lartra/SSL ~~~ JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
[2012-02-21 15:09:57] ~~~ /home/a-lartra/SSL ~~~ CERT_FILE=${JAVA_HOME}/lib/security/cacerts
[2012-02-21 15:10:24] ~~~ /home/a-lartra/SSL ~~~ keytool -keystore ${CERT_FILE} -import -file XXXX.corp.com.cert -alias jira.corp.com
[2012-02-21 15:10:44] ~~~ /home/a-lartra/SSL ~~~ vi XXXX.corp.com.cert
[2012-02-21 15:11:16] ~~~ /home/a-lartra/SSL ~~~ keytool -keystore ${CERT_FILE} -import -file XXXX.corp.com.cert -alias XXXX.corp.com.cert
[2012-02-21 15:13:16] ~~~ /home/a-lartra/SSL ~~~ cd /tmp/SSL
[2012-02-21 15:14:06] ~~~ /tmp/SSL ~~~ cat loadCerts.sh
[2012-02-21 15:22:52] ~~~ /tmp/SSL ~~~ java SSLPoke jira.corp.com 8443
[2012-02-23 11:29:25] ~~~ /tmp/SSL ~~~ cd ~
[2012-02-23 11:21:05] ~~~ /home/a-lartra ~~~ more db.py
[2012-02-23 11:21:08] ~~~ /home/a-lartra ~~~ vi db.py
[2012-02-23 11:29:25] ~~~ /home/a-lartra ~~~ cd SSL/
[2012-02-23 11:30:02] ~~~ /home/a-lartra/SSL ~~~ vi JavaEnv.java
[2012-02-23 13:50:45] ~~~ /home/a-lartra/SSL ~~~  # end session a-lartra@jira002:/dev/pts/0
[2012-02-28 12:47:45] ~~~ /home/a-lartra ~~~ ping jira
[2012-02-28 12:49:08] ~~~ /home/a-lartra ~~~ ssh jira001.corp.com
[2012-02-28 15:22:55] ~~~ /home/a-lartra ~~~ man rsync
[2012-02-28 16:13:10] ~~~ /home/a-lartra ~~~  # end session a-lartra@jira002:/dev/pts/1
[a-lartra@jira002 ~]$ hd 5
/home/a-lartra/SSL
/tmp/SSL
/home/a-lartra
/home/a-lartra/SSL
/home/a-lartra
[a-lartra@jira002 ~]$ cd --
 0  ~
 1  ~/SSL
 2  /tmp/SSL
 3  ~/fish
 4  ~/fish/sql
 5  ~/fish/uploads
 6  ~/fish/config
[a-lartra@jira002 ~]$ cd -2
[a-lartra@jira002 SSL]$
Sweet! I know exactly what, were, and when I last did, which brings all that stuff back to the front of my brain pan. All this in under 10 seconds! Seriously logging what you did is nothing new. People have been keeping journals since the beginning of written language. In computing this should be a simple task, and let’s face it history files from interactive shells is a serious time saver, memory jogger, and what the heck did I do answerer. Except that almost all of the Linux shells (sh, ksh, bash, csh, zsh, etc.) have the annoying feature that is as the shell exits it overwrites the history file with its history buffer contents. Net result last shell to exit wins the who’s history is saved game. As stated in a previous blog (http:///blog/leeland/2011/10/26-all-bash-history-forever-and-across-multiple-sessions) only having one interactive shell per machine is not a real solution. I already had a solution to this in place (see http://nodsw.com/blog/leeland/2011/10/26-all-bash-history-forever-and-across-multiple-sessions for details). But a side trip into the internet landed me on an interesting page: http://stackoverflow.com/questions/945288/saving-current-directory-to-bash-history. Being of the curious nature type I then searched to see if anyone did anything more with this and found a neat application of the idea by Jeet Sukumaran written up in a blog titled “Supplementary Command History Logging in Bash: Tracking Working Directory, Date/Times, etc.” (http://jeetworks.org/node/80) which of course got my interest. My solution took care of:
  1. Keeping track of every command on each unique tty/pty session.
  2. Capturing every command even the very last one causing an exit
  3. Logging when tty’s disconnected to the history
  4. Initializing each new shell’s history to the most current history of commands.
  5. Human readable history log (time stamps made sense).
Issue with my original solution was:
  1. Long login/new shell pauses as the history date computation were done.
  2. A number of external programs were required to be called (I am picky, I like things to be simple and really the original solution grew to a fairly complex solution so it annoyed me).
The idea of a built in function for logging seemed a lot cleaner to me. Further the idea that by judicial use of built in regular expression handling combined with a better field key really made me sit back and smile. Of course the added benefit of having a history of directories I was in, knowing where I was when I issued a particular command, and being able to have a stack of recently visited directories was very compelling. I added in a few minor changes and mixed the ideas of history log function with the cd stack function and my pre-loading new shells histories to resolve not only the limitations of my original solution but adding significantly handy new features. I'll leave the analysis of what I am doing to you. How to do this is pretty simple:
  1. Configure history file location to be unique to each system
  2. Make the history file size very large
  3. Configure history to not bother capturing uninteresting things like “ls”
  4. Setting HISTTIMEFORMAT to be a parseable but readable form
  5. Load two functions from some shell files (or just drop them in your bashrc if you want)
  6. On disconnect make sure to grab the last command
  7. Profit
There are three files to this solution:
  1. An entry into ${HOME}/.bashrc
  2. The history log function I store in ${HOME}/bin/a_loghistory_func.sh
  3. The cd override function I store in ${HOME}/bin/acd_func.sh

1. ${HOME}/.bashrc.user


# are we an interactive shell?
if [ "$PS1" ]; then

  HOSTNAME=`hostname -s || echo unknown`

  # add cd history function
  [[ -f ${HOME}/bin/acd_func.sh ]] && . ${HOME}/bin/acd_func.sh
  # make bash autocomplete with up arrow
  bind '"\e[A":history-search-backward'
  bind '"\e[B":history-search-forward'
  ##################################
  # BEG History manipulation section

    # Don't save commands leading with a whitespace, or duplicated commands
    export HISTCONTROL=ignoredups

    # Enable huge history
    export HISTFILESIZE=9999999999
    export HISTSIZE=9999999999

    # Ignore basic "ls" and history commands
    export HISTIGNORE="ls:ls -al:ll:history:h:h[dh]:h [0-9]*:h[dh] [0-9]*"

    # Save timestamp info for every command
    export HISTTIMEFORMAT="[%F %T] ~~~ "

    # Dump the history file after every command
    shopt -s histappend
    export PROMPT_COMMAND="history -a;"
    [[ -f ${HOME}/bin/a_loghistory_func.sh ]] && . ${HOME}/bin/a_loghistory_func.sh

    # Specific history file per host
    export HISTFILE=$HOME/.history-$HOSTNAME

    save_last_command () {
        # Only want to do this once per process
        if [ -z "$SAVE_LAST" ]; then
            EOS=" # end session $USER@${HOSTNAME}:`tty`"
            export SAVE_LAST="done"
            if type _loghistory >/dev/null 2>&1; then
                _loghistory
                _loghistory -c "$EOS"
            else
                history -a
            fi
            /bin/echo -e "#`date +%s`\n$EOS" >> ${HISTFILE}
        fi
    }
    trap 'save_last_command' EXIT

  # END History manipulation section
  ##################################

  # Preload the working directory history list from the directory history
  if type -t hd >/dev/null && type -t cd_func >/dev/null; then
      for x in `hd 20` `pwd`; do cd_func $x ; done
  fi
fi

2. ${HOME}/bin/a_loghistory_func.sh


export HOSTNAME=`hostname -s || echo unknown`

_loghistory() {

# Detailed history log of shell activities, including time stamps, working directory etc.
#
# Based on '_loghistory' by Jeet Sukumaran - 2011-11-23
# (http://jeetworks.org/node/80)
# Based on 'hcmnt' by Dennis Williamson - 2009-06-05 - updated 2009-06-19
# (http://stackoverflow.com/questions/945288/saving-current-directory-to-bash-history)
#
# Add this function to your '~/.bashrc':
#
# Set the bash variable PROMPT_COMMAND to the name of this function and include
# these options:
#
#     e - add the output of an extra command contained in the histentrycmdextra variable
#     h - add the hostname
#     y - add the terminal device (tty)
#     c - add a comment to the log
#     n - don't add the directory
#     t - add the from and to directories for cd commands
#     l - path to the log file (default = ${HOME}/.history_log.${HOSTNAME})
#     ext or a variable
#
# See bottom of this function for examples.
#

    # make sure this is not changed elsewhere in '.bashrc';
    # if it is, you have to update the reg-ex's below
    export HISTTIMEFORMAT="[%F %T] ~~~ "

    local script=$FUNCNAME
    local histentrycmd=
    local cwd=
    local comment=
    local extra=
    local text=
    local logfile="${HOME}/.history_log.${HOSTNAME}"
    local hostname=
    local histentry=
    local histleader=
    local datetimestamp=
    local histlinenum=
    local options=":hyntel:c:"
    local option=
    OPTIND=1
    local usage="Usage: $script [-h] [-y] [-n|-t] [-e] [text] [-l logfile]"

    local CommentOpt=
    local ExtraOpt=
    local NoneOpt=
    local ToOpt=
    local tty=
    local ip=

    # *** process options to set flags ***

    while getopts $options option
    do
        case $option in
            h ) hostname=$HOSTNAME;;
            y ) tty=$(tty);;
            n ) if [[ $ToOpt ]]
                then
                    echo "$script: can't include both -n and -t."
                    echo $usage
                    return 1
                else
                    NoneOpt=1       # don't include path
                fi;;
            t ) if [[ $NoneOpt ]]
                then
                    echo "$script: can't include both -n and -t."
                    echo $usage
                    return 1
                else
                    ToOpt=1         # cd shows "from -> to"
                fi;;
            c ) CommentOpt=1;       # This is just a comment add it and exit
                comment=$OPTARG;;
            e ) ExtraOpt=1;;        # include histentrycmdextra
            l ) logfile=$OPTARG;;
            : ) echo "$script: missing filename: -$OPTARG."
                echo $usage
                return 1;;
            * ) echo "$script: invalid option: -$OPTARG."
                echo $usage
                return 1;;
        esac
    done

    text=($@)                       # arguments after the options are saved to add to the comment
    text="${text[*]:$OPTIND - 1:${#text[*]}}"

    # add the previous command(s) to the history file immediately
    # so that the history file is in sync across multiple shell sessions
    history -a

    # grab the most recent command from the command history
    histentry=$(history 1)

    # parse it out
    histleader=`expr "$histentry" : ' *\([0-9]*  \[[0-9]*-[0-9]*-[0-9]* [0-9]*:[0-9]*:[0-9]*\]\)'`
    histlinenum=`expr "$histleader" : ' *\([0-9]*  \)'`
    datetimestamp=`expr "$histleader" : '.*\(\[[0-9]*-[0-9]*-[0-9]* [0-9]*:[0-9]*:[0-9]*\]\)'`
    histentrycmd=${histentry#*~~~ }

    # protect against relogging previous command
    # if all that was actually entered by the user
    # was a (no-op) blank line
    if [[ $CommentOpt ]]
    then
        if [[ -z $__PREV_COMMENT ]]
        then
            # first call with a comment save for next time
            export __PREV_COMMENT="$comment"
        elif [[ "$__PREV_COMMENT" == "$comment" ]]
        then
            # already added this comment
            return
        fi
    else
        if [[ -z $__PREV_HISTLINE || -z $__PREV_HISTCMD ]]
        then
            # new shell; initialize variables for next command
            export __PREV_HISTLINE=$histlinenum
            export __PREV_HISTCMD=$histentrycmd
            return
        elif [[ $histlinenum == $__PREV_HISTLINE  && $histentrycmd == $__PREV_HISTCMD ]]
        then
            # no new command was actually entered
            return
        else
            # new command entered; store for next comparison
            export __PREV_HISTLINE=$histlinenum
            export __PREV_HISTCMD=$histentrycmd
        fi
    fi

    if [[ -z $NoneOpt ]]            # are we adding the directory?
    then
        if [[ ${histentrycmd%% *} == "cd" || ${histentrycmd%% *} == "jd" ]]    # if it's a cd command, we want the old directory
        then                             #   so the comment matches other commands "where *were* you when this was done?"
            if [[ -z $OLDPWD ]]
            then
                OLDPWD="${HOME}"
            fi
            if [[ $ToOpt ]]
            then
                cwd="$OLDPWD -> $PWD"    # show "from -> to" for cd
            else
                cwd=$OLDPWD              # just show "from"
            fi
        else
            cwd=$PWD                     # it's not a cd, so just show where we are
        fi
    fi

    if [[ $ExtraOpt && $histentrycmdextra ]]    # do we want a little something extra?
    then
        extra=$(eval "$histentrycmdextra")
    fi

    if [[ $CommentOpt ]]
    then
        histentrycmd="${datetimestamp} ${tty:+[$tty] }${ip:+[$ip] }${extra:+[$extra] }~~~ ${hostname:+$hostname:}$cwd ~~~ ${comment}"
    else
        # strip off the old ### comment if there was one so they don't accumulate
        # then build the string (if text or extra aren't empty, add them with some decoration)
        histentrycmd="${datetimestamp} ${text:+[$text] }${tty:+[$tty] }${ip:+[$ip] }${extra:+[$extra] }~~~ ${hostname:+$hostname:}$cwd ~~~ ${histentrycmd# * ~~~ }"
    fi
    # save the entry in a logfile
    echo "$histentrycmd" >> $logfile || echo "$script: file error." ; return 1

} # END FUNCTION _loghistory

# dump regular history log
alias h='history'
# dump enhanced history log
#alias hh="cat ${HOME}/.history_log.${HOSTNAME}"
hh () {
    if [[ -z $1 ]]
    then
        cat ${HOME}/.history_log.${HOSTNAME}
    else
        tail -n $1 ${HOME}/.history_log.${HOSTNAME}
    fi
}
# dump history of directories visited
#alias hd="cat ${HOME}/.history_log.${HOSTNAME} | awk -F ' ~~~ ' '{print \$2}' | uniq"
hd () {
    if [[ -z $1 ]]
    then
        awk -F ' ~~~ ' -- '{print $2}' ${HOME}/.history_log.${HOSTNAME} | uniq
    else
        awk -F ' ~~~ ' -- '{print $2}' ${HOME}/.history_log.${HOSTNAME} | uniq | tail -n $1
    fi
}

export PROMPT_COMMAND='_loghistory'

3. ${HOME}/bin/acd_func.sh


# do ". acd_func.sh"
# acd_func 1.0.5, 10-nov-2004
# petar marinov, http:/geocities.com/h2428, this is public domain

cd_func ()
{
  local x2 the_new_dir adir index
  local -i cnt

  if [[ $1 ==  "--" ]]; then
    dirs -v
    return 0
  fi

  the_new_dir=$1
  [[ -z $1 ]] && the_new_dir=$HOME

  if [[ ${the_new_dir:0:1} == '-' ]]; then
    #
    # Extract dir N from dirs
    index=${the_new_dir:1}
    [[ -z $index ]] && index=1
    adir=$(dirs +$index)
    [[ -z $adir ]] && return 1
    the_new_dir=$adir
  fi

  #
  # '~' has to be substituted by ${HOME}
  [[ ${the_new_dir:0:1} == '~' ]] && the_new_dir="${HOME}${the_new_dir:1}"

  #
  # Now change to the new dir and add to the top of the stack
  pushd "${the_new_dir}" > /dev/null
  [[ $? -ne 0 ]] && return 1
  the_new_dir=$(pwd)

  #
  # Trim down everything beyond 11th entry
  popd -n +11 2>/dev/null 1>/dev/null

  #
  # Remove any other occurence of this dir, skipping the top of the stack
  for ((cnt=1; cnt <= 10; cnt++)); do
    x2=$(dirs +${cnt} 2>/dev/null)
    [[ $? -ne 0 ]] && return 0
    [[ ${x2:0:1} == '~' ]] && x2="${HOME}${x2:1}"
    if [[ "${x2}" == "${the_new_dir}" ]]; then
      popd -n +$cnt 2>/dev/null 1>/dev/null
      cnt=cnt-1
    fi
  done

  return 0
}

alias cd=cd_func

if [[ $BASH_VERSION > "2.05a" ]]; then
  # ctrl+w shows the menu
  bind -x "\"\C-w\":cd_func -- ;"
fi

Thread Slivers eBook at Amazon