Accountability, At The Shell Level

Recently one of my projects is having some 'accountability' problem. A service daemon was down at 3+am and we had no clue in why it was not responsing to requests. From the 'last' command, we realised that someone login to the server but we were not able to trace what had been done by the user. BTW, this user owns that service daemon and he/she has all the rights to start/stop the service.

To avoid such incident to happen again, may be we can 'track' that user activity. In Solaris, I can turn on the C2 log as what I mentioned in my previous blog. However, that service daemon is running on Linux.

One quick fix is to tap on to the history capability of Bash shell. According to the man page of bash, these are the variables that we can set to control the command history

       HISTTIMEFORMAT
              If  this  variable  is  set and not null, its value is used as a
              format string for strftime(3) to print the time stamp associated
              with  each  history  entry displayed by the history builtin.  If
              this variable is set, time stamps are  written  to  the  history
              file so they may be preserved across shell sessions.
       HISTFILE
              The name of the file in which command history is saved (see HIS-
              TORY  below).   The default value is ~/.bash_history.  If unset,
              the command history is  not  saved  when  an  interactive  shell
              exits.
       HISTFILESIZE
              The maximum number of lines contained in the history file.  When
              this variable is assigned a value, the  history  file  is  trun-
              cated,  if necessary, by removing the oldest entries, to contain
              no more than that number of lines.  The default  value  is  500.
              The history file is also truncated to this size after writing it
              when an interactive shell exits.
       HISTSIZE
              The number of commands to remember in the command  history  (see
              HISTORY below).  The default value is 500.

It is possible to set up the history file to be located in another location instead of the default $HOME/.bash_history. We can set that in /etc/profile like this where the file is unique for every login session based on time and process id.

_HISTDIR="/history/`whoami`"
[ -d $_HISTDIR ] || mkdir $_HISTDIR
HISTFILE="$_HISTDIR/`date '+%Y%m%d%H%M%S'`-$$"
HISTFILESIZE=3000
HISTSIZE=3000
HISTTIMEFORMAT="%Y-%m-%dT%H:%M:%S "

The /history has to made with sticky bit on so that any user can create directory and own the content. Because of the HISTTIMEFORMAT, timestamp is also registered in the history file.

$ cat /history/`whoami`/20081017220353-16324
#1224252235
history
#1224252247
cd /history/chihung/
#1224252248
ls
#1224252249
ls -l
#1224252252
more 20081017220337

Now we should get some accountability.