Regular Expression

My colleague was asking me how to list out the settings in the default /etc/squid/squid.conf. This file has over 4325 lines with 4030 comments, 260 blank lines and only 35 settings (in RedHat). You can see the configuration file is heavily commented and it is extremely hard to locate the settings.

Regular expression is to the rescue. Here I am going to walk you through how we can come to the final solution

1. grep -v # /etc/squid/squid.conf
   This will give you those lines without (-v) the occurrence of '#', but this will miss lines such as "acl Safe_ports port 80 # http"
2. egrep -v '^#' /etc/squid/squid.conf
   This "Extended Grep" is able to understand regular expression in the pattern. ^ is an anchor and it represents the start of the line. '^#' means matching lines start with #. What if my setting starts with a blank space and follows by the comment
3. egrep -v '^[ \t]*#' /etc/squid/squid.conf
   Anything inside the square bracket matches a single character that is contained within the brackets. In our case, the character set is a space and a tab. Since we cannot represent a tab as a literal character, we have to represent it as a escape sequence "\t". [ \t]* matches the preceding element (blank space) zero or more times. Although we can get rid of the comment, we still have a lot blank lines to deal with.
4. egrep -v '^[ \t]*#' /etc/squid/squid.conf | egrep -v '^$'
   How about taking advantage of a pipe to run through the previous step's output and apply another 'egrep' to get rid of the blank line. ^$ are anchors, start of line and end of line, i.e. no character in the line. Ok, that's what we want, but can we do with just a single egrep. Of course we can.
5. egrep -v '(^[ \t]*#|^$)' /etc/squid/squid.conf
   With the ability of grouping "()" and choice "|", we are telling egrep that match either comment or blank line. What if the blank lines are not really blank, but contains spaces or tabs
6. egrep -v '(^[ \t]*#|^[ \t]*$)' /etc/squid/squid.conf
   This will do the job!

If your command understands POSIX compliant regular expression, you can write it in a more compact syntax:
egrep -v '(^\s*#|^\s*$)' /etc/squid/squid.conf
\s is equivalent to [ \t\r\n\v\f], this character set is called whitespace characters (space, tab, carriage return, newline, vertical tab, form feed)

Regular expression is definitely your life saver if you need to manlipulate data. Do you know that lots of other commands have regular expression support built-in. Run this to find out what commands(1) has this support:

cd /usr/share/man/man1
for i in *gz
do
    zgrep -li regexp $i
done

BTW, sed (stream editor) can do the same job but without applying an inverted match (-v):
sed -e '/^\s*#/d;/^\s*$/d' /etc/squid/squid.conf

In Praise of Scripting: Real Programming Pragmatism

IEE Computer Society's July 2008 edition has published an article - In Praise of Scripting: Real Programming Pragmatism (you need to pay $19 to IEEE to view the original article, alternatively you can click here to view the draft). In the abstract, the author recommends that scripting, not Java, be taught first, asserting that students should learn to love their own possibilities before they learn to loathe other people's restrictions.

Eric Wendelin, a Software Engineer at Sun Microsystems mentioned in his blog: What I wanted to know before I left college: A programmer reflects. As he pointed out:

" ... Summer internship that forced me to use computing languages that I had not touched before: Perl, PHP, and other CL tools in a mostly command-line Linux environment - Going outside your comfort zone ended up being HUGE in my career because I realized how to pick up technologies and try to build something useful with them. ..."

I totally agreed that computer science students should be exposed to various type of programming langauges / operating systems during their undergraduate study. BTW, if a language or an operating system can survive that long (UNIX was first developed in 1969, Bourne shell was released in 1977, AWK was written in 1977, Perl / Python / Tcl were developed in the late 1980s), sure it has its own niche area.