Sep 19

Commands and the shell

Remember that it is the shell that interprets your commands

Two examples: bash and tcsh
ls -d V* means the same in both
but ls -d [!V]* works only in bash; it lists files and directories that dont start with the character "V".

How to check which shell you are running?
- One way is to run the ps command.

Searching within files

grep-ping

ls *Naren* lists files (and contents of directories) whose name contains "Naren" in them.
grep Naren testfile searches through the contents of textfile and displays lines that contain the word "Naren" in them.
grep Naren * searches through all files available and displays lines that contain the word "Naren" in them.
Note that you needed to say "*Naren*" for ls but just "Naren" for grep. By default, grep looks for the given string anywhere on a line. By default, ls looks for the given string as the exact name of a file.

More subtle differences

Wildcards versus regular expressions

grep Naren NOTES searches through the NOTES file and lists lines containing "Naren". Let us suppose this gives four lines.
grep N* NOTES lists zero lines! How can this be, when we know that there are at least four lines matching the pattern "N*"? The difference is that the expression "N*" is interpreted by the shell (not by grep). So, the shell looks for files whose names have an N followed by zero or more characters. In this case, the only file that matches is NOTES. So, the shell expands the given command to grep NOTES NOTES before passing it to grep. Of course, there are no lines containing NOTES in the file NOTES!
To pass the argument to grep without expanding or otherwise evaluating it, you need to double quote the pattern. Thus, grep "N*" NOTES is probably what you wanted to say.

Not so fast!

Even this doesn't work!

The above command lists all lines of the file NOTES, even though many of them don't contain an uppercase "N". Do a grep "N*" NOTES | wc -l and compare it to wc -l NOTES. Something is fishy.
The reason is that the expression "N*" means different things depending on whether it is interpreted by the shell or grep. In the shell "*" means "zero or more characters". In grep, "*" means "zero or more repetitions of the previous character" (in this case, "N"). So, any line will match!
So, what we really want to say is something like grep "N.*" NOTES which says find an "N followed by zero or more repetitions of any character (.)".

Learning to grep

Grep and its variants (most popularly, egrep) are widespread in text processing. Checkout A9.com, a subsidiary of Amazon.com that allows fast full-text searching on entire books. Udi Manber, a computer scientist who helped created many early versions of grep is now the chief scientist there. A9 apparently is a pun for "A followed by 9 characters", or "algorithm". egrep is what we will use from now on.
- Anchors, character sets, and repetitions/modifiers are the three main parts to a egrep command.
- There are two types of anchors: "^" anchors the pattern to the beginning of the line; "$" anchors it to the end of the line.
- Character sets are precisely that: ABC matches only the three-character string ABC. "." matches any character. To literally match a period, escape it: "\.". [ABC] matches only one character, either an A or a B or a C. [A-Z] does what you think it does, matches any character from A to Z. But be carefull that the LC_ALL flag is set to "C" (do a man egrep to see why you need to do this; the way to do it in the bash shell is "export LC_ALL=C".). Otherwise [A-Z] will match both lowercase and uppercase characters. [^ABC] negates the meaning, i.e., match only if it is NOT one of these three characters.
- The standard modifiers are: "*" (meaning zero or more occurrences of the previous pattern), "+" (one or more occurrences), "?" (zero or one, i.e., an optional occurrence). {3} means find three occurrences of the previous pattern. {3,8} means find at least three and upto 8 occurrences of the previous pattern.
- Other constructs you might find useful are "|": for optional parts; and "()": to group patterns (to build more complex patterns).
- The -n option to egrep also spits out line numbers, in addition to matching lines.

Egrep examples

Assume you are given the file sample.txt. In class, we handed out a printed copy of the output of cat -n sample.txt (so that you can see the line numbers) and wrote patterns to:

Find all lines referencing the supreme court.
```
egrep -n "[Ss]upreme court" sample.txt
```
Find all lines containing references to names. A reference begins with a "Mr.", "Mrs.", "Ms.", or "Dr.", followed by a space, followed by a word.
```
egrep -n "(Mr\.|Mrs\.|Ms\.|Dr\.) [A-Za-z]+" sample.txt
```
Find all lines containing references to names at the beginning of the line.
```
egrep -n "^(Mr\.|Mrs\.|Ms\.|Dr\.) [A-Za-z]+" sample.txt
```
Find all lines containing references to real people. A "real people" name begins with a "Mr.", "Mrs.", "Ms.", or "Dr.", followed by a space, followed by a first name, a mandatory space, an optional middle initial (in which case it will be followed by a period and a space, i.e., ". "), and a last name.
```
egrep -n "(Mr\.|Mrs\.|Ms\.|Dr\.) [A-Za-z]+ ([A-Za-z]\. )?[A-Za-z]+" sample.txt
```
Find all lines containing social security numbers.
```
egrep -n "[0-9]{3}-[0-9]{2}-[0-9]{4}" sample.txt
```