Lecture 4. Pattern Matching

Lecture 4. Click here to download the "theory" part of today's lecture. What you see below is the highlights + a number of practical examples.

Commands and the shell (carry-over from the last class )

Remember that it is the shell that interprets your commands

Examples: C-shell (and tcsh -- the improved version of it)
Here is a (well commented) example of a .cshrc file which defines variables that control your C-shell. Example

How to check which shell you are running?
- One way is to run the ps command.

Searching within files

grep-ping
Example sample text
- ls *Alexey* lists files (and contents of directories) whose name contains Alexey in them.
- grep "Alexey" testfile searches through the contents of textfile and displays lines that contain the word Alexey in them.
- grep Alexey * searches through all files available and displays lines that contain the word Alexey in them.
- Note that you needed to say *Alexey* for ls but just Alexey for grep. By default, grep looks for the given string anywhere on a line. By default, ls looks for the given string as the exact name of a file.

More subtle differences

Wildcards versus regular expressions -- similar but not the same. Be careful! When using grep always put the experession in " ", as in grep "Alexey" sample.txt to be on the safe side. Wildcards are interpreted by the shell, and the rules are slightly different. By using " " you make sure that the regular expression goes dierectly into grep , without being interpreted (and modified) by the shell. This is especialyy important when you use expressions that contain metacharacters like "." or "$". To grep for expressions containing those use escaping and single quotes , e.g. grep '\$' textfile to search for a string containg the dollar sign. There is a very subtle difference between the single and double quotes in this context which is not worth spending time on.

Learning to grep

Grep and its variants (most popularly, egrep) are widespread in text processing. Checkout A9.com, a subsidiary of Amazon.com that allows fast full-text searching on entire books. Udi Manber, a computer scientist who helped created many early versions of grep is now the chief scientist there. A9 apparently is a pun for "A followed by 9 characters", or "algorithm". egrep is what we will use from now on.
- Anchors, character sets, and repetitions/modifiers are the three main parts to a egrep command.
- There are two types of anchors: "^" anchors the pattern to the beginning of the line; "$" anchors it to the end of the line.
- Character sets are precisely that: ABC matches only the three-character string ABC. "." matches any character. To literally match a period, escape it: "\.". [ABC] matches only one character, either an A or a B or a C. [A-Z] does what you think it does, matches any character from A to Z. But be carefull that the LC_ALL flag is set to "C" (do a man egrep to see why you need to do this; the way to do it in the C-shell is "setenv LC_ALL C", in bash shell it is "export LC_ALL=C".). Otherwise [A-Z] will match both lowercase and uppercase characters. [^ABC] negates the meaning, i.e., match only if it is NOT one of these three characters.
- The standard modifiers are: "*" (meaning zero or more occurrences of the previous pattern), "+" (one or more occurrences), "?" (zero or one, i.e., an optional occurrence). {3} means find three occurrences of the previous pattern. {3,8} means find at least three and upto 8 occurrences of the previous pattern.
- Other constructs you might find useful are "|": for optional parts; and "()": to group patterns (to build more complex patterns).
- The -n option to egrep also spits out line numbers, in addition to matching lines.

Egrep examples

Assume you are given the file sample.txt. You can do cat -n sample.txt (so that you can see the line numbers). Let's do some pattern matching now.

Find all lines referencing the supreme court.
```
egrep -n "[Ss]upreme court" sample.txt
```
Find all lines containing references to names. A reference begins with a "Mr.", "Mrs.", "Ms.", or "Dr.", followed by a space, followed by a word.
```
egrep -n "(Mr\.|Mrs\.|Ms\.|Dr\.) [A-Za-z]+" sample.txt
```
Find all lines containing references to names at the beginning of the line.
```
egrep -n "^(Mr\.|Mrs\.|Ms\.|Dr\.) [A-Za-z]+" sample.txt
```
Find all lines containing references to real people. A "real people" name begins with a "Mr.", "Mrs.", "Ms.", or "Dr.", followed by a space, followed by a first name, a mandatory space, an optional middle initial (in which case it will be followed by a period and a space, i.e., ". "), and a last name.
```
egrep -n "(Mr\.|Mrs\.|Ms\.|Dr\.) [A-Za-z]+ ([A-Za-z]\. )?[A-Za-z]+" sample.txt
```

Same thing, but search through all of the files in the current directory:

egrep -n "(Mr\.|Mrs\.|Ms\.|Dr\.) [A-Za-z]+ ([A-Za-z]\. )?[A-Za-z]+" ./*

Recursive: use -R
Find all lines containing social security numbers.
```
egrep -n "[0-9]{3}-[0-9]{2}-[0-9]{4}" sample.txt
```