Oct 3

More sed commands

For this lecture, consider the following file called faculty.details:

Name: Lenwood S. Heath Office: 2160J Torgersen Hall Course: CS 4124
Name: Chris North Office: 634 McBryde Hall Course: CS 5764
Name: Godmar Back Office: 2160A Torgersen Hall Course: CS 5204
Name: Manuel Perez Office: 638 McBryde Hall Course: CS 1706
Name: Naren Ramakrishnan Office: 2160L Torgersen Hall Course: CS 2204
Name: Dennis Kafura Office: 660 McBryde Hall Course: CS 5204
Name: Liqing Zhang Office: 2160K Torgersen Hall Course: CS 5045

A tale of two buffers

Normally, sed reads one line at a time into its main buffer (sometimes called "pattern buffer"). There is another buffer available for our use, called the "hold buffer". Some commands to work with this buffer include:
- h copies the contents of the main buffer into the hold buffer, thus overwriting whatever it was that was already in the hold buffer.
- g copies the contents of the hold buffer into the main buffer, overwriting it.
- H does the same as h, except it appends the contents of the main buffer after the last line in the hold buffer.
- G does the same as g, again in the "append" sense.
- x exchanges contents of the two buffers. What was in hold is now in the pattern buffer, and vice versa.
- N reads in an additional line and appends it to the contents of the pattern buffer. In between the original line and the newly added line, N will insert a newline ("\n") character. Useful for reading in multiple lines at a time (see flip example below).

Worked out problems

Write sed commands/scripts to (put the solutions in a separate file, and invoke using sed -n -f option):

The department is moving. Move faculty in Torgersen Hall to McBryde Hall and move those in McBryde Hall to CRC. Let them keep their old office numbers, because they believe their numbers are lucky.
```
s/McBryde Hall/CRC/
s/Torgersen Hall/McBryde Hall/
```
Notice that you have to do the transformations in this order, else everybody gets assigned to CRC! Recall that sed reads in one line at a time, applies ALL the commands in the sequentially, then picks the next line, and so on. This is in contrast to reading all lines at once, applying the first command, then reading all lines again, applying the second command, and so on.

Pretty print the file so that each line has one line before it describing what it is about, e.g., "The next line is about Lenwood S. Heath" before the first line.

{
     h  # hold buffer now contains what was matched
     s/Name: \(.*\) Office: .* Course: .*/The next line is about: \1/
     G # appends hold buffer to pattern buffer
     p
}

Completely capitalize the names of faculty.

{
      h # save the current line in hold buffer
      s/Name: \(.*\) Office: .* Course: .*/\1/
      y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/
      G # current buffer contains a capital name, newline, old line
      s/\(.*\)\nName: \(.*\) Office: \(.*\)/Name: \1 Office: \3/
      p
}

Flip alternate lines.

$p
{
     N # read the next line, we now have two lines
     s/\(.*\)\n\(.*\)/\2\n\1/ # flip the two lines
     p # print it
}

Delete all the blank lines.
```
/^$/ {
    d
    b
}
p
```
Replace multiple blank lines wherever they occur with just one blank line.
```
/^$/ {
    N
    /^\n$/D
}
p
```
Notice that this uses a new command, namely D. D is just like d, it deletes the contents of the pattern (main) buffer. However, while d deletes the entire buffer, D deletes only till the first embedded newline.

Move the faculty lines around so that people in Torgersen Hall are listed on consecutive lines, after all the people in McBryde Hall.

This is a bit tricky. We can reason in the following way: At any point in time, let us have the hold buffer have the lines (those read so far) in the correct order. So, all these lines will have the McBryde Hall lines before the Torgersen Hall lines. Then we read in the next line. If the line contains 'McBryde Hall', we add it in the beginning of the buffer. If the line contains 'Torgersen Hall', we add it at the end of the buffer. So far so good. But how do we stop this process? We have to somehow print the lines at the end of the day. Here's a solution using branches and yet another command called t. t is like b, in that it is useful for branching. But while b branches unconditionally, t branches only if there has been a previously successful substitution on the line that was read. See how we use it below (it is only long because of the comments):

1{         # the story begins with the first line
	 h # save that line in the buffer, because we dont care
	   # if it has Torgersen Hall or McBryde Hall, 
           # the iteration starts here!
         d # delete the line from the current buffer
	 b # branch out, i.e., stop doing any more tests, 
           # and goto end of commands
}
${         # we have just read the last line
           # so far the hold buffer contains what we want. 
	   # the question is whether we should put this last 
           # line at the beginning or end.
	s/\(.*\) Torgersen Hall \(.*\)/\1 Torgy Hall \2/ 
	   # substitute Torgersen Hall for Torgy Hall. This 
	   # is an indirect way of seeing if there was a Torgersen 
           # Hall in the line.
        t777 # if there was a substitution, branch to line numbered 777
	G  # there was no substitution, that is why we are here.
           # so that means this line contains McBryde Hall.
           # we use a G to append the hold buffer after the current line.
        p  # print it...
	b  # and exit!
:777    s/\(.*\) Torgy Hall \(.*\)/\1 Torgersen Hall \2/
	   # we reach the 777 line because there was a Torgersen Hall.
           # replace the Torgy with Torgersen.
	H  # do a H, instead of a G, because we want the Torgersen Halls
           # to goto the end of the lines.
        x  # exchange the hold space and the current space
        p  # print the current space!
}
/Torgersen Hall/{
	 H # append this line to hold space, because we want the Torgersen Halls
           # to goto the end of the lines.
         d # delete the current line.
}
/McBryde Hall/{
	 G # append hold space to given line, because we want the McBryde Halls
           # to come at the beginning.
	 h # move it to hold, for safe storage.
         d # and delete.
}

When we apply this script on the previous file, we get:

Name: Dennis Kafura Office: 660 McBryde Hall Course: CS 5204
Name: Manuel Perez Office: 638 McBryde Hall Course: CS 1706
Name: Chris North Office: 634 McBryde Hall Course: CS 5764
Name: Lenwood S. Heath Office: 2160J Torgersen Hall Course: CS 4124
Name: Godmar Back Office: 2160A Torgersen Hall Course: CS 5204
Name: Naren Ramakrishnan Office: 2160L Torgersen Hall Course: CS 2204
Name: Liqing Zhang Office: 2160K Torgersen Hall Course: CS 5045

Note that the 'McBryde Hall's are printed in the reverse order in which they are encountered, because we keep adding them in the front. Whereas the 'Torgersen Hall's are printed in the order in which we encounter them, because we add them at the end.

awk: a more powerful sed

Didn't the last sed command get ugly? This is a clue that we must move to a more powerful tool, like awk.
Named after inventors, Aho, Weinberger, and Kernighan.
Follows sed style, but uses C syntax to specify commands.
Powerful for table manipulation and data summarization.
We will actually use gawk, which is basically awk with more thrills and frills.

Consider the following data file (student.grades) for this part of the lecture.

John    45      55      60      90
Mary    70      75      88      100
Jimmy   75      80      85      100
Kathy   80      70      70      95

Basic gawking

The following gawk script just cats a file (run it as you would run sed: gawk -f <gawk script name>):
```
{ 
 print
}
```
Note that the curly braces contain commands, just as in sed. Since there is nothing before {, it means that these commands must be applied to all lines. The only difference is that instead of p in sed, we have print.
gawk has two special patterns, BEGIN and END where you can put commands that must be done before any line is read, and after all lines are read (respectively). Here's an example:
```
BEGIN {
  print "I am going to start reading a file. Whoopie!"
} 
{
  print
}
END {
  print "I have finished reading the file. Sigh."
}
```
When gawk reads a line, it automatically parses the line and puts pieces of the line into defined variables such as $1 (first field), $2 (second field), etc. The default field separator is a tab. So, the gawk script
```
{
  print $1
}
```
will just print the names.
Can also create variables and tinker with them, just like we would in a C program. Here's how you will calculate the average value of scores in the first column of numbers (which is actually the second column of the file).
```
BEGIN {
  total = 0
  lc = 0
}
{
  total = total + $2
  ++lc
}
END {
  avg = total/lc
  print total, avg
}
```
Can tinker with some gawk system variables to modify the output, e.g., OFS stands for "output field separator". We can set it in the BEGIN part by:
```
BEGIN {
  total = 0
  lc = 0
  OFS = "---"
}
```
This will affect all subsequent printings done using the print command. In between two variables (listed in comma separated format), gawk will insert the output field separator. Similarly, there is a FS which is an input field separator tag. This will tell gawk to use some other input field separator than the default tab.
It is good practice to put one gawk command on each line. If you put multiple commands, you will need to use a ";" to separate them.