Today (2006 January 04 20:57) I finished a small sed project: converting the 5000-line body of an html page to a form that was much more useful (to me). It was a good excuse to put sed through its paces, and to finally learn about its hidden nooks and crannies. The main feature I missed: the ability to set variables and to copy their values into RHSs of s (substitution) commands. As it was, I got to practice my Forth programming: moving strings between the pattern (stack) and hold (return stack) spaces. ;-)

Unfortunately the FreeBSD man page is an awful tutorial (and a mediocre reference). After getting quite frustrated with it I searched and found some more useful sources. The FAQ was a good start. But the gold mine I found here: the original manual that the original sed author – Lee McMahon – wrote for the original version. (Here is the original troff source.)

As is often the case, the first telling of the story is clearer, crisper, and gentler than all its successors.

I have edited out the page headers and footers (which were printed inline), and have removed (and added) a few blank lines, for clarity.

Ladies and gentlemen, please give a warm welcome to...

               SED -- A Non-interactive Text Editor
                          Lee E. McMahon
                      AT&T Bell Laboratories
                   Murray Hill, New Jersey 07974
           Sed  is a non-interactive context editor that runs
      on the UNIX operating system.  Sed is  designed  to  be
      especially useful in three cases:
           1)  To edit files too large for comfortable inter-
                active editing;
           2) To edit any size  file  when  the  sequence  of
                editing  commands  is  too  complicated to be
                comfortably typed in interactive mode.
           3) To perform multiple `global' editing  functions
                efficiently in one pass through the input.

I’ve moved the text of the sed manual elsewhere, to give more room on this page for discussion.

It’s amazing what you can do with just the basic tools: ed, sed, lex, yacc, awk, cat, dd, rm, ls, mv, sh, cp, & grep.

Neddy Seagoon

Why was it necessary to use sed? Personally, I find awk more user friendly – and it works for most of the things I need. And I can remember all (well, most) of its builtins. --Michael Pruemm

The way I had to tear apart and rebuild the text seemed much harder in awk. I wasn’t breaking things up by spaces; instead I relied entirely on REs, and I didn’t see a clean way to do this in awk. I didn’t want to use substr and friends: REs were a much better match.

Am I mistaken here?

Awk has match, sub and gsub functions that take regular expressions. Also, split and the field and record separators can be regular expressions. So I would say, yes, you can do all that in awk and it is probably more readable.

Note that this applies to what is sometimes called new awk as described in the book The AWK Programming language by Aho, Weinberger and Kernighan. Some (older) systems have awk and nawk executables where the latter supports the new features described in the book. The programs gawk and mawk always did, as does the version that is available from the book page, the one true awk.

A small example of what you wanted to do might be helpful for further discussion.

Well, I had an HTML file full of references (inside a table) of the form:

 <td width="60">1/4/05</td>
 <td width="120">This is a title</td>
 <td width="480">This is a long description, probably several lines long,  
     describing the object</td>

There were several hundred of these triples, and a bunch of other junk. The first thing I did was to throw away everything that didn’t match one of the three lines. Then I canonicalized the date formats – they needed to be mm/dd/yyyy so I could later generate link names of the form yyyymmdd-special.

Then, for each triple I generated (roughly, given the above “example”):

 <a href="">This is a title</a></h2>
 <p>This is a long ... object</p>

Not complicated, mind, and quite easy to do in sed, but only once I had concatenated the three things into one long line (using h and H) so I could match all three parts in one long RE to build the final HTML.