sed (stream editor) is a Unix utility that parses and transforms text.
sed reads text input, line by line, either from a file or a stream, into an internal buffer called the pattern space. Each line read starts a cycle. To the pattern space, sed applies one or more operations which have been specified via a sed script. sed implements a programming language with about 25 commands that specify the operations on the text. When the sed script ends, sed outputs the pattern space, and reads the next line into the pattern space, starting a new cycle.
The sed script can either be specified on the command line or read from a separate file. Commands in the sed script may also take an optional address. The address determines when the command is run, for example '2d' would only run the d (delete) command on the second input line. Some sed operations can use regular expressions to add flexibility. And a separate special buffer, the hold space, may be used by a few sed commands to hold and accumulate text between cycles.
sed was developed from 1973 to 1974 as a Unix utility by Lee E. McMahon of Bell Labs,[1] and is available today for most operating systems.[2]
History
sed is one of the very early Unix commands built for command line processing of data files. It evolved as the natural successor to the popular grep command.[3] Cousin to the later AWK, sed allows powerful and interesting data processing to be done by shell scripts.
sed and AWK are often cited as the progenitors and inspiration for Perl. The s / / / syntax shown below is part of Perl's syntax and originated with ed, the precursor to sed.
sed's language does not have variables and has only primitive GOTO and branching functionality; nevertheless, the language is Turing-complete.[3] [4]
GNU sed added several new features. The best-known is in-place editing of files (i.e., replace the original file with the result of applying the sed program), which was later included in BSD sed too. This feature is nowadays often used instead of ed scripts: for example,
sed -i 's/abc/def/' file # (the BSDs require an extension with the -i flag, e.g. sed -i .bak 's/abc/def/' file)
Note: "sed -i" overwrites the original file with a new one, breaking any links the original may have had, while the above example using "ed" changes only the original file's contents, preserving file links. Note that if using -i'extension' the original file will be preserved by renaming it with the given extension. Note also that a space cannot be used between "-i" and extension since the extension is optional.
Super-sed is an extended version of sed that includes regular expressions compatible with Perl.
Another variant of sed is minised, originally reverse-engineered from the 4.1BSD sed by Eric S. Raymond and currently maintained by René Rebe. minised was used by the GNU Project until the GNU Project wrote a new version of sed based on the new GNU regular expression library. The current minised contains some extensions to BSD sed but is not as feature-rich as GNU sed. Its advantage is that it is very fast and uses little memory.[citation needed] It is used on embedded systems and is the version of sed provided with Minix.[citation needed]
Usage
The following example shows a typical, and the most common, use of sed, where the -e option indicates that the sed expression follows:
sed -e 's/oldstuff/newstuff/g' inputFileName > outputFileName
In many versions, the -e is not required to precede the expression. The s stands for substitute. The g stands for global, which means that all matching occurrences in the line would be replaced. The regular expression (i.e. pattern) to be searched is placed after the first delimiting symbol (slash here) and the replacement follows the second symbol. Slash (/) is the conventional symbol. Any other could be used to make syntax more readable if it does not occur in the pattern or replacement (see below), which is useful to avoid leaning toothpick syndrome.
Under Unix, sed is often used as a filter in a pipeline:
generate_data | sed 's/x/y/g'
That is, generate the data, and then make the small change of replacing x with y.
Several substitutions or other commands can be put together in a file called, for example, subst.sed and then be applied using the -f option to read the commands (such as s/x/y/g) from the file:
sed -f subst.sed inputFileName > outputFileName
Besides substitution, other forms of simple processing are possible. For example, the following uses the d command to delete lines that are either blank or only contain spaces:
sed '/^ *$/d' inputFileName
This example used some of the following regular expression metacharacters (sed supports the full range of regular expressions):
- The caret (
^) matches the beginning of the line. - The dollar sign (
$) matches the end of the line. - The asterisk (
*) matches zero or more occurrences of the previous character.
Complex sed constructs are possible, allowing it to serve as a simple, but highly specialised, programming language. Flow of control, for example, can be managed by the use of a label (a colon followed by a string) and the branch instruction b. An instruction b followed by a valid label name will move processing to the block following that label.
Samples
To delete any line containing the word "yourword" from the file "yourfile" (the address is '/yourword/'):
sed '/yourword/ d' yourfile
To delete all instances of the word "yourword":
sed 's/yourword//g' yourfile
To delete two words from a file simultaneously:
sed -e 's/firstword//g' -e 's/secondword//g' yourfile
or
sed 's/firstword//g; s/secondword//g' yourfile
In the next example, sed, which usually only works on one line, removes newlines from sentences where the second sentence starts with one space. Consider the following text:
This is my cat, whose name is Betty.This is my dog, whose name is Frank.This is my fish,whose name is George.This is my goat, whose name is Adam.
The sed script below will turn the text above into the following text. Note that the script affects only the input lines that start with a space:
This is my cat, whose name is Betty.This is my dog, whose name is Frank.This is my fish,whose name is George.This is my goat, whose name is Adam.
Here's the script:
sed 'N;s/\n / /;P;D;'
- (N) add the next line to the work buffer
- (s) substitute
- (/\n /) match a newline character and a " ": find a new line followed by a space
- (/ /) replace with: one space
- (P) print the top line of the work buffer
- (D) delete the top line from the work buffer and run the script again
Less portable but more complex sed usage is transposing a file containing an XML table into a CSV file:
sed -rn '{s/ *//g;/<field *\/>/{s/.*//g;H};/<field>/{s /<\/*field>//g;H};/<\/row> ;/{x;s/^\r*\n//;s/\r*\n\r*/","/g;s/^( [^\r\n]*)/"\1"/;p};/<row/{s/.*//g; x;}}' yourfile.xmlFirst clear out all the extraneous whitespace:
s/ *//g
If the current line is a blank <field> line, add a blank line to the hold space:
/<field *\/>/{s/.*//g;H}If there is an actual value in the field, strip the xml and add the value to the hold space:
/<field>/{s/<\/*field>//g ;H}If it is the end of a row (</row>), then get the hold space, replace the newlines with '","', add quotes to the beginning and end and then print the line:
/<\/row>/{x;s/^\r*\n//;s/\r*\n\ r*/","/g;s/^([^\r\n]*)/"\1"/;p}If it is the beginning of a row, clear the hold space by adding a blank line:
/<row/{s/.*//g;x;}Can anyone verify syntax in the above "XML table into a CSV file" example is correct? Can anyone provide sample "XML table" input, and resulting "CSV file" output? If cannot be verified, maybe that complex example (and this paragraph) should be removed?
Note: If an implementation of sed does not support the -r (extended regular expressions) option, try using the -E option instead, as in this example:
sed -E some_sed_commands [file ...]
Exotic examples
Despite the inherent limitations, sed scripts exist for games such as sokoban, arkanoid,[5] and an implementation of tetris.[6]
See also
References
Further reading
External links
Tutorials
Examples
