BBS水木清华站∶精华区

发信人: cybergene (基因～也许以后～～), 信区: Linux
标 题: New Regular Expression Features in Tcl 8.1
发信站: BBS 水木清华站 (Thu Dec 14 15:56:19 2000)

New Regular Expression Features in Tcl 8.1

TclPro Extensions | Wrap TclPro | Compile Tcl | Stub Libraries | Threads
| Windows Extensions | Regular Expressions | I18N

Tcl 8.1 now handles advanced regular expressions (REs). Previous regular
expression handling is almost unchanged except that clumsy handling
of escapes like \n has been much improved, and a few escapes that were
previously legal (but useless) now won't work.

Note that a few advanced features aren't useful yet but are ready for
future Tcl releases. That's because Tcl 8.1 (apart from the regular
expression engine) implements only the Unicode locale (where all
characters sort in Unicode order, there are no multi-character collating
elements and no equivalence classes).

This document has an overview of the new regular expression features.
For exact semantics and more details, see the new re_syntax(n) reference
page. (The re_syntax(n) page was split from the 8.1 regexp(n) reference
page, which used to cover RE syntax for all Tcl commands.) This howto
document covers:

1. Regular Expression Overview

What are Regular Expressions?
Regular Expressions in Tcl 8.0 and Before
Overview of regexp and regsub
Backslash Processing
2. Regular Expressions in Tcl 8.1

Non-Greedy Quantifiers
Backslash Escapes (\xxx)
Bounds ({})
Character Classes ([: :])
Collating Elements ([. .])
Equivalence Classes ([= =])
Noncapturing Subpatterns ((?:re))
Lookahead Assertions ((?=re) and (?!re))
Switches
Options ((?xyz)), Directors (***)
3. Summary: Regular Expression changes in Tcl 8.1

Part 1. Regular Expression Overview
This Part describes regular expressions (REs), explains REs from Tcl 8.0
and before, and describes the Tcl regexp and regsub commands. Part
Two describes the new Tcl 8.1 REs.

What are Regular Expressions?
A regular expression, or RE, describes strings of characters (words or
phrases or any arbitrary text). It's a pattern that matches certain
strings and doesn't match others. For example, you could write an RE
to tell you if a string contains a URL (World Wide Web Uniform
Resource Locator, such as http://somehost/somefile.html). Regular
expressions can be either broad and general or focused and precise.
A regular expression uses metacharacters (characters that assume special
meaning for matching other characters) such as *, [], $ and .. For
example, the RE [Hh]ello!* would match Hello and hello and Hello! (and
hello!!!!!). The RE [Hh](ello|i)!* would match Hello and Hi and Hi! (and
so on). A backslash (\) disables the special meaning of the following
character, so you could match the string [Hello] with the RE \[Hello\].


Regular Expressions in Tcl 8.0 and Before
Regular expressions in Tcl 8.0 and before had the following
metacharacters: . Match any single character (e.g., m.d matches mad,
mod, m3d, etc.)
[] Bracket expression: Match any one of the enclosed characters (e.g.,
[a-z0-9_] matches a lowercase ASCII letter, a digit, or an underscore)
^ Start-of-string anchor: Match only at the start of a string (e.g., ^hi
matches hi and his but not this)
$ End-of-string anchor: Match only at the end of a string (e.g., hi$
matches hi and chi but not this)
* Zero-or-more quantifier: makes the previous part of the RE match
zero or more times (e.g., M.*D matches MD, MAD, MooD, M.D, etc.)
? Zero-or-one quantifier: makes the previous part of the RE match zero
or one time (e.g., hi!? matches hi or hi!)
+ One-or-more quantifier: makes the previous part of the RE match one or
more times (e.g., hi!+ matches hi! or hi!! or hi!!! or ...)
| Alternation (vertical bar): Match just one alternative (e.g.,
this|that matches this or that)
() Subpattern: Group part of the RE. Many uses, such as:
Makes a quantifier apply to a group of text (e.g., ([0-9A-F][0-9A-F])+
matches groups of two hexadecimal digits: A9 or AB03 or 8A6E00, but
not A or A2C).
Set limits for alternation (e.g., "Eat (this|that)!" matches "Eat this!"
or "Eat that!").
Used for subpattern matching in the regexp and regsub commands.

\ Escape: Disables meaning of the following metacharacter (e.g., a\.*
matches a or a. or a.. or etc.). Note that \ also has special meaning to
the Tcl interpreter (and to applications, such as C compilers).

The syntax above is supported in Tcl 8.1. Tcl 8.1 also supports advanced
regular expressions (AREs). These powerful expressions are introduced
in more detail in Part Two. Briefly, though, AREs support
backreferences, lookahead, non-greedy matching, many escapes, features
that are useful for internationalization (handling collation elements,
equivalence classes and character classes), and much more.

The Tcl 8.1 regular expression engine almost always interprets 8.0-style
REs correctly. In the few cases that it doesn't, and when the problem
is too difficult to fix, the 8.1 engine has an option to select 8.0
("ERE") interpretation.

Overview of regexp and regsub
The Tcl commands regexp and regsub use regular expressions:
regexp compares a string to an RE. It returns a value of 1 if the RE
matches part or all of the string or 0 if there's no match. Optionally,
it stores the matched part of the string in a variable (and also can
store subparts of a string in multiple variables). For example, to
compare the string in $line against the RE [Hh]ello!*, you would write:

regexp {[Hh]ello!*} $line match
If part or all of the line variable matches the RE, regexp stores the
matching part in the match variable and returns a value of 1.

regsub substitutes part of a string that matches an RE. For instance,
the following command edits the string from $in_line to replace all
space or tab characters with a single space character; the edited line
is stored in the out_line variable:
regsub -all {[ \t]+} $in_line { } out_line
Please also read the following section about backslash processing.

Backslash Processing
If you've used Tcl, you probably recognize the \t in the previous
example as a character-entry escape that stands for a tab character.
We actually used the 8.1 syntax above; the example wouldn't have
worked under 8.0! In Tcl 8.0 and before, you had to surround the regular
expression with double quotes so the Tcl backslash processor could
convert the \t to a literal tab character. The square brackets had to be
hidden from the backslash processor by adding backslashes before them,
which made code harder to read and possibly more error-prone. Here's
the previous example rewritten for Tcl 8.0 and before:

regsub -all "\[ \t\]+" $in_line { } out_line
For more about the simplified 8.1 syntax, see the section Backslash
Escapes.

Part 2. Regular Expressions in Tcl 8.1
Tcl 8.1 regular expressions are basically a superset of 8.0 REs. This
howto document has an overview of the new features. Please see the
re_syntax(n) reference page for exact semantics and more details.

Non-Greedy Quantifiers
A quantifier specifies "how many." For example, the quantifier * in
the RE z* matches zero or more zs. By default, regular expression
quantifiers are greedy: they match as much text as they can. Tcl 8.1 REs
also have non-greedy quantifiers, which match the least text they can.
To make a non-greedy quantifier, add a question mark (?) at the end.
Let's start by storing some HTML text in a variable, then using two
regexp commands to match it. The first RE is greedy, and the second is
non-greedy:

% set x {He sits, but she stands.}
He sits, but she stands.
% regexp {.*} $x match; set match
He sits, but she
% regexp {.*?} $x match; set match
He

The first RE .* is "greedy." It matches from the first 
to the last . The second RE .*?, with a question mark
(?) after the * quantifier, is non-greedy: it matches as little text
as possible after the first . Could you write a greedy RE that works
like the non-greedy version? It isn't easy! A greedy RE like
[^<]* would do it in this case -- but it wouldn't work if there
were other HTML tags (with a < character) between the pair of tags
in the $x string.
Here are a new string and another pair of REs to match it:

% set y {123zzz456}
123zzz456
% regexp {3z*} $y match; set match
3zzz
% regexp {3z*?} $y match; set match
3

The greedy RE 3z* matches all the zs it can (three) under its "zero or
more" rule. The non-greedy RE 3z*? matches just 3 because it matches the
fewest zs it can under its "zero or more" rule.
To review, the greedy quantifiers from Tcl 8.0 are: *, +, and ?. So
the non-greedy quantifiers (added in Tcl 8.1) are: *?, +?, and ??. Tcl
8.1 also has the new quantifiers {m}, {m,}, and {m,n}, as well as the
non-greedy versions {m}?, {m,}?, and {m,n}?. The section on bounds
explains -- and has more examples of non-greedy matching.

Backslash Escapes
A backslash (\) disables the metacharacter after it. For example, a\*
matches the character a followed by a literal asterisk (*) character. In
Tcl 8.0 and before, it was legal to put a backslash before a
non-metacharacter -- for instance, regexp {\p} matched the character p.
(Note that regexp {\n} matched the character n, which was a source of
confusion. To get a newline character into an RE before version 8.1, you
had to write regexp "\n" so Tcl processing inside double quotes would
convert the \n to a newline.)
The Tcl 8.1 regular expression engine interprets backslash escapes
itself. So now regexp {\n} matches a newline, not the character n. REs
are simpler to write in 8.1 because of this. (You can still write regexp
"\n" -- and let Tcl conversion happen inside the double quotes -- so
most old code will still work.)

One of the most important changes in 8.1 is that a backslash inside a
bracket expression is treated as the start of an escape. In 8.0 and
before, a backslash inside brackets was treated as a literal backslash
character. For example, in 8.0 and before, regexp {[a\n]} would match
the characters a, \, or n. But in 8.1, regexp {[a\n]} would match the
characters a or newline (because \n is the backslash escape for
"newline").

Tcl 8.1 has also added many new backslash escapes. For instance, \d
matches a digit. Some of these are listed below, and the re_syntax(n)
reference page has the whole list.

In Tcl 8.1 regular expressions (but not in other parts of the language),
it's illegal to use a backslash before a non-metacharacter unless it
makes a valid escape. So regexp {\p} is now an error. If you have code
that (for some bizarre reason) has regular expressions with a
backslash before a non-metacharacter, like regexp {\p}, you'll need to
fix it.

As explained above, the Tcl 8.1 regular expression engine now interprets
backslash sequences like \n to mean "newline". It also has four new
kinds of escapes: character entry escapes, class shorthand escapes,
constraint escapes, and back references. Here's an introduction. (The
re_syntax(n) page has full details.)

A character entry escape is a convenient way to enter a non-printing
or other difficult character. For instance, \n represents a newline
character. \uwxyz (where wxyz is hexadecimal) represents the Unicode
character U+wxyz.
Class shorthand escapes are shorthand for common character classes.
For example, \d stands for [[:digit:]], which means "any single digit.
"
A constraint escape constrains an RE to match only at a certain place.
For example, the constraint escape \m matches only at the start of a
word -- so the RE \mhi will match the third word in the string he said
hi but won't match he said thigh.
A back reference matches the same string that was matched by a
previous parenthesized subexpression. (This works like subexpressions in
regsub, but it's used for matching instead of extracting.) For example,
(X.*Y)\1 matches any doubled string that starts with X and ends with Y,
such as XYXY, XabcYXabcY, X--YX--Y, etc.
Finally, remember that (as in Tcl 8.0 and before) some applications,
such as C compilers, interpret these backslash sequences themselves
before the regular expression engine sees them. You may need to double
(or quadruple, etc.) the number of backslashes for these applications.
Still, in straight Tcl 8.1 code, writing backslash escapes is now both
simpler and more powerful than in 8.0 and before.

Bounds
You've seen the quantifiers *, +, and ?. They specify "how many"
(respectively, zero or more, one or more, and zero or one). Tcl 8.1
added new quantifiers that let you choose exactly how many matches:
the bounds operators, {}.
These operators come in three greedy forms: {m}, {m,}, and {m,n}. The
corresponding non-greedy forms are {m}?, {m,}?, and {m,n}?.

The {m} quantifier matches exactly m occurrences. So does {m}?. For
example, either #{70} or #{70}? match a string of exactly 70 #
characters.
The {m,} quantifier matches at least m occurrences. Here's a demo of the
greedy and non-greedy versions:

% set x {a##b#######c}
a##b#######c
% regexp {#{4,}} $x match; set match
#######
% regexp {#{4,}?} $x match; set match
####

Notice that the first two number signs (##) in the string are never
matched because there aren't at least four of them.
The {m,n} quantifier matches at least m but no more than n occurrences.

For example, the RE http://([^/]+/?){1,3} would match Web URLs that have
3 components (like http://xyz.fr/euro/billets.htm), or with 2
components (like http://xyz.fr/euro/, or with just 1 component (like
http://xyz.fr). The RE matches a final slash (/) if there is one. As
always, a greedy match will match as long a string as possible: it would
try for 3 matches.

A non-greedy quantifier would try to match the least (1 match). But be
careful: http://([^/]+/?){1,3}? won't match all the way to a possible
slash because it matches the fewest characters possible! (With input
http://xyz.fr/, that RE would match just http://x.) This brings up one
of the many subtleties in these advanced regular expressions: that the
outer non-greedy quantifier overrides the inner greedy quantifiers and
makes all quantifiers non-greedy! There's an explanation in re_syntax(n)
reference page section named Matching.

Character Classes
A character class is a name for one or more characters. For example,
punct stands for the "punctuation" characters. A character class is
always written as part of a bracket expression, which is a list of
characters enclosed in [].
For instance, the character class named digit stands for any of the
digits 0-9 (zero through nine). The character class is written with
the class name inside a set of brackets and colons, like this: [[:
digit:]]. The old familiar expression for digits is written as a range:
[0-9]. When you compare the new character class to the old range
version, you can see that the outer square brackets are the same in
both. So a character class is written [:classname:].

The table below describes the Tcl 8.1 character classes. alpha A
letter (includes many non-ASCII characters).
upper An upper-case letter.
lower A lower-case letter.
digit A decimal digit.
xdigit A hexadecimal digit.
alnum An alphanumeric (letter or digit).
print An alphanumeric. (Same as alnum.)
blank A space or tab character.
space A character producing white space in displayed text. (Includes
en-space, hair space, many others.)
punct A punctuation character.
graph A character with a visible representation.
cntrl A control character.

You can use more than one character class in a bracket expression. You
can also mix character classes with ranges and single characters. For
instance, [[:digit:]a-cx-z] would match a digit (0-9), a, b, c, x, y, or
z -- and [^[:digit:]a-cx-z] would match any character except those.
This syntax can take some time to get familiar with! The key is to
look for the character class (here, [:digit:]) inside the bracket
expression.

The advantage of character classes (like [:alpha:]) over explicit ranges
in brackets (like [a-z]) is that character classes include characters
that aren't easy to type on ASCII keyboards. For example, the Spanish
language includes the character ?. It doesn't fall into the range [a-z],
but it is in the Tcl 8.1 character class [:alpha:]. In the same way,
the Spanish punctuation character ? isn't in a list of punctuation
characters like [.!?,], but it is part of [:punct:].

Tcl 8.1 has a standard set of character classes that are defined in
the source code file generic/regc_locale.c. Tcl 8.1 has one locale
defined: the Unicode locale. It may support other locales (and other
character classes) in the future.

Collating Elements
A collating symbol lets you represent other characters unambiguously.
A collating symbol is written surrounded by brackets and dots, like [.
number-sign.] Collating symbols must be written in a bracket
expression (inside []). So [[.number-sign.]] will match the character #,
as you can see here:

% regexp {[[.number-sign.]]+} {123###456} match
1
% set match
###

Tcl 8.1 has a standard set of collating symbols that are defined in
the source code file generic/regc_locale.c. Note: Tcl 8.1 does not
implement multi-character collating elements like ch (which is the
fourth character in the Spanish alphabet a, b, c, ch, d, e, f, g, h, i.
..) So the examples below are not supported in Tcl 8.1, but are here for
completeness. (Future versions of Tcl may have multi-character
collating elements.)
Suppose ch and c sort next to each other in your dialect, and ch is
treated as an atomic character. The example bracket expression below
uses two collating symbols. It matches one or more of ch and c. But it
doesn't match an h standing alone:

% set input "cchchh"
cchchh
% regexp {[[.ch.][.c.]]+} $input match; set match
cchch

Here's one tricky and surprising thing about collating symbols. A
caret at the start of a bracket expression ([^...) means that, in a
locale with multi-character collating elements, the symbol can match
more than one character. For instance, the RE in the example below
matches any character other than c, followed by the character b. So
the expression matches all of chb:

% set input chb
% regexp {[^[.c.]]b} $input match; set match
chb

Again, the two previous examples are not supported in Tcl 8.1, but are
here for completeness.
Equivalence Classes
An equivalence class is written as part of a bracket expression, like
[[=c=]]. It's any collating element that has the same relative order
in the collating sequence as c.
Note: Tcl 8.1 only implements the Unicode locale. It doesn't define
any equivalence classes. So, although the Tcl regular expression
engine supports equivalence classes, the examples below are not
supported in Tcl 8.1. (Future versions of Tcl may define equivalence
classes.)

Let's imagine that both of the characters A and a fall at the same place
in the collating sequence; they belong to the same equivalence class.
In that case, both of the bracket expressions [[=A=]b] and [[=a=]b]
are equivalent to writing [Aab]. As another example, if o and ? are
members of an equivalence class, then all of the bracket expressions
[[=o=]], [[=?=]], and [o?] match those same two characters.

Noncapturing Subpatterns
There are two reasons to put parentheses around all or part of an RE.
One is to make a quantifier (like * or +) apply to the parenthesized
part. For instance, the RE Oh,( no!)+ would match Oh, no! as well as Oh,
no! no! and so on. The other reason to use parentheses is that they
capture the matched text. Captured text is used in back references, in
"matching" variables in the regexp command, as well as in the regsub
command.
If you don't want parentheses to capture text, add ?: after the
opening parenthesis. For instance, in the example below, the
subexpression (?:http|ftp) matches either http or ftp but doesn't
capture it. So the back reference \1 will hold the end of the URL
(from the second set of parentheses):

% set x http://www.ajubasolutions.com
http://www.ajubasolutions.com
% regsub {(?:http|ftp)://(.*)} $x {The hostname is \1} answer
1
% set answer
The hostname is www.ajubasolutions.com

Lookahead Assertions
There are times you'd like to be able to test for a pattern without
including that text in the match. For instance, you might want to
match the protocol in a URL (like http or ftp), but only if that URL
ends with .com. Or maybe you want to match the protocol only if the
URL does not end with .edu. In cases like those, you'd like to "look
ahead" and see how the URL ends. A lookahead assertion is handy here.
A positive lookahead has the form (?=re). It matches at any place
ahead where there's a substring like re. A negative lookahead has the
form (?!re). It matches at any point where the regular expression re
does not match. Let's see some examples:

% set x http://www.ajubasolutions.com
http://www.ajubasolutions.com
% regexp {^[^:]+(?=.*\.com$)} $x match
1
% set match
http
% regexp {^[^:]+(?=.*\.edu$)} $x match
0
% regexp {^[^:]*(?!.*\.edu$)} $x match
1
% set match
http

The regular expressions above may seem complicated, but they're really
not bad! Find the lookahead expression in the first regexp command
above; it starts with (?= and ends at the corresponding parenthesis. The
"guts" of this lookahead expression is .*\.com$, which stands for "a
string that ends with .com". So the first regexp command above matches
any string containing non-colon (:) characters, as long as the rest of
the string ends with .com. The second regexp is similar but looks for
a string ending with .edu. Because regexp returns 0, you can see that
this doesn't match. The third regexp looks for a string not ending
with .edu. It matches because $x ends with .com.
Tcl 8.1 lets you document complex regular expressions by embedding
comments. See the next section.

Switches
Tcl 8.1 added command switches to regexp and regsub. For a complete
list, see the commands' reference pages. Let's look at two of the most
important changes.
Complex REs can be difficult to document. The -expanded switch sets
expanded syntax, which lets you add comments within a regular
expression. Comments start with a # character; whitespace is ignored.
This is mostly for scripting -- but you can also use it on a command
line, as we'll do in the example below. Let's look the same RE twice:
first in the standard compact syntax, and second in expanded syntax:

% set x http://www.ajubasolutions.com
http://www.ajubasolutions.com
% regexp {^[^:]+(?=.*\.com$)} $x match
1
% set match
http
% regexp -expanded {
 ^ # beginning of string
 [^:]+ # all characters to the first colon
 (?= # begin positive lookahead
 .*\.com$ # for a trailing .com
 ) # end positive lookahead
} $x match
1
% set match
http

In expanded syntax, you can use space and tab characters to indent and
make your code clear. To enter actual space and tab characters into your
RE, use the escapes \s and \t, respectively.
The other important new switch we'll cover here is -line. It enables
newline-sensitive matching. By default (without -line), Tcl regular
expressions have always treated newlines as an ordinary character. For
example, if a string contains several lines (separated by newline
characters), the end-of-string anchor $ wouldn't match at any of the
embedded newlines. To write code that matched line-by-line, you had to
read input lines one by one and do separate matches against each line.

With the -line switch, the metacharacters ^, $, ., and [] treat a
newline as the end of a "line." So, for example, the regular
expression ^San Jose matches the second line of input below:

% set x {Dolores Sanchez
San Jose, CA}
Dolores Sanchez
San Jose, CA
% regexp {^San Jose} $x match
0
% regexp -line {^San Jose} $x match
1
% set match
San Jose

The -line switch actually enables two other switches. You can set part
of the features from -line by choosing one of these switches instead:
The -lineanchor switch makes ^ and $ match at the beginning and end of a
line. The -linestop switch makes . and [] stop matching at a newline
character.
Options, Directors
This section introduces two more features from Tcl 8.1. Details are in
the re_syntax(n) reference page.
An 8.1 RE can start with embedded options. These look like (?xyz), where
xyz are one or more option letters. For instance, (?i)ouch matches OUCH
because i is the "case-insensitive matching" option. Other options
include (?e), which marks the rest of the regular expression as an 8.
0-style RE -- to let you avoid confusion with the new 8.1 syntax.

An RE can also start with three asterisks, which is a director. For
example, ***= is the director that says the rest of the regular
expression is literal text. So the RE ***=(?i)ouch matches exactly
(?i)ouch; the (?i) isn't treated as an option.

Part 3. Summary: Regular Expression Changes in Tcl 8.1
Tcl 8.1 added advanced regular expression syntax. The new re_syntax(n)
reference page has details.
This table below summarizes the new syntax: {m} Matches m instances of
the previous pattern item
{m}? Matches m instances of the previous pattern item. Non-greedy.
{m,} Matches m or more instances of the previous pattern item.
{m,}? Matches m or more instances of the previous pattern item.
Non-greedy.
{m,n} Matches m through n instances of the previous pattern item.
{m,n}? Matches m through n instances of the previous pattern item.
Non-greedy.
*? Matches zero or more of the previous pattern item. Non-greedy.
+? Matches one or more of the previous pattern item. Non-greedy.
?? Matches zero or one of the previous pattern item. Non-greedy.
(?:re) Groups a subpattern, re, but does not capture the result.
(?=re) Positive lookahead. Matches the point where re begins.
(?!re) Negative lookahead. Matches any point where re does not begin.
\c One of many backslash escapes.
[. .] Delimits a collating element within a bracketed expression.
[= =] Delimits an equivalence class within a bracketed expression.
[: :] Delimits a character class within a bracketed expression.
(?abc) Embedded options a, b, and c
*** Director

Some of the new switches for regexp and regsub are: -expanded Enable
expanded syntax (for comments)
-line Enable newline-sensitive matching
-linestop Make [] and . stop at newlines.
-lineanchor Make ^ and $ match the start and end of a line.



--
 桃花坞里桃花庵，桃花庵下桃花仙；桃花仙人种桃树，又摘桃花卖酒钱。
 酒醒只在花前坐，酒醉换来花下眠；半醒半醉日复日，花落花开年复年。
 但愿老死花酒间，不愿鞠躬车马前；车尘马足富者趣，酒盏花枝贫者缘。
 若将富贵比贫贱，一在平地一在天；若将贫贱比车马，他得驱驰我得闲。
 别人笑我忒疯癫，我笑他人看不穿；不见五陵豪杰墓，无花无酒锄做田。

※ 来源:·BBS 水木清华站 smth.org·[FROM: 202.204.7.234]