Perl version
Manual
Reference
Modules
perlretut - Perl regular expressions tutorial
This page provides a basic tutorial on understanding, creating and
using regular expressions in Perl. It serves as a complement to the
reference page on regular expressions perlre. Regular expressions
are an integral part of the m//
, s///
, qr//
and split
operators and so this tutorial also overlaps with
Regexp Quote-Like Operators in perlop and split.
Perl is widely renowned for excellence in text processing, and regular expressions are one of the big factors behind this fame. Perl regular expressions display an efficiency and flexibility unknown in most other computer languages. Mastering even the basics of regular expressions will allow you to manipulate text with surprising ease.
What is a regular expression? A regular expression is simply a string
that describes a pattern. Patterns are in common use these days;
examples are the patterns typed into a search engine to find web pages
and the patterns used to list files in a directory, e.g., ls *.txt
or dir *.*
. In Perl, the patterns described by regular expressions
are used to search strings, extract desired parts of strings, and to
do search and replace operations.
Regular expressions have the undeserved reputation of being abstract
and difficult to understand. Regular expressions are constructed using
simple concepts like conditionals and loops and are no more difficult
to understand than the corresponding if
conditionals and while
loops in the Perl language itself. In fact, the main challenge in
learning regular expressions is just getting used to the terse
notation used to express these concepts.
This tutorial flattens the learning curve by discussing regular expression concepts, along with their notation, one at a time and with many examples. The first part of the tutorial will progress from the simplest word searches to the basic regular expression concepts. If you master the first part, you will have all the tools needed to solve about 98% of your needs. The second part of the tutorial is for those comfortable with the basics and hungry for more power tools. It discusses the more advanced regular expression operators and introduces the latest cutting-edge innovations.
A note: to save time, 'regular expression' is often abbreviated as regexp or regex. Regexp is a more natural abbreviation than regex, but is harder to pronounce. The Perl pod documentation is evenly split on regexp vs regex; in Perl, there is more than one way to abbreviate it. We'll use regexp in this tutorial.
The simplest regexp is simply a word, or more generally, a string of characters. A regexp consisting of a word matches any string that contains that word:
- "Hello World" =~ /World/; # matches
What is this Perl statement all about? "Hello World"
is a simple
double-quoted string. World
is the regular expression and the
//
enclosing /World/
tells Perl to search a string for a match.
The operator =~
associates the string with the regexp match and
produces a true value if the regexp matched, or false if the regexp
did not match. In our case, World
matches the second word in
"Hello World"
, so the expression is true. Expressions like this
are useful in conditionals:
- if ("Hello World" =~ /World/) {
- print "It matches\n";
- }
- else {
- print "It doesn't match\n";
- }
There are useful variations on this theme. The sense of the match can
be reversed by using the !~
operator:
- if ("Hello World" !~ /World/) {
- print "It doesn't match\n";
- }
- else {
- print "It matches\n";
- }
The literal string in the regexp can be replaced by a variable:
- $greeting = "World";
- if ("Hello World" =~ /$greeting/) {
- print "It matches\n";
- }
- else {
- print "It doesn't match\n";
- }
If you're matching against the special default variable $_
, the
$_ =~
part can be omitted:
- $_ = "Hello World";
- if (/World/) {
- print "It matches\n";
- }
- else {
- print "It doesn't match\n";
- }
And finally, the //
default delimiters for a match can be changed
to arbitrary delimiters by putting an 'm'
out front:
- "Hello World" =~ m!World!; # matches, delimited by '!'
- "Hello World" =~ m{World}; # matches, note the matching '{}'
- "/usr/bin/perl" =~ m"/perl"; # matches after '/usr/bin',
- # '/' becomes an ordinary char
/World/
, m!World!
, and m{World}
all represent the
same thing. When, e.g., the quote ("
) is used as a delimiter, the forward
slash '/'
becomes an ordinary character and can be used in this regexp
without trouble.
Let's consider how different regexps would match "Hello World"
:
- "Hello World" =~ /world/; # doesn't match
- "Hello World" =~ /o W/; # matches
- "Hello World" =~ /oW/; # doesn't match
- "Hello World" =~ /World /; # doesn't match
The first regexp world
doesn't match because regexps are
case-sensitive. The second regexp matches because the substring
'o W'
occurs in the string "Hello World"
. The space
character ' ' is treated like any other character in a regexp and is
needed to match in this case. The lack of a space character is the
reason the third regexp 'oW'
doesn't match. The fourth regexp
'World '
doesn't match because there is a space at the end of the
regexp, but not at the end of the string. The lesson here is that
regexps must match a part of the string exactly in order for the
statement to be true.
If a regexp matches in more than one place in the string, Perl will always match at the earliest possible point in the string:
- "Hello World" =~ /o/; # matches 'o' in 'Hello'
- "That hat is red" =~ /hat/; # matches 'hat' in 'That'
With respect to character matching, there are a few more points you need to know about. First of all, not all characters can be used 'as is' in a match. Some characters, called metacharacters, are reserved for use in regexp notation. The metacharacters are
- {}[]()^$.|*+?\
The significance of each of these will be explained in the rest of the tutorial, but for now, it is important only to know that a metacharacter can be matched by putting a backslash before it:
- "2+2=4" =~ /2+2/; # doesn't match, + is a metacharacter
- "2+2=4" =~ /2\+2/; # matches, \+ is treated like an ordinary +
- "The interval is [0,1)." =~ /[0,1)./ # is a syntax error!
- "The interval is [0,1)." =~ /\[0,1\)\./ # matches
- "#!/usr/bin/perl" =~ /#!\/usr\/bin\/perl/; # matches
In the last regexp, the forward slash '/'
is also backslashed,
because it is used to delimit the regexp. This can lead to LTS
(leaning toothpick syndrome), however, and it is often more readable
to change delimiters.
- "#!/usr/bin/perl" =~ m!#\!/usr/bin/perl!; # easier to read
The backslash character '\'
is a metacharacter itself and needs to
be backslashed:
- 'C:\WIN32' =~ /C:\\WIN/; # matches
In addition to the metacharacters, there are some ASCII characters
which don't have printable character equivalents and are instead
represented by escape sequences. Common examples are \t
for a
tab, \n
for a newline, \r
for a carriage return and \a
for a
bell (or alert). If your string is better thought of as a sequence of arbitrary
bytes, the octal escape sequence, e.g., \033
, or hexadecimal escape
sequence, e.g., \x1B
may be a more natural representation for your
bytes. Here are some examples of escapes:
- "1000\t2000" =~ m(0\t2) # matches
- "1000\n2000" =~ /0\n20/ # matches
- "1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000"
- "cat" =~ /\o{143}\x61\x74/ # matches in ASCII, but a weird way
- # to spell cat
If you've been around Perl a while, all this talk of escape sequences may seem familiar. Similar escape sequences are used in double-quoted strings and in fact the regexps in Perl are mostly treated as double-quoted strings. This means that variables can be used in regexps as well. Just like double-quoted strings, the values of the variables in the regexp will be substituted in before the regexp is evaluated for matching purposes. So we have:
- $foo = 'house';
- 'housecat' =~ /$foo/; # matches
- 'cathouse' =~ /cat$foo/; # matches
- 'housecat' =~ /${foo}cat/; # matches
So far, so good. With the knowledge above you can already perform searches with just about any literal string regexp you can dream up. Here is a very simple emulation of the Unix grep program:
- % cat > simple_grep
- #!/usr/bin/perl
- $regexp = shift;
- while (<>) {
- print if /$regexp/;
- }
- ^D
- % chmod +x simple_grep
- % simple_grep abba /usr/dict/words
- Babbage
- cabbage
- cabbages
- sabbath
- Sabbathize
- Sabbathizes
- sabbatical
- scabbard
- scabbards
This program is easy to understand. #!/usr/bin/perl
is the standard
way to invoke a perl program from the shell.
$regexp = shift;
saves the first command line argument as the
regexp to be used, leaving the rest of the command line arguments to
be treated as files. while (<>)
loops over all the lines in
all the files. For each line, print if /$regexp/;
prints the
line if the regexp matches the line. In this line, both print
and
/$regexp/
use the default variable $_
implicitly.
With all of the regexps above, if the regexp matched anywhere in the
string, it was considered a match. Sometimes, however, we'd like to
specify where in the string the regexp should try to match. To do
this, we would use the anchor metacharacters ^
and $
. The
anchor ^
means match at the beginning of the string and the anchor
$
means match at the end of the string, or before a newline at the
end of the string. Here is how they are used:
- "housekeeper" =~ /keeper/; # matches
- "housekeeper" =~ /^keeper/; # doesn't match
- "housekeeper" =~ /keeper$/; # matches
- "housekeeper\n" =~ /keeper$/; # matches
The second regexp doesn't match because ^
constrains keeper
to
match only at the beginning of the string, but "housekeeper"
has
keeper starting in the middle. The third regexp does match, since the
$
constrains keeper
to match only at the end of the string.
When both ^
and $
are used at the same time, the regexp has to
match both the beginning and the end of the string, i.e., the regexp
matches the whole string. Consider
- "keeper" =~ /^keep$/; # doesn't match
- "keeper" =~ /^keeper$/; # matches
- "" =~ /^$/; # ^$ matches an empty string
The first regexp doesn't match because the string has more to it than
keep
. Since the second regexp is exactly the string, it
matches. Using both ^
and $
in a regexp forces the complete
string to match, so it gives you complete control over which strings
match and which don't. Suppose you are looking for a fellow named
bert, off in a string by himself:
- "dogbert" =~ /bert/; # matches, but not what you want
- "dilbert" =~ /^bert/; # doesn't match, but ..
- "bertram" =~ /^bert/; # matches, so still not good enough
- "bertram" =~ /^bert$/; # doesn't match, good
- "dilbert" =~ /^bert$/; # doesn't match, good
- "bert" =~ /^bert$/; # matches, perfect
Of course, in the case of a literal string, one could just as easily
use the string comparison $string eq 'bert'
and it would be
more efficient. The ^...$
regexp really becomes useful when we
add in the more powerful regexp tools below.
Although one can already do quite a lot with the literal string regexps above, we've only scratched the surface of regular expression technology. In this and subsequent sections we will introduce regexp concepts (and associated metacharacter notations) that will allow a regexp to represent not just a single character sequence, but a whole class of them.
One such concept is that of a character class. A character class
allows a set of possible characters, rather than just a single
character, to match at a particular point in a regexp. Character
classes are denoted by brackets [...]
, with the set of characters
to be possibly matched inside. Here are some examples:
- /cat/; # matches 'cat'
- /[bcr]at/; # matches 'bat, 'cat', or 'rat'
- /item[0123456789]/; # matches 'item0' or ... or 'item9'
- "abc" =~ /[cab]/; # matches 'a'
In the last statement, even though 'c'
is the first character in
the class, 'a'
matches because the first character position in the
string is the earliest point at which the regexp can match.
- /[yY][eE][sS]/; # match 'yes' in a case-insensitive way
- # 'yes', 'Yes', 'YES', etc.
This regexp displays a common task: perform a case-insensitive
match. Perl provides a way of avoiding all those brackets by simply
appending an 'i'
to the end of the match. Then /[yY][eE][sS]/;
can be rewritten as /yes/i;
. The 'i'
stands for
case-insensitive and is an example of a modifier of the matching
operation. We will meet other modifiers later in the tutorial.
We saw in the section above that there were ordinary characters, which
represented themselves, and special characters, which needed a
backslash \
to represent themselves. The same is true in a
character class, but the sets of ordinary and special characters
inside a character class are different than those outside a character
class. The special characters for a character class are -]\^$
(and
the pattern delimiter, whatever it is).
]
is special because it denotes the end of a character class. $
is
special because it denotes a scalar variable. \
is special because
it is used in escape sequences, just like above. Here is how the
special characters ]$\
are handled:
- /[\]c]def/; # matches ']def' or 'cdef'
- $x = 'bcr';
- /[$x]at/; # matches 'bat', 'cat', or 'rat'
- /[\$x]at/; # matches '$at' or 'xat'
- /[\\$x]at/; # matches '\at', 'bat, 'cat', or 'rat'
The last two are a little tricky. In [\$x]
, the backslash protects
the dollar sign, so the character class has two members $
and x
.
In [\\$x]
, the backslash is protected, so $x
is treated as a
variable and substituted in double quote fashion.
The special character '-'
acts as a range operator within character
classes, so that a contiguous set of characters can be written as a
range. With ranges, the unwieldy [0123456789]
and [abc...xyz]
become the svelte [0-9]
and [a-z]
. Some examples are
- /item[0-9]/; # matches 'item0' or ... or 'item9'
- /[0-9bx-z]aa/; # matches '0aa', ..., '9aa',
- # 'baa', 'xaa', 'yaa', or 'zaa'
- /[0-9a-fA-F]/; # matches a hexadecimal digit
- /[0-9a-zA-Z_]/; # matches a "word" character,
- # like those in a Perl variable name
If '-'
is the first or last character in a character class, it is
treated as an ordinary character; [-ab]
, [ab-]
and [a\-b]
are
all equivalent.
The special character ^
in the first position of a character class
denotes a negated character class, which matches any character but
those in the brackets. Both [...]
and [^...]
must match a
character, or the match fails. Then
- /[^a]at/; # doesn't match 'aat' or 'at', but matches
- # all other 'bat', 'cat, '0at', '%at', etc.
- /[^0-9]/; # matches a non-numeric character
- /[a^]at/; # matches 'aat' or '^at'; here '^' is ordinary
Now, even [0-9]
can be a bother to write multiple times, so in the
interest of saving keystrokes and making regexps more readable, Perl
has several abbreviations for common character classes, as shown below.
Since the introduction of Unicode, unless the //a
modifier is in
effect, these character classes match more than just a few characters in
the ASCII range.
\d matches a digit, not just [0-9] but also digits from non-roman scripts
\s matches a whitespace character, the set [\ \t\r\n\f] and others
\w matches a word character (alphanumeric or _), not just [0-9a-zA-Z_] but also digits and characters from non-roman scripts
\D is a negated \d; it represents any other character than a digit, or [^\d]
\S is a negated \s; it represents any non-whitespace character [^\s]
\W is a negated \w; it represents any non-word character [^\w]
The period '.' matches any character but "\n" (unless the modifier //s
is
in effect, as explained below).
\N, like the period, matches any character but "\n", but it does so
regardless of whether the modifier //s
is in effect.
The //a
modifier, available starting in Perl 5.14, is used to
restrict the matches of \d, \s, and \w to just those in the ASCII range.
It is useful to keep your program from being needlessly exposed to full
Unicode (and its accompanying security considerations) when all you want
is to process English-like text. (The "a" may be doubled, //aa
, to
provide even more restrictions, preventing case-insensitive matching of
ASCII with non-ASCII characters; otherwise a Unicode "Kelvin Sign"
would caselessly match a "k" or "K".)
The \d\s\w\D\S\W
abbreviations can be used both inside and outside
of character classes. Here are some in use:
- /\d\d:\d\d:\d\d/; # matches a hh:mm:ss time format
- /[\d\s]/; # matches any digit or whitespace character
- /\w\W\w/; # matches a word char, followed by a
- # non-word char, followed by a word char
- /..rt/; # matches any two chars, followed by 'rt'
- /end\./; # matches 'end.'
- /end[.]/; # same thing, matches 'end.'
Because a period is a metacharacter, it needs to be escaped to match
as an ordinary period. Because, for example, \d
and \w
are sets
of characters, it is incorrect to think of [^\d\w]
as [\D\W]
; in
fact [^\d\w]
is the same as [^\w]
, which is the same as
[\W]
. Think DeMorgan's laws.
An anchor useful in basic regexps is the word anchor
\b
. This matches a boundary between a word character and a non-word
character \w\W
or \W\w
:
- $x = "Housecat catenates house and cat";
- $x =~ /cat/; # matches cat in 'housecat'
- $x =~ /\bcat/; # matches cat in 'catenates'
- $x =~ /cat\b/; # matches cat in 'housecat'
- $x =~ /\bcat\b/; # matches 'cat' at end of string
Note in the last example, the end of the string is considered a word boundary.
You might wonder why '.'
matches everything but "\n"
- why not
every character? The reason is that often one is matching against
lines and would like to ignore the newline characters. For instance,
while the string "\n"
represents one line, we would like to think
of it as empty. Then
- "" =~ /^$/; # matches
- "\n" =~ /^$/; # matches, $ anchors before "\n"
- "" =~ /./; # doesn't match; it needs a char
- "" =~ /^.$/; # doesn't match; it needs a char
- "\n" =~ /^.$/; # doesn't match; it needs a char other than "\n"
- "a" =~ /^.$/; # matches
- "a\n" =~ /^.$/; # matches, $ anchors before "\n"
This behavior is convenient, because we usually want to ignore newlines when we count and match characters in a line. Sometimes, however, we w