Cari di Perl 
     Perl User Manual
Daftar Isi ☛
(Sebelumnya) Tips for Perl core C code hackingPerl regular expression plugin ... (Berikutnya)
Internals and C language interface

Description of the Perl regular expression engine.

Daftar Isi

NAME

perlreguts - Description of the Perl regular expression engine.

DESCRIPTION

This document is an attempt to shine some light on the guts of the regexengine and how it works. The regex engine represents a significant chunkof the perl codebase, but is relatively poorly understood. This documentis a meagre attempt at addressing this situation. It is derived from theauthor's experience, comments in the source code, other papers on theregex engine, feedback on the perl5-porters mail list, and no doubt otherplaces as well.

NOTICE! It should be clearly understood that the behavior andstructures discussed in this represents the state of the engine as theauthor understood it at the time of writing. It is NOT an APIdefinition, it is purely an internals guide for those who want to hackthe regex engine, or understand how the regex engine works. Readers ofthis document are expected to understand perl's regex syntax and itsusage in detail. If you want to learn about the basics of Perl'sregular expressions, see perlre. And if you want to replace theregex engine with your own, see perlreapi.

OVERVIEW

A quick note on terms

There is some debate as to whether to say "regexp" or "regex". In thisdocument we will use the term "regex" unless there is a special reasonnot to, in which case we will explain why.

When speaking about regexes we need to distinguish between their sourcecode form and their internal form. In this document we will use the term"pattern" when we speak of their textual, source code form, and the term"program" when we speak of their internal representation. Thesecorrespond to the terms S-regex and B-regex that Mark JasonDominus employs in his paper on "Rx" ([1] in REFERENCES).

What is a regular expression engine?

A regular expression engine is a program that takes a set of constraintsspecified in a mini-language, and then applies those constraints to atarget string, and determines whether or not the string satisfies theconstraints. See perlre for a full definition of the language.

In less grandiose terms, the first part of the job is to turn a pattern intosomething the computer can efficiently use to find the matching point inthe string, and the second part is performing the search itself.

To do this we need to produce a program by parsing the text. We thenneed to execute the program to find the point in the string thatmatches. And we need to do the whole thing efficiently.

Structure of a Regexp Program

High Level

Although it is a bit confusing and some people object to the terminology, itis worth taking a look at a comment that hasbeen in regexp.h for years:

This is essentially a linear encoding of a nondeterministicfinite-state machine (aka syntax charts or "railroad normal form" inparsing technology).

The term "railroad normal form" is a bit esoteric, with "syntaxdiagram/charts", or "railroad diagram/charts" being more common terms.Nevertheless it provides a useful mental image of a regex program: eachnode can be thought of as a unit of track, with a single entry and inmost cases a single exit point (there are pieces of track that fork, butstatistically not many), and the whole forms a layout with asingle entry and single exit point. The matching process can be thoughtof as a car that moves along the track, with the particular route throughthe system being determined by the character read at each possibleconnector point. A car can fall off the track at any point but it mayonly proceed as long as it matches the track.

Thus the pattern /foo(?:\w+|\d+|\s+)bar/ can be thought of as thefollowing chart:

  1.   [start]
    2.  
  2.  |
    3.  
  3.    <foo>
    4.  
  4.  |
    5.  
  5.    +-----+-----+
    6.  
  6.    | | |
    7.  
  7.  <\w+> <\d+> <\s+>
    8.  
  8.    | | |
    9.  
  9.    +-----+-----+
    10.  
  10.  |
    11.  
  11.    <bar>
    12.  
  12.  |
    13.  
  13.    [end]
    14.

The truth of the matter is that perl's regular expressions these days aremuch more complex than this kind of structure, but visualising it this waycan help when trying to get your bearings, and it matches thecurrent implementation pretty closely.

To be more precise, we will say that a regex program is an encodingof a graph. Each node in the graph corresponds to part ofthe original regex pattern, such as a literal string or a branch,and has a pointer to the nodes representing the next componentto be matched. Since "node" and "opcode" already have other meanings in theperl source, we will call the nodes in a regex program "regops".

The program is represented by an array of regnode structures, one ormore of which represent a single regop of the program. Structregnode is the smallest struct needed, and has a field structure which isshared with all the other larger structures.

The "next" pointers of all regops except BRANCH implement concatenation;a "next" pointer with a BRANCH on both ends of it is connecting twoalternatives. [Here we have one of the subtle syntax dependencies: anindividual BRANCH (as opposed to a collection of them) is neverconcatenated with anything because of operator precedence.]

The operand of some types of regop is a literal string; for others,it is a regop leading into a sub-program. In particular, the operandof a BRANCH node is the first regop of the branch.

NOTE: As the railroad metaphor suggests, this is not a treestructure: the tail of the branch connects to the thing following theset of BRANCHes. It is a like a single line of railway track thatsplits as it goes into a station or railway yard and rejoins as it comesout the other side.

Regops

The base structure of a regop is defined in regexp.h as follows:

  1.  struct regnode {
    2.  
  2.  U8  flags; /* Various purposes, sometimes overridden */
    3.  
  3.  U8  type; /* Opcode value as specified by regnodes.h */
    4.  
  4.  U16 next_off; /* Offset in size regnode */
    5.  
  5.  };
    6.

Other larger regnode-like structures are defined in regcomp.h. Theyare almost like subclasses in that they have the same fields asregnode, with possibly additional fields following inthe structure, and in some cases the specific meaning (and name)of some of base fields are overridden. The following is a morecomplete description.

  • regnode_1
  • regnode_2

    regnode_1 structures have the same header, followed by a singlefour-byte argument; regnode_2 structures contain two two-bytearguments instead:

    1.  regnode_1 U32 arg1;
      2.  
    2.  regnode_2 U16 arg1;  U16 arg2;
      3.
  • regnode_string

    regnode_string structures, used for literal strings, follow the headerwith a one-byte length and then the string data. Strings are padded onthe end with zero bytes so that the total length of the node is amultiple of four bytes:

    1.  regnode_string   char string[1];
      2.  
    2.  U8 str_len; /* overrides flags */
      3.
  • regnode_charclass

    Character classes are represented by regnode_charclass structures,which have a four-byte argument and then a 32-byte (256-bit) bitmapindicating which characters are included in the class.

    1.  regnode_charclass U32 arg1;
      2.  
    2.  char bitmap[ANYOF_BITMAP_SIZE];
      3.
  • regnode_charclass_class

    There is also a larger form of a char class structure used to representPOSIX char classes called regnode_charclass_class which has anadditional 4-byte (32-bit) bitmap indicating which POSIX char classeshave been included.

    1.  regnode_charclass_class  U32 arg1;
      2.  
    2.  char bitmap[ANYOF_BITMAP_SIZE];
      3.  
    3.  char classflags[ANYOF_CLASSBITMAP_SIZE];
      4.

regnodes.h defines an array called regarglen[] which gives the sizeof each opcode in units of size regnode (4-byte). A macro is usedto calculate the size of an EXACT node based on its str_len field.

The regops are defined in regnodes.h which is generated fromregcomp.sym by regcomp.pl. Currently the maximum possible numberof distinct regops is restricted to 256, with about a quarter alreadyused.

A set of macros makes accessing the fieldseasier and more consistent. These include OP(), which is used to determinethe type of a regnode-like structure; NEXT_OFF(), which is the offset tothe next node (more on this later); ARG(), ARG1(), ARG2(), ARG_SET(),and equivalents for reading and setting the arguments; and STR_LEN(),STRING() and OPERAND() for manipulating strings and regop bearingtypes.

What regop is next?

There are three distinct concepts of "next" in the regex engine, andit is important to keep them clear.

  • There is the "next regnode" from a given regnode, a value which israrely useful except that sometimes it matches up in terms of valuewith one of the others, and that sometimes the code assumes this toalways be so.

  • There is the "next regop" from a given regop/regnode. This is theregop physically located after the current one, as determined bythe size of the current regop. This is often useful, such as whendumping the structure we use this order to traverse. Sometimes the codeassumes that the "next regnode" is the same as the "next regop", or inother words assumes that the sizeof a given regop type is always goingto be one regnode large.

  • There is the "regnext" from a given regop. This is the regop whichis reached by jumping forward by the value of NEXT_OFF(),or in a few cases for longer jumps by the arg1 field of the regnode_1structure. The subroutine regnext() handles this transparently.This is the logical successor of the node, which in some cases, likethat of the BRANCH regop, has special meaning.

Process Overview

Broadly speaking, performing a match of a string against a patterninvolves the following steps:

  • A. Compilation
    1.
    Parsing for size
    2.
    Parsing for construction
    3.
    Peep-hole optimisation and analysis
  • B. Execution
    4.
    Start position and no-match optimisations
    5.
    Program execution

Where these steps occur in the actual execution of a perl program isdetermined by whether the pattern involves interpolating any stringvariables. If interpolation occurs, then compilation happens at run time. If itdoes not, then compilation is performed at compile time. (The /o modifier changes this,as does qr// to a certain extent.) The engine doesn't really care thatmuch.

Compilation

This code resides primarily in regcomp.c, along with the header filesregcomp.h, regexp.h and regnodes.h.

Compilation starts with pregcomp(), which is mostly an initialisationwrapper which farms work out to two other routines for the heavy lifting: thefirst is reg(), which is the start point for parsing; the second,study_chunk(), is responsible for optimisation.

Initialisation in pregcomp() mostly involves the creation and data-fillingof a special structure, RExC_state_t (defined in regcomp.c).Almost all internally-used routines in regcomp.h take a pointer to oneof these structures as their first argument, with the name pRExC_state.This structure is used to store the compilation state and contains manyfields. Likewise there are many macros which operate on thisvariable: anything that looks like RExC_xxxx is a macro that operates onthis pointer/structure.

Parsing for size

In this pass the input pattern is parsed in order to calculate how muchspace is needed for each regop we would need to emit. The size is alsoused to determine whether long jumps will be required in the program.

This stage is controlled by the macro SIZE_ONLY being set.

The parse proceeds pretty much exactly as it does during theconstruction phase, except that most routines are short-circuited tochange the size field RExC_size and not do anything else.

Parsing for construction

Once the size of the program has been determined, the pattern is parsedagain, but this time for real. Now SIZE_ONLY will be false, and theactual construction can occur.

reg() is the start of the parse process. It is responsible forparsing an arbitrary chunk of pattern up to either the end of thestring, or the first closing parenthesis it encounters in the pattern.This means it can be used to parse the top-level regex, or any sectioninside of a grouping parenthesis. It also handles the "special parens"that perl's regexes have. For instance when parsing /x(?:foo)y/ reg()will at one point be called to parse from the "?" symbol up to andincluding the ")".

Additionally, reg() is responsible for parsing the one or morebranches from the pattern, and for "finishing them off" by correctlysetting their next pointers. In order to do the parsing, it repeatedlycalls out to regbranch(), which is responsible for handling up to thefirst | symbol it sees.

regbranch() in turn calls regpiece() whichhandles "things" followed by a quantifier. In order to parse the"things", regatom() is called. This is the lowest level routine, whichparses out constant strings, character classes, and thevarious special symbols like $. If regatom() encounters a "("character it in turn calls reg().

The routine regtail() is called by both reg() and regbranch()in order to "set the tail pointer" correctly. When executing andwe get to the end of a branch, we need to go to the node following thegrouping parens. When parsing, however, we don't know where the end willbe until we get there, so when we do we must go back and update theoffsets as appropriate. regtail is used to make this easier.

A subtlety of the parsing process means that a regex like /foo/ isoriginally parsed into an alternation with a single branch. It is onlyafterwards that the optimiser converts single branch alternations into thesimpler form.

Parse Call Graph and a Grammar

The call graph looks like this:

  1.  reg() # parse a top level regex, or inside of parens
    2.  
  2.  regbranch()  # parse a single branch of an alternation
    3.  
  3.  regpiece()   # parse a pattern followed by a quantifier
    4.  
  4.  regatom() # parse a simple pattern
    5.  
  5.  regclass()   #   used to handle a class
    6.  
  6.  reg() #   used to handle a parenthesised subpattern
    7.  
  7.  ....
    8.  
  8.  ...
    9.  
  9.  regtail() # finish off the branch
    10.  
  10.  ...
    11.  
  11.  regtail() # finish off the branch sequence. Tie each
    12.  
  12.  # branch's tail to the tail of the sequence
    13.  
  13.  # (NEW) In Debug mode this is
    14.  
  14.  # regtail_study().
    15.

A grammar form might be something like this:

  1.  atom  : constant | class
    2.  
  2.  quant : '*' | '+' | '?' | '{min,max}'
    3.  
  3.  _branch: piece
    4.  
  4.    | piece _branch
    5.  
  5.    | nothing
    6.  
  6.  branch: _branch
    7.  
  7.   | _branch '|' branch
    8.  
  8.  group : '(' branch ')'
    9.  
  9.  _piece: atom | group
    10.  
  10.  piece : _piece
    11.  
  11.   | _piece quant
    12.

Debug Output

In the 5.9.x development version of perl you can use re Debug => 'PARSE'to see some trace information about the parse process. We will start with somesimple patterns and build up to more complex patterns.

So when we parse /foo/ we see something like the following table. Theleft shows what is being parsed, and the number indicates where the next regopwould go. The stuff on the right is the trace output of the graph. Thenames are chosen to be short to make it less dense on the screen. 'tsdy'is a special form of regtail() which does some extra analysis.

  1.  >foo< 1 reg
    2.  
  2.   brnc
    3.  
  3.  piec
    4.  
  4.   atom
    5.  
  5.  >< 4  tsdy~ EXACT <foo> (EXACT) (1)
    6.  
  6.   ~ attach to END (3) offset to 2
    7.

The resulting program then looks like:

  1.    1: EXACT <foo>(3)
    2.  
  2.    3: END(0)
    3.

As you can see, even though we parsed out a branch and a piece, it was ultimatelyonly an atom. The final program shows us how things work. We have an EXACT regop,followed by an END regop. The number in parens indicates where the regnext ofthe node goes. The regnext of an END regop is unused, as END regops meanwe have successfully matched. The number on the left indicates the position ofthe regop in the regnode array.

Now let's try a harder pattern. We will add a quantifier, so now we have the pattern/foo+/. We will see that regbranch() calls regpiece() twice.

  1.  >foo+< 1 reg
    2.  
  2.   brnc
    3.  
  3.  piec
    4.  
  4.   atom
    5.  
  5.  >o+<  3 piec
    6.  
  6.   atom
    7.  
  7.  >< 6 tail~ EXACT <fo> (1)
    8.  
  8.    7  tsdy~ EXACT <fo> (EXACT) (1)
    9.  
  9.   ~ PLUS (END) (3)
    10.  
  10.   ~ attach to END (6) offset to 3
    11.

And we end up with the program:

  1.    1: EXACT <fo>(3)
    2.  
  2.    3: PLUS(6)
    3.  
  3.    4:   EXACT <o>(0)
    4.  
  4.    6: END(0)
    5.

Now we have a special case. The EXACT regop has a regnext of 0. This isbecause if it matches it should try to match itself again. The PLUS regophandles the actual failure of the EXACT regop and acts appropriately (goingto regnode 6 if the EXACT matched at least once, or failing if it didn't).

Now for something much more complex: /x(?:foo*|b[a][rR])(foo|bar)$/

  1.  >x(?:foo*|b... 1 reg
    2.  
  2.   brnc
    3.  
  3.  piec
    4.  
  4.   atom
    5.  
  5.  >(?:foo*|b[... 3 piec
    6.  
  6.   atom
    7.  
  7.  >?:foo*|b[a... reg
    8.  
  8.  >foo*|b[a][...   brnc
    9.  
  9.  piec
    10.  
  10.   atom
    11.  
  11.  >o*|b[a][rR... 5 piec
    12.  
  12.   atom
    13.  
  13.  >|b[a][rR])... 8 tail~ EXACT <fo> (3)
    14.  
  14.  >b[a][rR])(... 9  brnc
    15.  
  15.   10 piec
    16.  
  16.   atom
    17.  
  17.  >[a][rR])(f...   12 piec
    18.  
  18.   atom
    19.  
  19.  >a][rR])(fo... clas
    20.  
  20.  >[rR])(foo|...   14 tail~ EXACT <b> (10)
    21.  
  21.  piec
    22.  
  22.   atom
    23.  
  23.  >rR])(foo|b... clas
    24.  
  24.  >)(foo|bar)...   25 tail~ EXACT <a> (12)
    25.  
  25.   tail~ BRANCH (3)
    26.  
  26.   26  tsdy~ BRANCH (END) (9)
    27.  
  27.   ~ attach to TAIL (25) offset to 16
    28.  
  28.   tsdy~ EXACT <fo> (EXACT) (4)
    29.  
  29.   ~ STAR (END) (6)
    30.  
  30.   ~ attach to TAIL (25) offset to 19
    31.  
  31.   tsdy~ EXACT <b> (EXACT) (10)
    32.  
  32.   ~ EXACT <a> (EXACT) (12)
    33.  
  33.   ~ ANYOF[Rr] (END) (14)
    34.  
  34.   ~ attach to TAIL (25) offset to 11
    35.  
  35.  >(foo|bar)$<   tail~ EXACT <x> (1)
    36.  
  36.  piec
    37.  
  37.   atom
    38.  
  38.  >foo|bar)$< reg
    39.  
  39.   28  brnc
    40.  
  40.  piec
    41.  
  41.   atom
    42.  
  42.  >|bar)$< 31  tail~ OPEN1 (26)
    43.  
  43.  >bar)$<  brnc
    44.  
  44.   32 piec
    45.  
  45.   atom
    46.  
  46.  >)$< 34  tail~ BRANCH (28)
    47.  
  47.   36  tsdy~ BRANCH (END) (31)
    48.  
  48.   ~ attach to CLOSE1 (34) offset to 3
    49.  
  49.   tsdy~ EXACT <foo> (EXACT) (29)
    50.  
  50.   ~ attach to CLOSE1 (34) offset to 5
    51.  
  51.   tsdy~ EXACT <bar> (EXACT) (32)
    52.  
  52.   ~ attach to CLOSE1 (34) offset to 2
    53.  
  53.  >$< tail~ BRANCH (3)
    54.  
  54.  ~ BRANCH (9)
    55.  
  55.  ~ TAIL (25)
    56.  
  56.  piec
    57.  
  57.   atom
    58.  
  58.  ><   37 tail~ OPEN1 (26)
    59.  
  59.  ~ BRANCH (28)
    60.  
  60.  ~ BRANCH (31)
    61.  
  61.  ~ CLOSE1 (34)
    62.  
  62.   38  tsdy~ EXACT <x> (EXACT) (1)
    63.  
  63.   ~ BRANCH (END) (3)
    64.  
  64.   ~ BRANCH (END) (9)
    65.  
  65.   ~ TAIL (END) (25)
    66.  
  66.   ~ OPEN1 (END) (26)
    67.  
  67.   ~ BRANCH (END) (28)
    68.  
  68.   ~ BRANCH (END) (31)
    69.  
  69.   ~ CLOSE1 (END) (34)
    70.  
  70.   ~ EOL (END) (36)
    71.  
  71.   ~ attach to END (37) offset to 1
    72.

Resulting in the program

  1.    1: EXACT <x>(3)
    2.  
  2.    3: BRANCH(9)
    3.  
  3.    4:   EXACT <fo>(6)
    4.  
  4.    6:   STAR(26)
    5.  
  5.    7: EXACT <o>(0)
    6.  
  6.    9: BRANCH(25)
    7.  
  7.   10:   EXACT <ba>(14)
    8.  
  8.   12:   OPTIMIZED (2 nodes)
    9.  
  9.   14:   ANYOF[Rr](26)
    10.  
  10.   25: TAIL(26)
    11.  
  11.   26: OPEN1(28)
    12.  
  12.   28:   TRIE-EXACT(34)
    13.  
  13.  [StS:1 Wds:2 Cs:6 Uq:5 #Sts:7 Mn:3 Mx:3 Stcls:bf]
    14.  
  14.   <foo>
    15.  
  15.   <bar>
    16.  
  16.   30:   OPTIMIZED (4 nodes)
    17.  
  17.   34: CLOSE1(36)
    18.  
  18.   36: EOL(37)
    19.  
  19.   37: END(0)
    20.

Here we can see a much more complex program, with various optimisations inplay. At regnode 10 we see an example where a character class with onlyone character in it was turned into an EXACT node. We can also see wherean entire alternation was turned into a TRIE-EXACT node. As a consequence,some of the regnodes have been marked as optimised away. We can see thatthe $ symbol has been converted into an EOL regop, a special piece ofcode that looks for \n or the end of the string.

The next pointer for BRANCHes is interesting in that it points at whereexecution should go if the branch fails. When executing, if the enginetries to traverse from a branch to a regnext that isn't a branch thenthe engine will know that the entire set of branches has failed.

Peep-hole Optimisation and Analysis

The regular expression engine can be a weighty tool to wield. On longstrings and complex patterns it can end up having to do a lot of workto find a match, and even more to decide that no match is possible.Consider a situation like the following pattern.

  1.    'ababababababababababab' =~ /(a|b)*z/
    2.

The (a|b)* part can match at every char in the string, and then failevery time because there is no z in the string. So obviously we canavoid using the regex engine unless there is a z in the string.Likewise in a pattern like:

  1.    /foo(\w+)bar/
    2.

In this case we know that the string must contain a foo which must befollowed by bar. We can use Fast Boyer-Moore matching as implementedin fbm_instr() to find the location of these strings. If they don't existthen we don't need to resort to the much more expensive regex engine.Even better, if they do exist then we can use their positions toreduce the search space that the regex engine needs to cover to determineif the entire pattern matches.

There are various aspects of the pattern that can be used to facilitateoptimisations along these lines:

  • anchored fixed strings
  • floating fixed strings
  • minimum and maximum length requirements
  • start class
  • Beginning/End of line positions

Another form of optimisation that can occur is the post-parse "peep-hole"optimisation, where inefficient constructs are replaced by more efficientconstructs. The TAIL regops which are used during parsing to mark the endof branches and the end of groups are examples of this. These regops are usedas place-holders during construction and "always match" so they can be"optimised away" by making the things that point to the TAIL point to thething that TAIL points to, thus "skipping" the node.

Another optimisation that can occur is that of "EXACT merging" which iswhere two consecutive EXACT nodes are merged into a singleregop. An even more aggressive form of this is that a branchsequence of the form EXACT BRANCH ... EXACT can be converted into aTRIE-EXACT regop.

All of this occurs in the routine study_chunk() which uses a specialstructure scan_data_t to store the analysis that it has performed, anddoes the "peep-hole" optimisations as it goes.

The code involved in study_chunk() is extremely cryptic. Be careful. :-)

Execution

Execution of a regex generally involves two phases, the first beingfinding the start point in the string where we should match from,and the second being running the regop interpreter.

If we can tell that there is no valid start point then we don't bother runninginterpreter at all. Likewise, if we know from the analysis phase that wecannot detect a short-cut to the start position, we go straight to theinterpreter.

The two entry points are re_intuit_start() and pregexec(). These routineshave a somewhat incestuous relationship with overlap between their functions,and pregexec() may even call re_intuit_start() on its own. Neverthelessother parts of the perl source code may call into either, or both.

Execution of the interpreter itself used to be recursive, but thanks to theefforts of Dave Mitchell in the 5.9.x development track, that has changed: now aninternal stack is maintained on the heap and the routine is fullyiterative. This can make it tricky as the code is quite conservativeabout what state it stores, with the result that two consecutive lines in thecode can actually be running in totally different contexts due to thesimulated recursion.

Start position and no-match optimisations

re_intuit_start() is responsible for handling start points and no-matchoptimisations as determined by the results of the analysis done bystudy_chunk() (and described in Peep-hole Optimisation and Analysis).

The basic structure of this routine is to try to find the start- and/orend-points of where the pattern could match, and to ensure that the stringis long enough to match the pattern. It tries to use more efficientmethods over less efficient methods and may involve considerablecross-checking of constraints to find the place in the string that matches.For instance it may try to determine that a given fixed string must benot only present but a certain number of chars before the end of thestring, or whatever.

It calls several other routines, such as fbm_instr() which doesFast Boyer Moore matching and find_byclass() which is responsible forfinding the start using the first mandatory regop in the program.

When the optimisation criteria have been satisfied, reg_try() is calledto perform the match.

Program execution

pregexec() is the main entry point for running a regex. It containssupport for initialising the regex interpreter's state, runningre_intuit_start() if needed, and running the interpreter on the stringfrom various start positions as needed. When it is necessary to usethe regex interpreter pregexec() calls regtry().

regtry() is the entry point into the regex interpreter. It expectsas arguments a pointer to a regmatch_info structure and a pointer toa string. It returns an integer 1 for success and a 0 for failure.It is basically a set-up wrapper around regmatch().

regmatch is the main "recursive loop" of the interpreter. It isbasically a giant switch statement that implements a state machine, wherethe possible states are the regops themselves, plus a number of additionalintermediate and failure states. A few of the states are implemented assubroutines but the bulk are inline code.

MISCELLANEOUS

Unicode and Localisation Support

When dealing with strings containing characters that cannot be representedusing an eight-bit character set, perl uses an internal representationthat is a permissive version of Unicode's UTF-8 encoding[2]. This uses singlebytes to represent characters from the ASCII character set, and sequencesof two or more bytes for all other characters. (See perlunitutfor more information about the relationship between UTF-8 and perl'sencoding, utf8. The difference isn't important for this discussion.)

No matter how you look at it, Unicode support is going to be a pain in aregex engine. Tricks that might be fine when you have 256 possiblecharacters often won't scale to handle the size of the UTF-8 characterset. Things you can take for granted with ASCII may not be true withUnicode. For instance, in ASCII, it is safe to assume thatsizeof(char1) == sizeof(char2), but in UTF-8 it isn't. Unicode case folding isvastly more complex than the simple rules of ASCII, and even when notusing Unicode but only localised single byte encodings, things can gettricky (for example, LATIN SMALL LETTER SHARP S (U+00DF, ß)should match 'SS' in localised case-insensitive matching).

Making things worse is that UTF-8 support was a later addition to theregex engine (as it was to perl) and this necessarily made things a lotmore complicated. Obviously it is easier to design a regex engine withUnicode support in mind from the beginning than it is to retrofit it toone that wasn't.

Nearly all regops that involve looking at the input string havetwo cases, one for UTF-8, and one not. In fact, it's often more complexthan that, as the pattern may be UTF-8 as well.

Care must be taken when making changes to make sure that you handleUTF-8 properly, both at compile time and at execution time, includingwhen the string and pattern are mismatched.

The following comment in regcomp.h gives an example of exactly howtricky this can be:

  1.  Two problematic code points in Unicode casefolding of EXACT nodes:
    2.  
  2.  
  3.  U+0390 - GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS
    4.  
  4.  U+03B0 - GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS
    5.  
  5.  
  6.  which casefold to
    7.  
  7.  
  8.  Unicode  UTF-8
    9.  
  9.  
  10.  U+03B9 U+0308 U+0301 0xCE 0xB9 0xCC 0x88 0xCC 0x81
    11.  
  11.  U+03C5 U+0308 U+0301 0xCF 0x85 0xCC 0x88 0xCC 0x81
    12.  
  12.  
  13.  This means that in case-insensitive matching (or "loose matching",
    14.  
  14.  as Unicode calls it), an EXACTF of length six (the UTF-8 encoded
    15.  
  15.  byte length of the above casefolded versions) can match a target
    16.  
  16.  string of length two (the byte length of UTF-8 encoded U+0390 or
    17.  
  17.  U+03B0). This would rather mess up the minimum length computation.
    18.  
  18.  
  19.  What we'll do is to look for the tail four bytes, and then peek
    20.  
  20.  at the preceding two bytes to see whether we need to decrease
    21.  
  21.  the minimum length by four (six minus two).
    22.  
  22.  
  23.  Thanks to the design of UTF-8, there cannot be false matches:
    24.  
  24.  A sequence of valid UTF-8 bytes cannot be a subsequence of
    25.  
  25.  another valid sequence of UTF-8 bytes.
    26.

Base Structures

The regexp structure described in perlreapi is common to allregex engines. Two of its fields that are intended for the private useof the regex engine that compiled the pattern. These are theintflags and pprivate members. The pprivate is a void pointer toan arbitrary structure whose use and management is the responsibilityof the compiling engine. perl will never modify either of thesevalues. In the case of the stock engine the structure pointed to bypprivate is called regexp_internal.

Its pprivate and intflags fields contain dataspecific to each engine.

There are two structures used to store a compiled regular expression.One, the regexp structure described in perlreapi is populated bythe engine currently being. used and some of its fields read by perl toimplement things such as the stringification of qr//.

The other structure is pointed to be the regexp struct'spprivate and is in addition to intflags in the same structconsidered to be the property of the regex engine which compiled theregular expression;

The regexp structure contains all the data that perl needs to be aware ofto properly work with the regular expression. It includes data aboutoptimisations that perl can use to determine if the regex engine shouldreally be used, and various other control info that is needed to properlyexecute patterns in various contexts such as is the pattern anchored insome way, or what flags were used during the compile, or whether theprogram contains special constructs that perl needs to be aware of.

In addition it contains two fields that are intended for the private useof the regex engine that compiled the pattern. These are the intflagsand pprivate members. The pprivate is a void pointer to an arbitrarystructure whose use and management is the responsibility of the compilingengine. perl will never modify either of these values.

As mentioned earlier, in the case of the default engines, the pprivatewill be a pointer to a regexp_internal structure which holds the compiledprogram and any additional data that is private to the regex engineimplementation.

Perl's pprivate structure

The following structure is used as the pprivate struct by perl'sregex engine. Since it is specific to perl it is only of curiosityvalue to other engine implementations.

  1.  typedef struct regexp_internal {
    2.  
  2.  regexp_paren_ofs *swap; /* Swap copy of *startp / *endp */
    3.  
  3.  U32 *offsets;   /* offset annotations 20001228 MJD 
    4.  
  4.    data about mapping the program to the 
    5.  
  5.    string*/
    6.  
  6.  regnode *regstclass; /* Optional startclass as identified or constructed
    7.  
  7.    by the optimiser */
    8.  
  8.  struct reg_data *data;  /* Additional miscellaneous data used by the program.
    9.  
  9.    Used to make it easier to clone and free arbitrary
    10.  
  10.    data that the regops need. Often the ARG field of
    11.  
  11.    a regop is an index into this structure */
    12.  
  12.  regnode program[1]; /* Unwarranted chumminess with compiler. */
    13.  
  13.  } regexp_internal;
    14.
  • swap

    swap formerly was an extra set of startp/endp stored in aregexp_paren_ofs struct. This was used when the last successful matchwas from the same pattern as the current pattern, so that a partialmatch didn't overwrite the previous match's results, but it caused aproblem with re-entrant code such as trying to build the UTF-8 swashes.Currently unused and left for backward compatibility with 5.10.0.

  • offsets

    Offsets holds a mapping of offset in the programto offset in the precomp string. This is only used by ActiveState'svisual regex debugger.

  • regstclass

    Special regop that is used by re_intuit_start() to check if a patterncan match at a certain position. For instance if the regex engine knowsthat the pattern must start with a 'Z' then it can scan the string untilit finds one and then launch the regex engine from there. The routinethat handles this is called find_by_class(). Sometimes this fieldpoints at a regop embedded in the program, and sometimes it points atan independent synthetic regop that has been constructed by the optimiser.

  • data

    This field points at a reg_data structure, which is defined as follows

    1.  struct reg_data {
      2.  
    2.  U32 count;
      3.  
    3.  U8 *what;
      4.  
    4.  void* data[1];
      5.  
    5.  };
      6.

    This structure is used for handling data structures that the regex engineneeds to handle specially during a clone or free operation on the compiledproduct. Each element in the data array has a corresponding element in thewhat array. During compilation regops that need special structures storedwill add an element to each array using the add_data() routine and then storethe index in the regop.

  • program

    Compiled program. Inlined into the structure so the entire struct can betreated as a single blob.

SEE ALSO

perlreapi

perlre

perlunitut

AUTHOR

by Yves Orton, 2006.

With excerpts from Perl, and contributions and suggestions fromRonald J. Kimball, Dave Mitchell, Dominic Dunlop, Mark Jason Dominus,Stephen McCamant, and David Landgren.

LICENCE

Same terms as Perl.

REFERENCES

[1] http://perl.plover.com/Rx/paper/

[2] http://www.unicode.org

Page index
 
Source : perldoc.perl.org - Official documentation for the Perl programming language
Site maintained by Jon Allen (JJ)     See the project page for more details
Documentation maintained by the Perl 5 Porters
(Sebelumnya) Tips for Perl core C code hackingPerl regular expression plugin ... (Berikutnya)