Skip to main content

Text Patterns

This page explains how a rule checks text against a pattern: which patterns rule-line accepts, how they are read, and when a pattern is refused. matches and regexMatch are the two operators that take a pattern, and both follow everything below.

Every pattern runs in linear time​

rule-line runs every pattern on a matcher whose cost grows in step with the length of the text and the size of the pattern. It does not grow exponentially, whatever the pattern looks like. A pattern therefore cannot make a run hang on a long extracted text, and you can write repeated and nested repeats such as (a+)+$, .*.*c or (?:a|ab)*c without a special form to avoid. They are accepted when the ruleset is saved, and they match quickly on any text the platform allows.

The same matcher runs in the editor and in the job workers, so a pattern that checks out while you write it behaves the same way on a real run.

Limits keep the work of one match bounded: on how long a pattern may be, on how large it may grow once its repeat counts are expanded, on how large a repeat count may be, and on how long a text a pattern is applied to. Their values are generated from the engine and listed in the text pattern limits table, together with what happens when each is exceeded.

The pattern language​

A pattern is written like a JavaScript regular expression, restricted to a set of constructs that read the same everywhere. A JavaScript pattern that stays inside the list below works as it is; one that uses anything else, such as the escapes \n or \t, is refused (see What is not available):

  • literal characters, and escapes for the characters that have a special meaning, such as \., \( and \{;
  • the wildcard ., character classes such as [A-Z0-9] and [^,;], and the shorthands \d, \D, \w, \W, \s and \S;
  • the anchors ^ and $, and the word boundaries \b and \B;
  • alternation a|b, and groups ( ... ) and (?: ... );
  • the quantifiers *, +, ?, {n}, {n,} and {n,m}, each with a lazy form such as +?.

The operators answer true or false. They report no captured text and no match position, so a capturing group and a non-capturing group behave the same.

What is not available​

A pattern that uses any of these is refused with RE_DISALLOWED_CONSTRUCT (or, for a flag, RE_DISALLOWED_FLAG), and the diagnostic points at the construct. An unrecognised construct is always refused, never accepted by omission.

  • Lookahead and lookbehind, positive and negative: (?=...), (?!...), (?<=...), (?<!...).
  • Back-references such as \1, and named groups such as (?<name>...).
  • Characters outside the Basic Multilingual Plane written in the pattern itself, for example an emoji. The text a pattern is applied to may contain them. To match one, use a wildcard or a negated class, which take such a character whole.
  • Repeat counts above the limit, and counts that multiply past it when nested, such as (?:a{100}){11}. The limit is in the limits table.
  • Letter and number escapes other than \d, \D, \w, \W, \s, \S, \b and \B, including the JavaScript escapes \n, \t, \r, \0, \xNN and \uNNNN. To match such a character, write the character itself in the pattern, or use \s for whitespace. A { that does not start a repeat count is written \{; an unescaped one is refused.
  • Spellings that belong to other regular-expression dialects: \Q...\E, (?P<name>...), \pL and inline flags such as (?i).
  • Flags other than i, m and s. The flag i ignores case, m makes ^ and $ match at every line, and s lets . match a line break. The other flags are excluded for a reason:
    • g and y make a compiled pattern remember where it stopped, so the same rule would answer differently the second time. A rule must give one answer for one input.
    • u and v change how classes and escapes are read and open up Unicode property escapes.
    • d adds match positions that no operator uses.

How text is read​

Where a regular-expression dialect leaves a choice open, rule-line fixes it as follows:

  • Digits, word characters and word boundaries are ASCII. \d matches 0 to 9, \w matches A to Z, a to z, 0 to 9 and _, and \b sits between those and anything else. An accented letter is not a word character; write it in a class, such as [A-Za-zÀ-ÿ].
  • Whitespace is wide. \s matches every character JavaScript calls whitespace. That includes the no-break space and the other Unicode spaces, the tab, the line feed and the carriage return.
  • The flag i folds case by Unicode simple case folding. Beyond the ordinary letters, the Kelvin sign matches k, the long s matches s, the Angstrom sign matches å, and the capital sharp s matches ß.
  • Only the line feed ends a line. For ., and for ^ and $ under the flag m, the character \n is the only line break. A carriage return is an ordinary character, so a line that ends with \r\n still matches ^Total.*$. Without m, $ matches only at the very end of the text, so a$ does not match a followed by a line feed.
  • A character outside the Basic Multilingual Plane counts as one character. . and [^a] take an emoji whole. The longest-text limit, however, counts UTF-16 code units, as the limits table says.

When a refusal is reported​

A pattern is refused for the reasons above and for the limits in the limits table. When it is reported depends on where the pattern, and its flags, come from:

  • A pattern and flags written as text in a formula or in a plausibility check are read when the ruleset is saved. Every refusal of them is a warning while the ruleset is a draft and an error when it is activated, so a ruleset with such a pattern cannot go live. That covers a construct outside the language (RE_DISALLOWED_CONSTRUCT), a flag that is not permitted (RE_DISALLOWED_FLAG), a pattern that is too long or too large (RE_PATTERN_TOO_LONG), a pattern that is not valid at all (RE_INVALID_PATTERN), and, in a matches call, a pattern that anchors itself (RE_ANCHOR_IN_WHOLE_MATCH).
  • What can only be refused when a run reaches it is what the saved text cannot show: a flags argument that is not written as text, a pattern read from data (for example from an input field or a dataset row), and the longest-text limit, which concerns the data a pattern is applied to and never the pattern. A pattern read from data is refused with the same codes as above.

Where a refusal appears in a run​

A refusal at run time carries the same code, but where you see it depends on where the pattern sits:

  • In a formula, the step fails with the failure code STEP_EXECUTION_FAILED. The regular-expression code (RE_INPUT_TOO_LONG, RE_DISALLOWED_CONSTRUCT, RE_PATTERN_TOO_LONG or RE_ANCHOR_IN_WHOLE_MATCH) is in the failure's details, not in the failure code itself, so filter on STEP_EXECUTION_FAILED and read the details for the cause.
  • In a plausibility check of an extract step, a refused pattern makes that check fail. The failed check is recorded and lowers the confidence of the extracted value; it does not fail the step.

matches and regexMatch​

regexMatch matches anywhere in the text. matches requires the pattern to match the whole text, from its first character to its last. Because matches supplies that whole-text requirement itself, a pattern that already anchors itself with ^ or $ is refused with RE_ANCHOR_IN_WHOLE_MATCH. When the pattern is written as text, the refusal is reported when the ruleset is saved, as a warning in a draft and an error at activation; a pattern read from data is refused when a run reaches it. Use regexMatch when you want an anchor of your own.