JP2003242179A - Character string collating method, document processing device using the method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make POSIX regular expressions including a partial regular expression and a longest match or a regular expression including a plurality of character collating elements processible by a determinative finite state automaton. <P>SOLUTION: This document processing system is provided with an automaton constructing part 210 constructing an indeterminative finite state automaton from the regular expression of a character string and constructing the determinative finite state automaton based on the indeterminative finite state automaton and an automaton deciding part for performing a matching for character strings by using the determinative finite state automaton. The automaton deciding part 240 specifies a match range of the character strings by using the indeterminative finite state automaton and the determinative finite state automaton, regarding the matched character strings. The automaton deciding part 240 performs a matching for each element of the character string to be processing object, while dynamically determining the state of a transition destination in a state transition of the determinative finite state automaton. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for performing pattern matching (character string matching) by using a finite state automaton, and in particular, pattern matching using a character string regular expression is executed by a deterministic finite state automaton. For technology.

[0002]

2. Description of the Related Art In a text processing tool using a computer, a regular expression is widely used as a pattern matching used for character string search or the like, that is, a mechanism for finding and manipulating a pattern in a character string. For example,
The regular expression "(ab) *" matches zero or more repetitions of "ab"("","ab","abab", ...), and "[ac
d] ”matches“ a ”,“ c ”or“ d ”.
"." Matches any single character, "ab | cd" is "ab"
Or matches either "cd".

When performing processing such as character string search, a Finite State Automaton (Finite State Automaton), which is an evaluator for pattern matching by the regular expression, is used from this regular expression.
omaton) and process this finite state automaton with respect to the string to be searched. Here, a finite state automaton is a nondeterministic finite state automaton (No
n deterministic Finite state Automaton: NFA)
And the deterministic finite state automaton (Deterministic Fini
te state Automaton (DFA) and DFA is NF
It can be created from A.

FIG. 34 shows an example of NFA that matches the regular expression "([ab] c) * ac". In addition, FIG.
It is a DFA equivalent to the NFA. These finite state automata are accepted because they match the regular expression "([ab] c) * ac" when the input is "acac" or "ac", for example.
If the input is “aa”, the regular expression “([ab] c) * a
It is not accepted because it does not match c ”. Generally, although there are multiple NFAs corresponding to a given regular expression, DFA
Is uniquely determined. In actual processing, D is superior in processing speed
It is preferable to use FA. Therefore, for example, in the case of a character string search, the search condition is described by a regular expression, an NFA is created from this regular expression, and the DFA is generated from this NFA.
Is created and the search process is performed by processing this DFA. Pattern matching using a regular expression and a finite state automaton is described in detail in the following documents, for example. Reference: VJ Rayward-Smith. Introduction to Language Theory. Kyoritsu Shuppan, 198
6.Translated by Kenzo Inoue.

[0005]

By the way, UNIX
In the (registered trademark) system, in order to ensure compatibility between operating systems, a standard API (Applicat
POSIX (Portable) which is an ion program interface
Operating System Interface for UNIX), IEEE
Is determined by. The regular expression conforming to POSIX (hereinafter referred to as POSIX regular expression) has functions such as partial regular expression and longest match, which are beyond the scope of the original regular expression. Therefore, it may be difficult to implement the pattern matching function used for character string search or the like in the DFA. In POSIX, there is a longest leftmost rule that matches the leftmost substring in a given string, and each regular expression element matches the longest (longest) string possible. Therefore, for example, if the input “acac” is given to “([ab] c) * ac” in the POSIX regular expression, “([ab] c) *” will match the longest and the partial regular expression “([ab ]
c) ”can be obtained about the matched part (the beginning“ ac ”of the input). This information may be necessary when using a POSIX-compliant function. However, this information can be obtained from the NFA. It is lost when converted to DFA, and it is impossible to identify which element of the regular expression matched with which character, so it cannot be processed by DFA and pattern matching is performed using NFA, which has a slow processing speed. I had to.

Further, the text processing tool has a variable length matching element (hereinafter, plural character matching) such as matching the elements'a ',' b 'and'aa' with the regular expression "[ab]". There are things that can handle elements). FIG. 36 is a diagram showing an example of an NFA corresponding to a regular expression including a multiple character matching element. In this case, the length of the characters to be accepted is different between the case of transition from the first state 0 to the state 1 and the case of transition to the state 2 in FIG. 36. Therefore, the DFA is constructed as it is and pattern matching is performed. I can't. On the other hand, a character string that allows multiple character matching elements (in the above regular expression “[ab]”, the elements'a ',' b ',
It is conceivable to construct a DFA that enumerates (each of'aa ') and make a state transition individually to deal with it. However, an API (Application Program Interface) that enumerates all the character strings that the multiple character matching element matches is a PO.
It is not possible to build such a DFA as it is not provided by SIX. Therefore, in such a case, the pattern matching must be performed using the NFA having a slow processing speed.

Therefore, the present invention makes it possible to process a POSIX regular expression including a partial regular expression and a longest match by a DFA (deterministic finite state automaton), and to realize such a processing device (computer device). With the goal. Another object is to enable a regular expression including a multi-character collating element to be processed by a DFA (deterministic finite state automaton), and to realize such a processing device (computer device).

[0008]

In order to achieve the above object, the present invention is realized as a character string collating method for collating character strings using a computer. That is, this string matching method is a step of creating a nondeterministic finite state automaton from a regular expression of a string, a step of creating a deterministic finite state automaton based on this nondeterministic finite state automaton, and this deterministic finite state automaton. A step of matching the character strings using
With respect to the matched character string, a step of specifying a matching range of the character string by using the nondeterministic finite state automaton and the deterministic finite state automaton.

Here, the step of creating the non-deterministic finite state automaton is, in detail, for a regular expression of a character string, one for each element except elements for designating a certain range in the regular expression. The step of generating the state of the nondeterministic finite state automaton that has been made to correspond to the ε transition for the element that means repetition and the element that means selection, and to the next element for other elements Corresponding to the transition to the state.
Further, the step of identifying the match range is specifically based on the step of deleting an unnecessary state sequence that does not reach the end state among the state sequences showing the state transition by the deterministic finite state automaton, and the remaining state sequence. And specifying the matching range of the character string. Furthermore, more preferably, the step of identifying the match range is a step of identifying the match range of the character string based on the state sequence that satisfies the longest leftmost rule among the state sequences showing the state transition by the deterministic finite state automaton. including.

Further, the present invention is realized as another character string collating method as follows. That is, this character string matching method reads a deterministic finite state automaton created based on a regular expression of a predetermined character string from a memory, performs character string matching using this deterministic finite state automaton, and stores the processing result in a memory. Based on the narrowed state sequence and the second step of narrowing down the state sequence showing the state transition by this deterministic finite state automaton to the state sequence that can reach the end state. And a third step of identifying which part of the regular expression matches which part of the regular expression. Here, in the third step, more preferably, of the narrowed-down state strings, the state string having the largest number of repetitions that appears first is selected, and the character string to be processed is selected based on this state string. And determining which part of the regular expression each character in and matched.

Further, the present invention is realized as the following other character string collating method. That is, this character string matching method reads a deterministic finite state automaton created based on a regular expression of a predetermined character string from a memory, performs character string matching using this deterministic finite state automaton, and stores the processing result in a memory. And a second step of restoring a second state sequence showing the state transition in the non-deterministic finite state automaton based on the first step of storing the state transition in the deterministic finite state automaton And a third step of obtaining information about which character in the string matched which part of the regular expression based on the restored second state sequence. Here, this second step more preferably includes a step of restoring a state sequence satisfying the longest leftmost rule as the second state sequence.

Furthermore, the present invention is implemented as another character string collating method as follows. That is, this string matching method is a step of creating a nondeterministic finite state automaton from a regular expression of a character string, a step of creating a deterministic finite state automaton based on this nondeterministic finite state automaton, and a character to be processed. The step of performing matching while dynamically determining the transition destination state in the state transition of this deterministic finite state automaton for each element of the sequence.

More specifically, in this matching step, the character string to be processed is pre-read to determine whether or not the character string includes a character string that can correspond to a multi-character collating element. If the determination step and this character string include a character string that can correspond to a multi-character collating element, reflect the state transition when this character string is a multi-character collating element to reflect the state of the transition destination. Dynamically determining. Further, more preferably, in this string matching method, after performing matching, using a nondeterministic finite state automaton and a deterministic finite state automaton, information about the matching range of this string is obtained. The configuration further includes steps.

Further, the present invention is implemented as another character string collating method as follows. That is, this string matching method is a step of creating a nondeterministic finite state automaton from a regular expression of a character string, a step of creating a deterministic finite state automaton based on this nondeterministic finite state automaton, and a character to be processed. When a string is read ahead and a string that can be a multi-character matching element is included in this string, a state transition of a deterministic finite state automaton is generated for each element of the string and A step of virtually generating a state transition corresponding to a character string that can correspond to a multi-character collating element, and performing matching based on these state transitions.

Another object of the present invention to achieve the above object is to provide a non-deterministic finite state automaton for constructing a non-deterministic finite state automaton from a regular expression of a character string in a document processing device for searching a character string using a regular expression. State automaton construction means, deterministic finite state automaton construction means for constructing deterministic finite state automaton based on nondeterministic finite state automaton constructed by this nondeterministic finite state automaton construction means, and this deterministic finite state automaton construction means And a determining means for matching the character strings by using the deterministic finite state automaton constructed as above. The determination means is characterized in that the matching range of the character string is specified by using a nondeterministic finite state automaton and a deterministic finite state automaton for the matched character string.

Here, the nondeterministic finite state automaton constructing means corresponds to the regular expression of the character string one by one to each element other than the element designating a certain range in the regular expression. Generates the state of the state automaton, and makes the ε transition correspond to the element that means repetition and the element that means selection, and the transition to the state associated with the next element to other elements. It is characterized by Also preferably, the determination means is a state sequence narrowing means for deleting and narrowing down unnecessary state sequences that do not reach the end state among the state sequences showing the state transition by the deterministic finite state automaton, and the state sequence narrowing means. The match range determination means specifies the match range of the character string based on the narrowed-down state string.

The present invention can also be implemented as another document processing apparatus as follows. That is, in a document processing device for searching a character string using a regular expression, a nondeterministic finite state automaton constructing means for constructing a nondeterministic finite state automaton from a regular expression for a character string and a nondeterministic finite state automaton constructing means are provided. The deterministic finite state automaton construction means for constructing a deterministic finite state automaton based on the nondeterministic finite state automaton constructed by the above and the deterministic finite state automaton constructed by this deterministic finite state automaton construction means are processing targets. It is characterized by further comprising: determination means for performing matching while dynamically determining a transition destination state in the state transition of the deterministic finite state automaton for each element of the character string.

More preferably, the determining means pre-reads the character string to be processed and determines whether or not the character string includes a character string that can be a multi-character collating element. When it is determined that the character string that can be applied to the multi-character collating element is included, the state transition of the transition destination is dynamically determined by reflecting the state transition when the character string is the multi-character collating element. Further, this determination means deletes unnecessary state sequences that do not reach the end state from the state sequence showing the state transition by the deterministic finite state automaton and narrows down the state sequence narrowing down means, and the state sequence narrowing down means narrows down. And a match range determination unit that specifies the match range of the character string based on the state sequence.

Further, the present invention is a program for controlling a computer to execute the above-mentioned character string collating method,
Alternatively, it can be realized as a program that causes a computer to function as the above document processing device. Such a program can be provided by being stored in a magnetic disk, an optical disk, a semiconductor memory, or another recording medium for distribution, or distributed via a network.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION First and second embodiments shown in the accompanying drawings
The present invention will be described in detail based on the embodiments.
In the description of the embodiments below, the regular expression is
POSIX regular expression (regular expression based on POSIX)
Means [First Embodiment] In the first embodiment, a computer realizes a document processing system for performing a character string search using a DFA (deterministic finite state automaton) based on a search condition expressed by a regular expression. To do. This system
The state of transition of the DFA when the DFA is created is recorded. Then, from the sequence of recorded states, the corresponding NF
By obtaining the sequence of states to follow when using A (nondeterministic finite state automaton), the necessary information lost in the conversion to DFA is obtained. As a result, it is possible to perform high-speed pattern matching by DFA even for the partial regular expression used in the regular expression and the longest match.

FIG. 1 is a diagram schematically showing an example of a hardware configuration of a computer device suitable for realizing the document processing system according to the first embodiment. The computer device shown in FIG. 1 is a CPU (Centra) which is a calculation means.
l Processing Unit: central processing unit) 101, M / B
(Motherboard) Main memory 103 connected to CPU 101 via chipset 102 and CPU bus
And M / B chipset 102 and AGP (Acce
a video card 104 connected to the CPU 101 via an integrated graphics port (PCI) and a PCI (Peripheral Com
A hard disk 105 and a network interface 106 connected to the M / B chipset 102 via a ponent interconnect (BUS), and a bridge circuit 107 and an ISA (Industry Stand) from the PCI bus.
ard Architecture) M / via a low-speed bus such as a bus
Floppy (registered trademark) disk drive 108 and keyboard / mouse 1 connected to B chipset 102
09 and. Although not shown in FIG. 1, the computer device includes a clock oscillator and a controller thereof as means for controlling the operating performance (operating clock) of the CPU 101, as will be described later. Note that FIG.
Only exemplifies the hardware configuration of the computer device that realizes the present embodiment, and various other configurations can be taken as long as the present embodiment is applicable. For example, instead of providing the video card 104, only a video memory may be mounted and the CPU 101 may process image data, a sound mechanism for inputting and outputting by voice, or an ATA (AT Attachmen).
CD-ROM (Com
pact Disc Read Only Memory) and DVD-ROM (Digi
A tal Versatile Disc Read Only Memory) drive may be provided.

FIG. 2 is a block diagram illustrating the configuration of the document processing system according to the first embodiment. Referring to FIG. 2, the document processing system 200 according to the present embodiment is
An automaton building unit 210 that creates a DFA that performs pattern matching of character strings, an automaton holding unit 220 that holds the created DFA, a document holding unit 230 that holds the document data that is the search target, and a character using the DFA. An automaton determination unit 240 that executes pattern matching in the column search. Further, the document processing unit 250 for executing processing other than the character string search according to the present embodiment, and the processing program holding unit 26 for holding processing programs for realizing these functions in the computer device.
With 0 and. As shown in the figure, the document processing system 2
A user 00 (document user) is connected to an input / output device 300 for outputting a processing result while a user (document user) inputs a search key or a command for performing document data to be searched and various kinds of processing.

In FIG. 2, the automaton holding unit 22
0, the document holding unit 230 and the processing program holding unit 260
Are realized in the main memory 103. The data held in the main memory 103 can be saved in a storage device such as the hard disk 105 as needed. The automaton construction unit 210, the automaton determination unit 240, and the document processing unit 250 are software blocks implemented by the CPU 101 controlled by the processing program stored in the processing program holding unit 260. A processing program that controls the CPU 101 to realize these functions is provided by being stored in a magnetic disk, an optical disk, a semiconductor memory, or another recording medium for distribution, or distributed via a network.

FIG. 3 is a flow chart showing a schematic flow of the DFA construction processing by the automaton construction unit 210. As shown in FIG. 3, the automaton building unit 210
Upon receiving an input of a regular expression of a character string serving as a search key (step 301), first constructs a syntax tree from the regular expression (step 302). The constructed syntax tree is
It is stored in the main memory 103 or a cache memory (not shown) of the CPU 101. Next, the automaton construction unit 2
10 reads the syntax tree from the main memory 103,
Build an NFA based on the syntax tree (step 30).
3). The constructed NFA is the main memory 103 and CP.
It is stored in a cache memory (not shown) of U101.
Furthermore, the automaton building unit 210 uses the main memory 10
The NFA is read from the third grade and the DF is read based on the NFA.
Construct A (step 304). Constructed DFA
Are stored in the main memory 103 or a cache memory (not shown) of the CPU 101. That is, the automaton building unit 210 functions as a syntax tree building unit, an NFA building unit, and a DFA building unit in order to create the DFA used in this embodiment. The constructed NFA and DFA are stored in the automaton holding unit 220 realized in the main memory 103.

The syntax tree (binary tree) constructed from the regular expressions in step 301 may have any configuration as long as it includes information for creating an appropriate NFA described later. Can be any one of the following four: 1. Operators excluding the elements'(',')'that specify a certain range (partial regular expression) in the regular expression (one operator corresponds to one element of the regular expression) In this node,' [abc] 'Is regarded as one operator,
There is also one corresponding node. Also, '*' has one child (a subtree corresponding to <re> of “<re> *”. A subtree is a node in the syntax tree and its descendants (child, grandchild, ...
-) Is a tree that contains all of the) and corresponds to a part of the regular expression). In addition, '| (<ALT>)' has two children (“<re1> |
(A subtree corresponding to <re1> and <re2> of <re2> ”. Other operators (“ a ”,“ [ac] ”, etc.) have no children.) 2. <TERM> Regular expression end Virtual operator that indicates (in the case of regular expression "abc", consider that there is a virtual terminal character after'c ', and let this be <TERM>) This node has no children. <CONCAT> Virtual node representing the concatenation of regular expressions (there is no corresponding NFA state and operator) For example, the regular expression "abc" is the concatenation of elements'a 'and'b', and It is a connection with the element'c ', and this "connection" is one node. In addition, this node has two children (subtrees corresponding to two concatenated partial regular expressions). 4. <SUBEXP> Represents a partial regular expression enclosed in parentheses (there is no corresponding NFA state. Corresponds to the pair of '(', ')' in the regular expression) This node has one child ('(', Subtree corresponding to the subexpression in ')').

FIG. 4 is a diagram showing a syntax tree constructed from the regular expression "ab | c (de) *". In FIG. 4, for the regular expression “[ab] | c (de) *”, “[ab]”, “|”, “c”,
"D", "e", and "*" are operators, and corresponding nodes are generated. Also, node <C> is CONCA
T, node | for ALT, node <S> for SUBEXP, node <T>
Is TERM. A known technique can be used for the process of constructing the syntax tree. FIG. 5 is an example of a program for executing the process of constructing a syntax tree.

N constructed from the syntax tree in step 302
FA has the conditions for use in the present embodiment.
That is, it is possible to restore from the state sequence of DFA, and to follow the longest leftmost rule of the regular expression. As described above, multiple NFAs can be created from a given regular expression.
However, in the present embodiment, in the pattern matching by DFA, since the information of the sequence that is traced when NFA is used, it is possible to construct an NFA suitable for this purpose, that is, satisfying the above conditions. Required (hereinafter, this NFA is referred to as a proper NFA). This suitable NFA has the following characteristics. 1. All elements except "(", ")"("a","[a-
c] ”,“ * ”, etc.) corresponds to one NFA state 2. ε transition (transition without accepting characters) is repeated (*, +,?, etc.) and selected (|) 3. Only one transition destination (except ε transition).

In order to create such an appropriate NFA, in this embodiment, three types of functions, first, epsdest and next, are defined. Each function will be described below. firs
t: Returns the NFA state corresponding to the operator that "appears" first in the subregular expression to which a predetermined subtree (= subtree whose root is the node of the argument) in the syntax tree corresponds. Here, the operator that “appears” first means that it must be processed first. Also, NF
The A state is indicated by one circle (node) in the NFA. In the present embodiment, the NFA state corresponds to the operator,
Every NFA state has one corresponding node (the opposite is not true, and some nodes do not have a corresponding NFA state). In the prior art, a set of operators is returned in the function first defined for constructing NFA / DFA, whereas in the present embodiment, one operator (= NF
Only one A state) is returned. From the above definition, the following holds. first (<char>) ＝ <char> (character) first (<re1><re2>) ＝ first (<re1>) (concatenation) first (<re1> | <re2>) ＝ | (selection) first (<re1> *) = * (repetition) first ((<re>)) = first (<re>) (partial regular expression enclosed in parentheses) FIG. 6 is a diagram illustrating a definition code of the function first.

Epsdest: NFA corresponding to the argument node
Returns a set of NFA states that can undergo ε transition from a state. Here, it is only the "iteration operator" and the "selection operator" that epsdest returns a non-empty set, that is, ε transitions are allowed. From this definition, the following holds. epsdest (*) ＝ first (<re>) ∪next (<re>) (however, "*" is surrounded by "<re>*") epsdest (|) ＝ first (<re1>) ∪first (<re2 >) (However, "<re1> | <re2>" is set around |) epsdest (<op>) ＝ φ (However, <op> is an operator "a", "[other than * and | a
b] ”etc.) FIG. 7 is a diagram illustrating a definition code of the function epsdest.

Next: N corresponding to the operator that appears next to the partial regular expression to which the subtree whose root is the node of the argument corresponds
Returns FA status. That is, the function next indicates the transition destination of the NFA state to which the node of the argument corresponds. Also, the function firs
Similar to t, since the value to be returned is not a set of operators but only one operator, the transition destination of each NFA state except “*” and “|” is uniquely determined. Characters accepted at this time are characters that the operator corresponding to the node matches. From the above definition, the following holds. Note that next (<re1>)
Is defined according to the surroundings of <re1> in the regular expression. When the circumference of <re1> is <re1><re2>, next (<re1>) ＝ first (<re2>) (concatenation) When the circumference of <re1> is <re1> | <re2>, next (<re1>) ＝ next (<re1> | <re2>) (selection) When the circumference of <re1> is <re1> *, next (<re1>) ＝ * (repeated) The circumference of <re1> is When (<re1>), next (<re1>) ＝ next ((<re1>)) (partial regular expression enclosed in parentheses) <re1> represents the entire regular expression, that is, around <re1> Is <re1>, next (<re1>) = <TERM> (the outermost regular expression) FIG. 8 is a diagram illustrating the definition code of the function next.

FIG. 9 shows the appropriate NF using the function described above.
8 is a flowchart illustrating a process of constructing A. In the following operation, the NFA start state is the first (= operator to be processed first in the regular expression) of the syntax tree. In addition, to the NFA state, an operator excluding '(', ')' in the regular expression and one NFA state are assigned to <TERM>. Also, for a given NFA state ε
The transition destination is epsdest of the node corresponding to the NFA state.
(By definition) Here, epsdest is not empty only in the case of '*' or '|', and ε transitionable states are also only '*' and '|'. Furthermore, a predetermined NFA state transition destination does not have a normal transition destination if the NFA state has an ε transition destination. On the other hand, when there is no ε transition destination, it is the next of the node corresponding to the NFA state. At this time, the accepted character is the NFA.
The state is a character that the corresponding operator matches. For example, the operator “a” matches the character “a” and the operator “[a
bc] ”matches any of the characters“ a ”,“ b ”, or“ c. ”The NFA end status is <TERM>(<T
All NFA states except ERM> have “two ε transition destinations” or “one normal transition destination”).

Referring to FIG. 9, the automaton construction unit 2
The 10 first generates [nfa_st_num] NFA states (step 901). Here, nfa_st_num is the number of NFA states generated from the regular expression by this method, and is the number of elements in the regular expression excluding '(', ')' + 1 (for <TERM>). Next, the start state (nfa_init) of the NFA is the first node (first (tree)) of the syntax tree, that is, the operator to be processed first in the regular expression. In addition, the NFA stop state (nfa_halt) indicates the NFA state number (nfa_st_num-
1) (step 902).

Next, the variable i is initialized (i = 0), and i
The reference (state_trees [i]) to the node on the syntax tree corresponding to the nth NFA state (state i) is checked (step 90).
3, 904). When the operator that is the reference to this node is '*' or '|', epsdest (state_trees [i]) obtained by the function epsdest is set as the ε transition destination of the state i (step 905). On the other hand, when the operator that is the reference to the node on the syntax tree corresponding to the state i is other than '*' and '|', the reference to the node (state_trees
[i]) represents the transition destination of the state i for the element represented by the function n
Next (state_trees [i]) obtained by ext is set (step 906).

Thereafter, the value of the variable i is incremented by 1 (step 907), and it is checked whether or not the obtained new variable i has reached the number of NFA states (step 908). Then, if the value of the new variable i is less than the number of NFA states, the process of step 904 is executed using the new variable i. On the other hand, if the value of the new variable i reaches the number of NFA states, the NFA stop state is reached and the appropriate NFA is reached.
Has been created, the process ends. The generated appropriate NFA is stored in the automaton holding unit 220 because it is used later for the processing by the automaton determination unit 240.

Step 303: From the appropriate NFA to DFA
A known method can be used for the process of constructing.
That is, it is a process of sequentially collecting from the start state to the final state the work of collecting one piece of information about how the state transitions when one character is received in each state of the original NFA and creating one new state. D created as above
The FA is stored in the automaton holding unit 220 realized by the main memory 103 and the like, and the automaton determination unit 240
Used for processing by.

FIG. 10 is a flow chart showing a schematic flow of the pattern matching processing by the automaton judging section 240. As illustrated in FIG. 10, the automaton determination unit 240 reads and inputs the DFA stored in the automaton holding unit 220 and the search target character string (document data) stored in the document holding unit 230 ( First, in step 1001), the DFA determines (matches) the input character string (step 1002). The determination result is stored in the main memory 103 or a cache memory (not shown) of the CPU 101. Next, the automaton determination unit 240, if the input character string is accepted, reads the NFA read from the automaton holding unit 220.
And DFA to narrow down the DFA state sequence and match range (predetermined part of regular expression (partial regular expression)
Part of the character string that matches (substring range)
Is performed (steps 1003 to 1005). The processing result is stored in the main memory 103 or a cache memory (not shown) of the CPU 101. Finally, the automaton determination unit 240 reads the determination result of step 1002 and the processing results of steps 1004 and 1005 from the main memory 103 or the like, and outputs them via an output device such as a display device (step 1006). If the input character string is not accepted by the DFA, the result of rejection is output (steps 1003, 1).
006). The above processing is executed for all input character strings (step 1007). That is, the automaton determining unit 240 functions as an input character string determining unit, a DFA state sequence narrowing unit, and a match range determining unit.

A known method can be used for the character string matching processing in step 1002. Here, even if the input character string matches the DFA and is accepted, it is not possible to obtain information about the matched portion of the partial regular expression in the input character string. That is, since the DFA state sequence includes all NFA state sequences that can be traced to the input, it is impossible to identify which element of the regular expression matches which character, and the POSIX
I do not know the state transition that conforms to the longest leftmost rule in. Therefore, the automaton determination unit 240 of the present embodiment
Acquires the information of the longest NFA state sequence candidate by narrowing down the DFA and determining the matching range by restoring the leftmost NFA state sequence.

In the narrowing down of the DFA state sequence in step 1004, an unnecessary NFA state sequence that cannot reach the end state is selected from the DFA state sequence that includes all NFA states that can be reached for the input. To delete. This narrowing down of the DFA status row is performed by the NFA that constitutes the DFA.
By tracing the DFA's state sequence from the back based on (appropriate NFA). FIG. 11 is a flow chart for explaining the operation in narrowing down the DFA state sequence.
As shown in FIG. 11, the automaton determination unit 240 first determines the state sequence (stat) of the DFA determined in step 1002.
e_log) and the input string index (idx) are initialized. Specifically, the state of the DFA (state_log [last]) for the last stop state is set to the stop state of the NFA (nfa_hal).
The DFA state includes only t). Also, the index (id of the input string currently being processed (during narrowing))
x) is the last stop state (las
The state (last-1) immediately before t) is set (step 110).
1). For example, the state sequence (state_log) of DFA is {{0,1,3}, {2,3,4}, {3,5},
{4}} and when the NFA stop status is 5, DFA
Only {3, 5} is the stopped state in the state sequence of. At this time, the value of the last stop state (last) is 2 (because counting from 0). That is, state_log [last] = [nfa_st_nu
m] is state_log [2] = [5] in this example, and as a result, the state sequence of DFA is {{0,1,3}, {2,3,
4}, {5}, {4}}.

Next, the automaton determination unit 240 determines the state string (state_lo) of the DFA for the input character string currently being processed.
g [idx]) is updated (step 1102). Then, the index of the input character string to be processed is the state (idx = idx-idx = idx-idx = idx-
1) (step 1103), if there is an unprocessed input character string, the process returns to step 1102 to execute the processing, and if all the input character strings have been processed, the DFA
Ends the process of narrowing down the state column (step 110).
4).

FIG. 12 is a flow chart for explaining the details of the processing for updating the state string (state_log [idx]) of the DFA in step 1102. Referring to FIG. 12, the automaton determination unit 240 first initializes the memory area (state_buf) used in this process, and sets the value of the index (nfa_idx) to the NFA state referenced in this process to 0 (step 1201). . Then nfa_idx and state_log [i
The number of elements is compared with dx] (step 1202). When the number of elements of state_log [idx] is larger, the NFA state (nfa_st) being referred to is changed to state_log [idx] [nfa_id
x], and the pointer (node) to the node on the syntax tree corresponding to the NFA state in the memory area is state_trees.
[nfa_st] (step 1203). Then, it is checked whether the character string representation (node.label) of the node in the syntax tree is '*' or '|'. If it is either "*" or "|", the value of nfa_idx is incremented by 1 and the process returns to step 1202 (steps 1204, 1208).

When the character string representation (node.label) of the node is neither "*" nor "|", the automaton determination unit 240 next determines ne for the function value next (node).
It is checked whether xt (node) εstate_log [idx + 1] holds (steps 1204 and 1205). If this is not the case, add 1 to the value of nfa_idx and step 1202
Return to (step 1208). On the other hand, next (node) ∈ stat
When e_log [idx + 1] holds, the automaton determination unit 24
0 then checks whether the character string representation of the node accepts the input character (input (idx)) corresponding to the index currently being processed (step 1206). If not accepted, add 1 to the value of nfa_idx and step 1
Return to 202 (step 1208). Furthermore, when the character string representation of the node accepts the character string input (idx), the automaton determination unit 240 next stores n in the memory area state_buf.
fa_st is added (step 1207). And nfa_i
The value of dx is incremented by 1, and the process returns to step 1202 (step 1208).

If the value of nfa_idx reaches the number of elements of state_log [idx] by repeating the processing of steps 1202 to 1208 and adding the NFA state stored in state_buf, then the automaton determination unit 240 , Ε transition source is added (step 1209). The process for adding the ε transition source will be described later. After this, the automaton determination unit 240 updates state_log [idx] to the processing result accumulated in state_buf (step 1210).

FIG. 13 is a flowchart for explaining the details of the ε transition source addition processing. Referring to FIG. 13, the automaton determination unit 240 first sets the value of the index (nfa_idx) to the NFA state referred to in this processing to 0 (step 1301). Then nfa_idx and state_log
The number of elements is compared with [idx] (step 1302). Then, when the number of elements of state_log [idx] is larger, the NFA state (nfa_st) being referred to is changed to state_log [idx] [nfa_i
dx], and the pointer (node) to the node on the syntax tree corresponding to the NFA state in the memory area is state_tree.
Let s [nfa_st] (step 1303). And the string representation of the node in the syntax tree (node.label) is '*'
Or check for '|'. If it is neither "*" nor "|", the value of nfa_idx is incremented by 1 and the process returns to step 1302 (steps 1304 and 1307).

When the character string representation (node.label) of the node is "*" or "|", the automaton determination unit 2
40, next, for the function value epsdest (node), epsdest
It is checked whether (node) εstate_buf holds (steps 1304 and 1305). If this is not the case, the value of nfa_idx is incremented by 1 and the process returns to step 1302 (step 1307). On the other hand, epsdest (node) ∈ state_b
When uf is satisfied, the automaton determination unit 240 next adds nfa_st to the memory area state_buf and returns to step 1301 (step 1306).

The NFA state stored in state_buf is added by repeating the above processing, and the value of nfa_idx is state.
When the number of elements of _log [idx] is reached, the addition process of the ε transition source is ended. This updates the DFA state string (state_log [idx]) for the input string currently being processed,
The process of narrowing down the DFA status column is completed.

Next, a concrete operation example of the above-mentioned narrowing-down process of the DFA state sequence will be described. Here, the case where the matching of the input character string “aaaa” is checked against the regular expression “((a *) a) * a” is taken as an example. FIG. 14 is a diagram showing an appropriate NFA of the regular expression “((a *) a) * a”. 14
The NFA shown in 6 has six states of 0, 1, 2, 3, 4, and 5. FIG. 15 is a DF of the NFA shown in FIG.
It is a figure which shows the state transition with respect to A and the character string "aaaa." Referring to the DFA of FIG. 15, the input character string “aaaa”
Is accepted, the state transition in DFA is {0,1,2,3,4} as the start state, and all DFA states for the input character string after the first character are {0,1,2,3,4.
4, 5}. Therefore, according to the longest leftmost rule,
The first two'a's in the input string "aaaa" partially match the'a * 'in the regular expression "((a *) a) * a". No information is available.

FIG. 16 shows N for the input character string "aaaa".
It is a figure which shows the mode of transition of FA state. FIG. 16A shows a list of the ε transition destinations and the characters of the NFA states and the NFA states of the transition destinations for the characters from each NFA state. In addition, in FIG.
The state transition path is shown by the state string of the DFA narrowed down based on (A).

With reference to the NFA of FIG. 14 and the transition table of FIG. 16 (A), consider narrowing down the state sequence of the DFA. First, since the rightmost column in FIG. 16A is the NFA end state, the NFA state 5 based on the NFA in FIG.
Must. Next, the NFA state that can reach NFA state 5 by entering one letter'a 'is NFA
Only state 4 and NFA state 3 that makes an epsilon transition to NFA state 4. Then enter the letter'a 'to enter NFA
NFA states that can reach state 4 or NFA state 3 are:
NFA state 2 and NFA state 1 that makes an epsilon transition to the NFA state
Then, only the NFA state 3 that makes an epsilon transition to the NFA state 1. Similarly, enter one letter'a 'to get NF
The NFA state that transits to A state 1, 2 or 3 is NFA.
It can be seen that it is in states 0, 1, 2 or 3. further,
It can be seen that the NFA state (corresponding to the start state) that transitions to the NFA state 1, 2 or 3 by inputting one character'a 'is the NFA state 0, 1, 2 or 3. In FIG. 16 (A), extraction is performed in this way,
That is, the narrowed-down state row of the DFA is shown in italics.

FIG. 16B shows a possible route from the NFA state 3 at the start state to the NFA state 5 at the end state regarding the state sequence of the DFA narrowed down as described above. In the figure, the ε transition is shown as a vertical path, and the transition for the character'a 'is shown as a horizontal path. Further, the notation (1) of going to the NFA state 0 means that the NFA state 0 transits from the NFA state 1 similarly to the NFA state 2.

Next, the matching range determination processing of step 1005 in the processing by the automaton determination section 240 (see FIG. 10) will be described. In this processing, the matching range of the character string is determined based on the state string of the DFA narrowed down in step 1004, using the information about the state transition obtained by an appropriate NFA. This makes the PO
Information such as matching or longest match of the partial regular expression according to the longest leftmost rule of SIX is obtained.

FIG. 17 is a flow chart for explaining the operation in the match range determination process. As shown in FIG. 17, the automaton determination unit 240 first initializes various parameters stored in the register (step 17).
01). Specifically, it refers to the NFA status (nf
a_state) as an appropriate NFA start state (nfa_init) input from the automaton holding unit 220, and data matches [i] = (-1,-
1) (0 ≦ i ≦ MAX) is set and the index (idx) currently referred to in the character string to be processed is idx = matc
Let hes [0] .first = 0.

Then, the automaton determining section 240 performs a register updating process (step 1702) and a transition process from the currently referred NFA state to the next NFA state (step 1703). These processes will be described later. Next, the automaton determination unit 240 determines whether the NFA state (nfa_state) is the stop state (nfa_halt) (step 1704). And nfa_state = nfa_h
If it is not alt, the process returns to step 1702 to repeat the process. If nfa_state = nfa_halt, the match range determination process is ended.

FIG. 18 is a flow chart for explaining the details of the register updating process. Referring to FIG. 18, the automaton determination unit 240 first initializes a variable i (i =
0) (step 1801), NFA status (nfa_stat
whether e) is in the range of the subexpression (subexps
[i] .first ≦ nfa_state <subexps [i] .last) is determined (step 1802). When the NFA state is not included in this range, the data matches [i] .last is set to the value of the index (idx) (since i = 0 in the initial state, FIG.
From step 1701 of [matches [0] .last = matches [0].
first = 0) (step 1803). Then, the value of the variable i is incremented by 1 (step 1806), and if the value of the variable i has not reached MAX set in step 1701 of FIG. 17, the process returns to step 1802 (step 1).
807).

In step 1802, if the NFA state is included in the range of the partial regular expression, the automaton determination unit 240 then determines whether matches [i] .first = -1 or
It is checked whether muches [i] .last ≠ -1 holds (step 1804). If this relationship holds, the data
Matches [i] .first is set as an index (idx) value, and data matches [i] .last is set as -1 (step 18).
05). Then, the value of the variable i is incremented by 1 (step 18
06), if the value of the variable i has not reached MAX set in step 1701 of FIG. 17, the process returns to step 1802 (step 1807). Also, in step 1804, matc
If hes [i] .first = -1 or muches [i] .last ≠ -1 does not hold, no processing is performed on the data and the value of the variable i is incremented by 1 (step 1806), If the value of the variable i has not reached MAX set in step 1701 of FIG. 17, the process returns to step 1802 (step 1
807). As described above, steps 1802 to 180
When the value of the variable i reaches MAX, the process of 7 is repeated, and the updating process of the register is ended.

FIG. 19 is a flow chart for explaining the details of the transition processing to the next NFA state. Referring to FIG. 19, the automaton determination unit 240 first sets a pointer (node) to a node on the syntax tree in the memory area corresponding to the currently referred NFA state to state_trees [nfa_st.
ate] (step 1901). Then, it is checked whether the character string representation (node.label) of the node in the syntax tree is '*' or '|'. If it is '*' or '|', the state with the smallest number in the set epsdest (node) is the NFA state (nfa_state) (step 190).
2, 1903). On the other hand, when the character string representation (node.label) of the node in the syntax tree is neither '*' nor '|', the NFA state (nfa_state) is set to the next node (nex).
The process ends as t (node)) (step 1902,
1904).

Next, a specific operation example of the above-described match range determination processing will be described. Here, FIGS.
As an example, a case of checking the matching of the input character string “aaaa” with the regular expression “((a *) a) * a” described with reference to FIG. As shown in FIG. 16, possible state sequences from the appropriate NFA start state to the appropriate NFA end state are extracted by the DFA state sequence narrowing processing in step 1004 of FIG. 20 is based on the transition table (FIG. 16 (A)) shown in FIG. 16 and the state transition path (FIG. 16 (B)) based on the narrowed DFA state sequence.
It is a figure which shows a mode that a match range is determined. Note that FIG.
In the transition table of 0 (A), NFA states deleted by narrowing down the DFA state string are described by parenthesized numbers.

Possible DF with reference to FIGS.
If you follow the A state sequence, first you will see NF from the start state 3 of the NFA.
It is possible to make a transition to A state 1 and then to NFA state 0 or NFA state 2. However, according to the longest leftmost rule of POSIX, as many iterations that appear first as possible are selected, so that the transition destination is the NFA state 0. continue,
It is possible to make a transition from NFA state 0 to NFA state 1 and then to NFA state 0 or NFA state 2. Again, according to the longest leftmost rule, NFA state 0 is selected as the transition destination. After this, NFA state 0 to NFA state 1, N
The FA state 2, the NFA state 3, the NFA state 4, and the NFA state 5 are transited to reach the end state. In FIGS. 20A and 20B, the DFA status string thus selected is shown in italics.

As described above, the regular expression "((a *) a) *
The NFA state sequence to be traced when the input character string “aaaa” is matched with “a” is restored. FIG. 21 is a diagram showing the restored NFA state sequence. 21 is a transition for a character, and the arrow without the character'a 'is an ε transition.
The first two letters'a 'of aa ”correspond to the transition from NFA state 0 to NFA state 1, and the inside of the double brackets in the regular expression“ ((a *) a) * a ” You can see that it matches'a * '. Similarly, the third character'a' matches the'a 'in the outer parentheses in the regular expression, and the last character'a' in the regular expression. You can see that it matches the outermost'a '.

In this way, it is possible to obtain information as to which character of the input character string matches which part of the regular expression. When the above-described pattern matching is performed using the NFA, the time required for processing is about O (2 ⁿ ), which is a time proportional to 2 n raised to the input n. On the other hand, in the present embodiment,
The time required for pattern matching using DFA is O
It is about (n), and the time required for narrowing down the DFA state and determining the match range is about O (n), and both are completed in a time proportional to n with respect to the input n. Therefore, in the present embodiment, the time required for the processing can be significantly shortened as compared with the processing using NFA.

Here, an example in which the present embodiment is used for a character string search in a specific text file will be described. Suppose you have a text file that contains an address book. In this text file, each item such as the name is separated by ',', and the first item is the name.
However, it is not unified what kind of items are included in one entry. Now, assume that the following entry is lined up in the text file of this address book. Taro Nippon, taro @ yamato.ibm.com, 046-xxx-xxxx, Yamato East Yamato XXX, Kanagawa XXX Yamato Jiro, jiro@jp.ibm.com, Yamato Shimotsuruma XXX, Kanagawa XXX From this text file, Address is "Yamato City, Kanagawa Prefecture"
Consider enumerating the names and email addresses of people whose email addresses contain the string ibm.com. In such a case, it is possible to easily search by using a regular expression. For example, you can search with the following regular expressions. ([^,] *), ([^,] *,) * ([^,] * @ [^,] * ibm.com), ([^,] *,) * Yamato, Kanagawa This regular expression In, [^,] matches any character except ',', and * is zero or more repetitions. Therefore,
[^,] * Matches one item in the text file entry. For example, the first ([^,] *), matches the name and the ',' immediately following it, and the part in parentheses matches the name. Further, ([^,] *,) * matches a predetermined item and ',' repeated zero or more times, that is, any number of items between the name and the mail address. In addition, contains the character @ and the string ibm.com ([^,] * @
[^,] * ibm.com) matches email addresses. After that, 0 or more items are matched, and finally, "Yamato City, Kanagawa Prefecture" is matched.

When this is processed by the conventional DFA, it is not known where the part enclosed by the parentheses '('')' in the regular expression matches the original sentence, so that a specific item such as a name or mail address is specified. I can't pull it out. on the other hand,
In NFA, it takes exponential time of the input length to execute. However, in the first embodiment, the character string can be collated in a short time by the pattern matching using the DFA, and the DFA status string and the NFA status string can be narrowed down to the matched character string. By determining the matching range by restoration, it is possible to obtain information as to which item matches which part of the regular expression, and thus it is possible to extract the desired item.

[Second Embodiment] The second embodiment uses a DFA (deterministic finite state automaton) based on a search condition expressed by a regular expression, as in the first embodiment described above. A computer realizes a document processing system that searches for character strings. Then, the plural character collating element is allowed for the character string serving as the search key. This system pre-reads an input character string when performing pattern matching of a character string using DFA, and determines whether or not a multi-character matching element is included. Also,
When determining whether the input character strings match (whether the DFA accepts them), the state of the transition destination in the state transition of the DFA is dynamically determined and evaluated while being dynamically generated.

The document processing system according to the second embodiment is realized by the same computer device as the document processing system according to the first embodiment illustrated in FIG. Further, the document processing system according to the first embodiment shown in FIG. 2 has the same system configuration. Therefore, in the present embodiment, each component will be described with the reference numerals used in FIGS. 1 and 2, and the hardware configuration of the computer device that realizes the document processing system and the functional configuration of the system will be described. Is omitted.

Further, in the second embodiment, the operation of the automaton building unit 210 is the same as the operation of the automaton building unit 210 in the first embodiment described with reference to FIGS. 3 to 9. . Therefore, in the present embodiment, description of the operation of the automaton construction unit 210 will be omitted.

The automaton determining unit 240 in the second embodiment operates roughly similarly to the automaton determining unit 240 in the first embodiment. That is,
As shown in FIG. 10, the DFA stored in the automaton holding unit 220 and the search target character string (document data) stored in the document holding unit 230 are input (step 1001), and the input character is input by the DFA. The columns are matched (step 1002), and the DFA state column narrowing process and the matching range determination process are performed (step 10).
03 to 1005), and outputs the processing result (step 10).
06, 1007).

Here, in the first embodiment, a known method is used for the character string matching processing in step 1002. On the other hand, in the second embodiment, since the multi-character collation element is supported, the state transition of the DFA is dynamically determined and evaluated while being generated. 22, 2
FIG. 3 is a flowchart illustrating an input character string determination process by the DFA according to the second embodiment. Figure 2
Referring to 2 and 23, the automaton determination unit 240
First, the index (id of the input string to be processed
x) is initialized (idx = 0), and the DFA state sequence (state_lo
g [0]) is the eclosu in the NFA start state (nfa_init)
It is set to re (nfa_init) (step 2201). Where ec
losure (state: NFA state) is epsdest (state) ∪epsde
epsdest (state ') ... with each element of st (state) as state'
Is the maximum set that is expanded. Further, the value of the index (nfa_idx) to the NFA state referred to in this processing is set to 0 (step 2202).

Next, the number of elements of nfa_idx and state_log [idx] are compared (step 2203). And state_lo
If g [idx] has a larger number of elements, it refers to the NFA
The state (nfa_st) is set to state_log [idx] [nfa_idx], and the pointer (node) to the node on the syntax tree corresponding to the NFA state in the memory area is set to state_trees [nfa_st] (step 2204). Then, it is checked whether the character string representation (node.label) of the node in the syntax tree is '*' or '|'. Nfa_ if it is '*' or '|'
The value of idx is incremented by 1, and the process returns to step 2203 (steps 2205 and 2212).

In step 2205, if the character string representation of the node (node.label) is neither '*' nor '|', the automaton determination unit 240 next determines the index ( Check if there is a multi-character collating element that starts with idx) (step 2
206). If there is a multi-character collating element, the data mblen
Is defined as mblen = length of multi-character matching element starting from index (idx) of input string (step 2
207) Next, it is checked whether the node on the syntax tree corresponding to the NFA state can accept the multi-character collating element (step 2208). If it is acceptable, eclosure (n is added to the state string state_log [idx + mblen] of DFA.
ext (node)) is added (step 2209).

When it is determined in step 2206 that the input character string does not have a multi-character collating element, or when it is determined that the node cannot accept the multi-character collating element in step 2208, or the node is a multi-character collating element. Is determined to be acceptable and the process of step 2209 is performed, the automaton determination unit 240 determines that the character string expression (node.label) of the regular expression corresponding to the node being processed is the input (input (idx )) Is accepted (step 2210). And if accepted,
In the state string state_log [idx + 1] of DFA, eclosure (next (nod
e)) is added (step 2211). Step 221
The string expression (node.label) of the regular expression corresponding to the node being processed with 0 is the input (input (id
x)) is not accepted, or step 2211
After the processing of step 220, the value of nfa_idx is incremented by 1
Return to step 3 (step 2212).

If the values of nfa_idx reach the number of elements of state_log [idx] by repeating the processing of steps 2203 to 2212, then the automaton judging unit 240 advances the index (idx) of the input character string by one. (Step 221
3), the index of the input character string has reached the last character of the input character string, or state_log [i] = 0 (i ≧ id
It is checked whether x) holds (step 221).
4). If neither of these are true, then step 2
Returning to 202, the process is repeated, and if either of the conditions is satisfied, the input character string determination process by the DFA ends.

Next, the automaton judging section 240 is operated by the process shown in FIG.
As shown in 0, the process of narrowing down the DFA state sequence (step 1004) is performed. DFA in the second embodiment
The process of narrowing down the state string is almost the same as the process in the first embodiment shown in FIG. 11, but state_log [idx]
The update processing (step 1102) is different in that it corresponds to the multi-character collating element. 24 and 25 show
7 is a flowchart illustrating a process of narrowing down a DFA status string according to the present embodiment. 24 and 25,
The processing from step 2401 to step 2404 is
The processing is the same as the processing from step 1201 to step 1204 in FIG.
The processing up to 14 is the same as the processing from step 1208 to step 1210 in FIG.

If it is determined in step 2404 that the character string representation (node.label) of the node is neither "*" nor "|", the automaton determination unit 240 next determines whether the input character string is currently being processed by the index ( It is checked whether or not there is a multi-character collating element that starts with idx) (step 240).
5). If there is a multi-character collation element, replace the data mblen with m
blen is defined as the length of the multi-character collating element starting from the index (idx) of the input character string (step 240
6) Next, it is checked whether the node on the syntax tree corresponding to the NFA state can accept the multi-character collating element (step 2407). If the function value next (node) is acceptable, next (node) ∈ state_lo
Check whether g [idx + mblen] holds (Step 2
408).

If it is determined in step 2405 that the input character string does not have the multi-character collating element, or if it is determined in step 2407 that the node cannot accept the multi-character collating element, or in step 2408 next (node ) ∈
When it is determined that state_log [idx + mblen] does not hold, the automaton determination unit 240 next determines the function value next (n
ode), it is checked whether next (node) εstate_log [idx + 1] holds (step 2409). If this does not hold, add 1 to the value of nfa_idx and step 2
Return to step 402 (step 2412). On the other hand, step 2
If next (node) εstate_log [idx + 1] holds in 409, the automaton determination unit 240 next determines the input character (input (idx)) corresponding to the index currently processed by the character string representation of the node. It is checked whether or not to accept (step 2410). And if you don't accept, nfa_idx
Is incremented by 1 and the process returns to step 2402 (step 2
412). Furthermore, if the character string representation of the node accepts the input character input (idx) in step 2410, or if next (node) εstate_log [idx + idx] holds in step 2408, the automaton determination unit 240 then Nfa_st is added to the memory area state_buf (step 241)
1). Then, the value of nfa_idx is incremented by 1, and step 24 is performed.
It returns to 02 (step 2412). As described above, the operation thereafter is the same as the operation in the first embodiment shown in FIG.

Next, the automaton judging section 240 is operated by the process shown in FIG.
As shown in 0, the matching range determination process (step 10
05) is performed. The process of determining the match range in the second embodiment is substantially the same as the process in the first embodiment shown in FIG. 11, but in the transition process to the next NFA state (step 1703), a plurality of characters They differ in that they correspond to matching elements. FIG. 26 is a flowchart illustrating the matching range determination processing according to the present embodiment. In FIG. 26, the operation from step 2601 to step 2603 is the same as the operation from step 1901 to step 1903 in FIG.

If the character string representation (node.label) of the node in the syntax tree is neither "*" nor "|" in step 2602, the automaton determination unit 240 next determines whether the input character string is currently being processed. It is checked whether or not there is a multi-character collating element having the index (idx) of 1 as the head (step 2604). If it does not have a multi-character collation element, set the index (idx) currently being processed to 1
(Step 2608), NFA status (nfa_stat
The process is terminated by setting e) as the next node (next (node)) (step 2609).

If it is determined in step 2604 that the input character string has a multi-character collating element, the data mblen is set to mbl.
It is defined as en = length of the multi-character collating element starting from the index (idx) of the input character string (step 2605), and then the node on the syntax tree corresponding to the NFA state can accept the multi-character collating element. It is checked whether or not (step 2606). And if acceptable, state_log
eclosure (next (node)) is added to [idx + mblen] (step 2607), and the process ends with the NFA state (nfa_state) as the next node (next (node)) (step 26).
09). If it is determined in step 2606 that the node cannot accept the multi-character collating element, the index (idx) currently being processed is advanced by 1 (step 2608),
NFA state (nfa_state) to the next node (next (node))
Then, the processing ends (step 2609).

Next, a specific operation example of the input character string determination processing by the DFA in the second embodiment will be described. Here, the case where the matching of the input character string “aac” is checked against the regular expression “[ab] c | aab” is taken as an example.
Note that “aa” is a valid multi-character collating element, that is, “aa” ε “[ab]”. FIG. 27 is a diagram showing an appropriate NFA of the regular expression “[ab] c | aab” created by the automaton building unit 210. The NFA shown in FIG. 27 has seven states of 0, 1, 2, 3, 4, 5, and 6. 28 to 33 are diagrams showing how the automaton determination unit 240 dynamically determines and generates the transition destination state in the DFA state transition.

Referring to FIGS. 27 and 28, first, N
The DFA start state {0, 2, 3} corresponding to the FA start state NFA state 2 is generated. In this example, the DF corresponding to each character of the input character string "aac"
Each state of A will be generated.

Next, the processing for the first character'a 'of the input character string "aac" is performed. Referring to the NFA shown in FIG. 27, since the NFA state 0 can be transited to the NFA state 1 or the NFA state 3 can be transited to the NFA state 4 for the input character'a ', as shown in FIG. }
Is generated. Here, the input string is prefetched,
Since the first character is'a ', there is a possibility that the first and second character strings are the multi-character collating element “aa”. Therefore, based on the NFA of FIG. 27, the state {1} is temporarily generated as shown in FIG. 30 as the transition destination when this character string is the plural-character collating element “aa”. In addition,
Since this state {1} is an incomplete state due to prefetching, it is indicated by a broken line in FIG.

Next, the process for the second character'a 'of the input character string is performed. Referring to the NFA shown in FIG. 27, there is no transition from NFA state 1 to the character'a ',
Since only a transition is made from NFA state 4 to NFA state 5, state {5} is generated as shown in FIG. Here, the first and second multi-character collation elements “aa” of the input character string correspond to “[ab]”, and the NFA status 0 to NFA
Transition to state 1 or NF for each letter'a '
It was found that there was one of transitions from A state 3 → NFA state 4 → NFA state 5. Therefore, as shown in FIG. 32, a state {1, 5} is generated by fusing the state {1} and the state {5}.

Next, from the NFA of FIG. 27, the NFA state 1 changes to the NFA state 6 for the input character'c ', and the NFA state 5 changes to the NFA state 6 for the input character'b'. I understand. Therefore, as shown in FIG. 33, the state {6} is generated. Since these are the end states of NFA, D for the input character string "aac"
Dynamic generation of the transition destination state in the FA state transition is completed. The last character of the input string "aac"is'c'.
Therefore, the end state {6} is reached, and it can be seen that the input character string "aac" matches this DFA.

In this way, it is possible to perform high-speed pattern matching using DFA even for a regular expression containing multiple character matching elements. When the above-described pattern matching is performed using the NFA, the time required for processing is about O (2 ⁿ ), which is a time proportional to 2 n raised to the input n. On the other hand, in the present embodiment, the time required for the pattern matching using the DFA is about O (n), which is completed in a time proportional to n with respect to the input n. Therefore, in the present embodiment, the time required for the processing can be significantly shortened as compared with the processing using NFA. Moreover, since the number of characters of the matching element can be made variable, it is possible to apply high-speed pattern matching using DFA to character search for multi-byte characters such as Japanese characters.

In the second embodiment, in addition to the first embodiment, the transition destination state in the DFA state transition is dynamically determined and generated when the input character string is determined. , Such a method is used independently of the method according to the first embodiment (that is, a method of obtaining information about a matched portion of a partial regular expression in an input character string by narrowing down the DFA and restoring the NFA state). It is also possible. Also in this case, it is possible to perform high-speed pattern matching using DFA for the regular expression including the plural-character matching element as described above. However, by using it in combination with the first embodiment as described above, it can be applied to back reference. Here, backreference refers to "an operator that waits for a character string that the specified partial regular expression matches". For example, the regular expression "(ab (cd)) \ 1 \ 2" matches "abcdabcdcd", but \ 1 and \ 2 in this case are backreferences. \ 1 and \ 2 1 and 2 indicate the order of appearance of the partial regular expression. For example, in the case of the regular expression "(ab (cd))", it is enclosed in the outer (first appearing) parentheses. "(Ab (cd))" corresponds to \ 1, and "(cd)" in the inner parenthesis that appears next corresponds to \ 2. And each matched string "abc
\ 1 and \ 2 match d "and" cd ". Similarly, the regular expression" (ab) (cd) \ 2 \ 1 "matches" abcdcdab "and" (a
+) \ 1 ”matches“ aa ”,“ aaaa ”,“ aaaaaa ”, ..., That is, an even number of character strings of a. At this time, the first half of the character string becomes (a +) It matches, and the back half matches \ 1. In this way, backreference needs the information "what matched the subexpression until \ 1 appeared", and backreference itself 1
Matches multiple characters with one operator ". Of the two conditions, the former is dealt with in the first embodiment, and the latter is dealt with in the second embodiment.
It is possible to collate and search for character strings using.

[0084]

As described above, according to the present invention,
D for POSIX regular expressions including partial regular expressions and longest match
It becomes possible to process by FA (deterministic finite state automaton). Further, according to the present invention, it becomes possible to process a regular expression including a multi-character collating element by a DFA (deterministic finite state automaton).

[Brief description of drawings]

FIG. 1 is a diagram schematically showing an example of a hardware configuration of a computer device suitable for realizing a document processing system according to a first embodiment.

FIG. 2 is a diagram illustrating a configuration of a document processing system according to the first embodiment.

FIG. 3 is a flowchart showing a schematic flow of a DFA construction process by the automaton construction unit of the first embodiment.

FIG. 4 is a diagram showing a syntax tree constructed from a regular expression “ab | c (de) *”.

FIG. 5 is a diagram showing an example of a program for executing a process of constructing a syntax tree.

FIG. 6 is a diagram illustrating a definition code of a function first used for constructing an appropriate NFA according to the first embodiment.

FIG. 7 is a diagram exemplifying a definition code of a function epsdest used for constructing an appropriate NFA according to the first embodiment.

FIG. 8 is a diagram illustrating a definition code of a function next used for constructing an appropriate NFA according to the first embodiment.

FIG. 9 is a flowchart illustrating a process of constructing an appropriate NFA according to the first embodiment.

FIG. 10 is a flowchart showing a schematic flow of pattern matching processing by the automaton determination unit according to the first embodiment.

FIG. 11 is a flowchart illustrating an operation of narrowing down a DFA status string according to the first embodiment.

FIG. 12 is a state sequence (state_lo) of the DFA in FIG.
It is a flowchart explaining the detail of the process which updates g [idx]).

FIG. 13 is a flowchart illustrating the details of ε transition source addition processing in FIG.

FIG. 14 is a diagram showing an appropriate NFA of a regular expression “((a *) a) * a”.

15 is a diagram showing state transitions for the DFA of the NFA and the character string "aaaa" shown in FIG.

FIG. 16 is a diagram showing a state of NFA state transition with respect to an input character string “aaaa”.

FIG. 17 is a flowchart illustrating an operation in a match range determination process according to the first embodiment.

FIG. 18 is a flowchart illustrating details of register update processing in FIG. 17;

FIG. 19 is a flowchart illustrating details of a transition process to the next NFA state in FIG.

20 is a diagram showing a state of transition shown in FIG. 16 and a diagram showing a state of determining a match range based on a state transition route by a narrowed DFA state sequence.

FIG. 21 is a diagram showing a restored NFA state sequence.

FIG. 22 is a flowchart illustrating an input character string determination process by DFA according to the second embodiment.

FIG. 23 is a flowchart illustrating an input character string determination process by DFA according to the second embodiment.

FIG. 24 is a flowchart illustrating a process of narrowing down a DFA status string according to the second embodiment.

FIG. 25 is a flowchart illustrating a process of narrowing down a DFA status string according to the second embodiment.

FIG. 26 is a flowchart illustrating a match range determination process according to the second embodiment.

FIG. 27 is a diagram showing an appropriate NFA of a regular expression “[ab] c | aab” created by the automaton building unit of the second embodiment.

FIG. 28 is a diagram showing how a transition destination state in a DFA state transition is dynamically determined and generated according to the second embodiment, and is a diagram showing how states up to {0} are generated.

FIG. 29 is a diagram showing how a transition destination state in the DFA state transition is dynamically determined and generated according to the second embodiment, and is a diagram showing how states {1, 2} are generated. is there.

FIG. 30 is a diagram showing how a transition destination state in a DFA state transition is dynamically determined and generated according to the second embodiment, and is a diagram showing how state {1} is temporarily generated.

FIG. 31 is a diagram showing how a transition destination state in the DFA state transition is dynamically determined and generated according to the second embodiment, and is a diagram showing how states up to state {3} are generated.

FIG. 32 is a diagram showing how a transition destination state in DFA state transition is dynamically determined and generated according to the second embodiment, and is a diagram showing how states {1, 3} are generated. is there.

FIG. 33 is a diagram showing a state in which a transition destination state in the DFA state transition is dynamically determined and generated according to the second embodiment, and is a state {4} and a state {5} which are end states.
It is a figure which shows a mode that what was generated.

FIG. 34 is a diagram illustrating an example of an NFA that matches a regular expression “([ab] c) * ac”.

FIG. 35 is a diagram showing a DFA equivalent to the NFA of FIG. 34.

FIG. 36 is a diagram showing an example of an NFA corresponding to a regular expression including a multi-character matching element.

[Explanation of symbols]

101 ... CPU (central processing unit), 102 ... M / B (motherboard) chip set, 103 ... Main memory, 1
05 ... hard disk, 200 ... document processing system, 2
Reference numeral 10 ... Automata construction unit, 220 ... Automaton holding unit, 230 ... Document holding unit, 240 ... Automaton judging unit, 250 ... Document processing unit, 260 ... Processing program holding unit, 300 ... Input / output device

Continued front page    (72) Inventor Isamu Hasegawa             1623 1423 Shimotsuruma, Yamato-shi, Kanagawa Japan             BM Co., Ltd. Daiwa Office F-term (reference) 5B075 ND03 QM10

Claims (23)

[Claims] 1. A character string collating method for collating a character string using a computer, the step of creating a non-deterministic finite state automaton from a regular expression of a character string and storing it in a memory; and the non-deterministic finite state automaton from the memory. Reading a state automaton, creating a deterministic finite-state automaton based on the nondeterministic finite-state automaton, and storing in a memory; and reading the deterministic finite-state automaton from the memory, using the deterministic finite-state automaton to generate a character string And a step of specifying a matching range of the character string by using the nondeterministic finite state automaton and the deterministic finite state automaton read from the memory for the matched character string. Character string matching method characterized by. 2. The non-deterministic finite state automaton generating step is characterized in that the regular expression of a character string is associated with each non-deterministic element in the regular expression except for an element designating a certain range. Generating a state of a finite state automaton, and corresponding ε transition to an element meaning repetition and an element meaning selection, and to other states to the state associated with the next element The character string matching method according to claim 1, further comprising a step of associating transitions. 3. The step of identifying the match range includes a step of deleting an unnecessary state sequence that does not reach an end state from the state sequence showing a state transition by the deterministic finite state automaton, and the remaining state sequence. The method according to claim 1, further comprising the step of: specifying a matching range of the character string based on the above. 4. The step of identifying the match range comprises:
2. The method according to claim 1, further comprising the step of identifying a matching range of the character string based on a state sequence satisfying a longest leftmost rule among state sequences showing state transitions by the deterministic finite state automaton. String matching method. 5. A character string collating method for collating a character string using a computer, wherein a deterministic finite state automaton created based on a regular expression of a predetermined character string is read from a memory and the deterministic finite state automaton is used. The first step of matching the character strings and storing the processing result in the memory;
And a third step of identifying which character in the character string to be processed matches which part of the regular expression based on the narrowed-down state sequence. String matching method. 6. The third step is to select, from among the narrowed-down state strings, a state string having the largest number of repetitions that appears first, and based on the state string, the character string to be processed is selected. 6. The character string matching method according to claim 5, further comprising a step of determining which part of the regular expression and each part of the regular expression are matched. 7. A character string collating method for collating character strings using a computer, wherein a deterministic finite state automaton created based on a regular expression of a predetermined character string is read from a memory and the deterministic finite state automaton is used. Based on a first state sequence indicating a state transition by the deterministic finite state automaton, and a state transition in the non-deterministic finite state automaton. A second step of recovering a second state sequence shown, and obtaining information about which character in the string matched which part of the regular expression based on the restored second state sequence And a third step of performing a character string matching method. 8. The character string matching method according to claim 7, wherein the second step includes a step of restoring a state sequence satisfying a longest leftmost rule as the second state sequence. . 9. A character string collating method for collating a character string using a computer, the step of creating a non-deterministic finite state automaton from a regular expression of a character string and storing it in a memory; and the non-deterministic finite state automaton from the memory. Reading a state automaton, creating a deterministic finite state automaton based on the non-deterministic finite state automaton, and storing it in a memory; and for each element of the character string to be processed, the deterministic finite state automaton read out from the memory. And a step of performing matching while dynamically determining a transition destination state in the state transition of the state automaton. 10. The step of performing the matching includes a step of pre-reading the character string to be processed and determining whether or not the character string includes a character string that may correspond to a multi-character collating element. , If the character string includes a character string that may correspond to a multi-character collating element, the state of the transition destination is dynamically changed by reflecting the state transition when the character string is a multi-character collating element. The character string matching method according to claim 9, further comprising a step of determining. 11. After performing the matching, regarding the matched character string, using the nondeterministic finite state automaton and the deterministic finite state automaton read from the memory, information regarding a matching range of the character string is obtained. The character string matching method according to claim 9, further comprising a step of obtaining. 12. A character string collating method for collating a character string using a computer, the step of creating a non-deterministic finite state automaton from a regular expression of a character string and storing it in a memory, and the non-deterministic finite state automaton from the memory. Read the state automaton, create a deterministic finite state automaton based on the non-deterministic finite state automaton, store it in memory, and pre-read the character string to be processed, and apply the multi-character matching element to the character string. If a possible character string is included, for each element of the character string, the state transition of the deterministic finite state automaton read from the memory is generated, and the character that can be the multi-character matching element Virtually generating a state transition corresponding to the column and performing matching based on the state transition. A character string matching method characterized by the fact that it is used. 13. A document processing device for searching a character string using a regular expression, a nondeterministic finite state automaton constructing means for constructing a nondeterministic finite state automaton from a regular expression of a character string, and the nondeterministic finite state automaton. A deterministic finite state automaton constructing means for constructing a deterministic finite state automaton based on the non-deterministic finite state automaton constructed by the constructing means, and using the deterministic finite state automaton constructed by the deterministic finite state automaton constructing means And a determining means for matching the character strings, the determining means, with respect to the matched character string, further using the nondeterministic finite state automaton and the deterministic finite state automaton, the match range of the character string A document processing device characterized by specifying. 14. The nondeterministic finite state automaton constructing means associates, with respect to a regular expression of a character string, one nondeterministic finite state corresponding to each element of the regular expression except elements designating a certain range. Generate the state of the automaton,
And, the element that means repetition and the element that means selection are associated with the ε transition, and the other elements are associated with the transition to the state associated with the next element. Item 13. The document processing device according to item 13. 15. The state sequence narrowing means for deleting and narrowing down an unnecessary state sequence that does not reach an end state among the state sequences showing the state transitions by the deterministic finite state automaton, and the state sequence narrowing down. 14. The document processing apparatus according to claim 13, further comprising a match range determination unit that specifies a match range of the character string based on a state string narrowed down by the unit. 16. A non-deterministic finite state automaton constructing means for constructing a non-deterministic finite state automaton from a regular expression of a character string in a document processing device for searching a character string using a regular expression, and the non-deterministic finite state automaton. A deterministic finite state automaton constructing means for constructing a deterministic finite state automaton based on the non-deterministic finite state automaton constructed by the constructing means, and using the deterministic finite state automaton constructed by the deterministic finite state automaton constructing means A document processing apparatus, comprising: a determination unit that performs matching while dynamically determining a transition destination state in the state transition of the deterministic finite state automaton for each element of the character string to be processed. . 17. The multi-character collating means pre-reads the character string to be processed, judges whether or not the character string includes a character string that may correspond to a multi-character collating element, When it is determined that an element includes a character string that may be applicable, the state transition destination is dynamically determined by reflecting the state transition in the case where the character string is a multi-character collating element. The document processing device according to claim 16. 18. The state sequence narrowing means for deleting and narrowing down an unnecessary state sequence that does not reach an end state among the state sequences showing the state transition by the deterministic finite state automaton, and the state sequence narrowing down. The document processing apparatus according to claim 16, further comprising a match range determination unit that specifies a match range of the character string based on the state string narrowed down by the unit. 19. A program for controlling a computer to collate a character string, the process comprising: creating a nondeterministic finite state automaton from a regular expression of a character string; storing the nondeterministic finite state automaton in the memory; and the nondeterminism from the memory. Read a finite state automaton, create a deterministic finite state automaton based on the non-deterministic finite state automaton, and store it in memory, and read the deterministic finite state automaton from the memory, character using the deterministic finite state automaton A process of performing string matching, and a process of specifying the matching range of the character string by using the nondeterministic finite state automaton and the deterministic finite state automaton read from the memory for the matched character string, A program characterized by causing a computer to execute Beam. 20. The non-deterministic finite-state automaton is created by a non-deterministic one in which a regular expression of a character string is associated with each element of the regular expression except elements designating a certain range. A process for generating a state of a finite state automaton, and an ε transition is made to correspond to an element that means repetition and an element that means selection, and to other states to the state associated with the next element. 20. The program according to claim 19, including a process of associating transitions. 21. The process of identifying the match range includes a process of deleting an unnecessary state sequence that does not reach an end state among state sequences showing a state transition by the deterministic finite state automaton, and the remaining state sequence. 20. The program according to claim 19, further comprising: a process of specifying a matching range of the character string based on the above. 22. A program for controlling a character string by controlling a computer, a process of creating a nondeterministic finite state automaton from a regular expression of a character string and storing the nondeterministic finite state automaton in the memory, and the nondeterministic property from the memory. A process of reading a finite state automaton, creating a deterministic finite state automaton based on the non-deterministic finite state automaton, and storing it in a memory, and for each element of the character string to be processed, the deterministic read from the memory A program for causing the computer to perform a process of performing matching while dynamically determining a transition destination state in a state transition of a finite state automaton. 23. The matching process includes a process of pre-reading the character string to be processed and determining whether or not the character string includes a character string that may correspond to a multi-character collating element. When the character string includes a character string that can correspond to a multi-character collating element, the state of the transition destination is dynamically determined by reflecting the state transition when the character string is the multi-character collating element. The program according to claim 22, further comprising: