JP2010225156A - Apparatus and program for collating character string - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a memory capacity required for storing a state transition table for the collation of a character string, having regular expression as a collation condition. <P>SOLUTION: A character string collation apparatus includes: a state transition table generator 3 for generating the state transition table, based on a collation condition 2 described with regular expression; and an automaton that shifts based on the state transition table generated by the state transition table generator 3. In the state transition table generated on the basis of the collation condition 2, when a next shift destination state relative to the set of the present state and an input character does not exist, the automaton shifts to either an initial state or a predetermined state, without proceeding to reading the input character. Thus, the memory capacity required for storing the state transition table is reduced. In the state transition table generated based on the collation condition 2, when the next shift destination state relative to the set of the present state and the input character does not exist, the state transition table is generated by setting an exclusion character for shifting to the predetermined state without proceeding to reading the input character. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for collating a pattern designated by a regular expression with text in a sentence.

  In recent years, the digitization of documents is progressing in various fields, and an efficient document search method is required. As a search method, there is a method of matching a pattern specified by a regular expression with text in a document. The regular expression is described in, for example, Non-Patent Document 1, and is a notation method for expressing a class of a language called a regular language. As a character string matching method using a regular expression as a search condition, a method using DFA (Deterministic Finite Automaton) is known.

  The string matching method by DFA is based on a model of state transition machine (automaton). The state transition machine has a state and a state transition function inside. The state transition function is a function that determines the next state for the current state and the input character. In the character string matching method using DFA, the input text is read one character at a time, and a transition is made to the next state obtained by applying the state transition function to the set of the current state and the input character. According to this method, it is possible to perform collation by scanning the text once without going back, and high-speed character string collation is possible. When collating with multiple conditions, a finite automaton with output (Moore machine) that extends DFA and defines the output for each state is also used to distinguish the conditions that succeeded in collation.

  The DFA state transition function is determined by the regular expression that is the matching condition. Conventionally, the regular expression is once converted to NFA (Non-deterministic Finite Automaton), and the procedure to convert NFA to DFA For example, it is described in Non-Patent Document 1. The DFA character string matching method has a feature that it is fast, but on the other hand, there is a drawback that the state transition table for realizing the DFA state transition function tends to be huge.

  As an example, the collation condition of FIG. 52 disclosed in Patent Document 3 is taken as an example. FIG. 53 shows a state transition table and a failure function generated from the matching condition of FIG. 52 in the conventional finite automaton with output. Thus, it is necessary to generate a state transition table that holds 90 combinations for the number of states 18 and the five types of characters.

  As a method for solving such a problem, Patent Document 1 and Patent Document 2 describe that a state transition table based on the AC (Aho-Corasick) method is converted to DFA, then a transition operation to the initial state and the next of the initial state. A method of reducing the storage capacity of the state transition table by removing the transition operation to the state from the state transition table is shown. However, in the character string matching methods shown in Patent Document 1 and Patent Document 2, since the object of collation is limited to fixed character string keywords, general regular expressions cannot be the object of collation.

  Patent Document 3 discloses a method for reducing state transition tables by defining a failure function in DFA. However, in the method disclosed in Patent Document 3, there is a case where the transition fails again in a state where the transition is once performed by the failure function. That is, transition failures may occur in a chain. In such a case, there is a problem in that it is necessary to refer to the failure function repeatedly and the collation speed is reduced.

  As an example, the collation condition of FIG. 52 disclosed in Patent Document 3 is taken as an example. FIG. 54 shows a state transition table and a failure function generated from the matching conditions shown in FIG.

  For example, the collation condition is the condition of FIG. 52 and the input character string is “aaca”.

  In the method disclosed in Patent Document 3, the state is first initialized to state 1. Next, the first character “a” is read, and the state transitions to state 3 indicated in the column of input character “a” in the state 1 row of the state transition table. Next, the second character “a” is read, and similarly, the state 3 is changed to the state 6. Next, the state 6 is changed to the state 10 by reading the third character “c”. However, when the next character “a” appears, since there is no transition destination corresponding to the character “a” in state 10, first, state 5 that is the transition destination at the time of failure of state 10 Transition to. Further, since there is no transition destination corresponding to the character “a” in the state 5, the state transitions to the state 2 that is the transition destination when the state 5 fails. However, since there is no transition destination corresponding to the character “a” even in the state 2, the state transitions to the state 1 that is the transition destination when the state 2 fails. In state 1, there is a transition destination state 3 corresponding to the character “a”, so transition is made to state 3. In this way, the state transition table is referenced and state transitions are made four times in total for the fourth input character, and seven state transitions are required for the four input characters as a whole. And As described above, in the method of Patent Document 3, it is sometimes necessary to repeat the state transition failure and refer to the transition destination at the time of the failure each time. For this reason, there has been a problem that the number of times the state transition table is referenced increases and collation performance deteriorates.

E.J.Hopcroft, D.J.Ullman, "Formal Languages and their Relation to Automata", Addison Wesley (1969) JP 2004-103035 A JP 2004-103034 A Japanese Patent No. 2994926

  The present invention has been made to solve the above-described problems, and aims to reduce the storage capacity required to store a state transition table for character string matching using a regular expression as a matching condition. To do.

  In addition, the number of times the state transition table is referred to due to failure of transition is set to 2 or less per character to prevent performance degradation due to performance degradation due to repeated failure of transition and to enable high-speed character string matching And

  A character string matching device according to the present invention is based on a state transition table generating unit that generates a state transition table based on a matching condition described by a regular expression, and a state transition table generated by the state transition table generating unit. The automaton does not read the input character when there is no next transition destination state for the set of the current state and the input character in the state transition table generated based on the matching condition. To the initial state.

  A state transition table generating unit that generates a state transition table based on a matching condition described in a regular expression; and an automaton that transitions based on the state transition table generated by the state transition table generating unit, and the automaton In the state transition table generated based on the collation condition, when there is no next transition destination state for the set of the current state and the input character, the state is determined by transitioning to the initial state without reading the input character. The storage capacity required to store the transition table can be reduced.

It is explanatory drawing which shows the structure of a character string collation apparatus. It is explanatory drawing which shows the structure of the collation conditions 2. FIG. FIG. 10 is an explanatory diagram showing a configuration of conditional expression 17; It is explanatory drawing which shows the structure of the state transition table production | generation part 3. FIG. It is explanatory drawing which shows the structure of the state transition table storage part. 4 is an explanatory diagram showing a configuration of an output table storage unit 5. FIG. It is explanatory drawing which shows the structure of the collation result. It is a flowchart which shows operation | movement of a character string collation apparatus. Flowchart showing verification condition compilation operation Flow chart showing the procedure for adding failure transition to the initial state Flow chart showing the procedure for adding a failed transition to the initial state (when there is no transition destination due to σ _any from the initial state) Flow chart showing the procedure for adding a failed transition to the initial state (when there is a transition destination by σ _any from the initial state) Flow chart showing the procedure for removing non-deterministic transitions Flow chart showing the procedure for initializing the state set Flow chart showing the procedure for removing non-deterministic transitions on Σ Non-deterministic transition removal procedure for σ _other Flow chart showing the procedure for creating a new state Flow chart showing the procedure for correction of state transition by σ _other Flow chart showing the procedure for removing unused state Flow chart showing procedure for removing redundant state Flow chart showing the procedure for deleting redundant state transitions Flow chart showing the procedure for generating state transition table and output table Flow chart showing input document verification procedure Explanatory drawing explaining replacement of metacharacter "." Explanatory drawing explaining replacement of metacharacter "^" Explanatory drawing explaining NFA including ε transition to initial state Explanatory drawing explaining omission of ε transition Explanatory drawing explaining when failure transition needs to be added Explanatory drawing explaining addition of failure transition Explanatory drawing explaining the case where transition by σ _any exists from the initial state Explanatory drawing explaining the addition of the failure transition when transition by σ _any exists from the initial state Explanatory drawing explaining removal of non-deterministic transition (before removal) Explanatory drawing explaining removal of non-deterministic transition (after removal) Explanatory drawing explaining removal of non-deterministic transition (before removal) Explanatory drawing explaining removal of non-deterministic transition (after removal) Explanatory drawing explaining removal of non-deterministic transition (before removal) Explanatory drawing explaining removal of non-deterministic transition (after removal) Explanatory drawing explaining removal of non-deterministic transition (before removal) Explanatory drawing explaining removal of non-deterministic transition (after removal) Explanatory drawing explaining removal of non-deterministic transition (before removal) Explanatory drawing explaining removal of non-deterministic transition (after removal) Explanatory drawing explaining deletion of redundant state transition Explanatory drawing explaining merge of redundant states Explanatory drawing showing the configuration of verification conditions Explanatory diagram showing the structure of the state transition table Explanatory diagram showing the structure of the output table Explanatory drawing showing an example of operation Explanatory drawing showing the configuration of verification conditions Explanatory diagram showing the structure of the state transition table Explanatory diagram showing the structure of the output table Explanatory drawing showing an example of operation Verification conditions disclosed in Patent Document 3 State transition table of conventional DFA with output according to Patent Document 3 Conventional state transition table and output table according to Patent Document 3

DESCRIPTION OF SYMBOLS 1 Character string collation apparatus 2 Collation conditions 3 State transition table production | generation part 4 State transition table storage part 5 Output table storage part 6 Input document 7 Input character reading part 8 SDFA automaton 9 State storage part 10 Collation result 11 State transition 12 Output description 13 Current state 14 Input character 15 Next state 16 Condition number 17 Conditional expression 18 Condition description 21 State transition table generation control unit 22 NFA state set 23 NFA state transition set 24 NFA output description set 25 State set 26 State transition set 27 Output description set 31 Hash value calculator 32 Hash value 33 State transition hash pointer 34 State transition hash chain 35 State transition hash table 36 Comparison unit 41 Condition number index 42 Condition number chain

Example 1.
FIG. 1 is a block diagram showing a character string collating apparatus according to the present invention.

  In FIG. 1, a character string matching device 1 is a device that performs character string matching using a regular expression according to the present invention, and outputs as a matching result 10 whether or not the input document 6 includes a document that satisfies the matching condition 2. . The collation condition 2 is a condition describing a character string collation condition and is input to the character string collation apparatus 1. The state transition table generation unit 3 generates a state transition 11 and an output description 12 from the matching condition 2 and passes them to the state transition table storage unit 4 and the output table storage unit 5, respectively. The state transition table storage unit holds a set of state transitions 11. The output table storage unit 5 holds an output description 12. The input document 6 is a document to be collated. The input character reading unit 7 extracts characters included in the input document 6 one by one and sends them to the SDFA automaton 8 as input characters 14. The SDFA automaton 8 stores the current state 13 in the internal state storage unit 9, receives the input character 14 from the input character reading unit 7, and refers to the state transition table storage unit 4 and the output table storage unit 5 to refer to the state storage unit 9. The current state 13 stored in the table is updated and the collation result 10 is output. The state storage unit 9 stores the state held in the SDFA automaton 8. Reference numeral 10 denotes a collation result. Reference numeral 11 denotes a state transition, which is a set of a current state 13, an input character 14, and a next state 15. Reference numeral 12 denotes an output description, which is a set of a current state 13 and a condition number 16. 13 is the current state, and 14 is the input character. 15 is the next state. 16 is a condition number.

  FIG. 2 is a diagram showing a configuration of collation condition 2 in the present invention. In the figure, the conditional expression 17 is an individual condition constituting the collation condition 2, and the collation condition 2 includes one or a plurality of conditional expressions 17.

  FIG. 3 is a diagram showing the configuration of conditional expression 17 in the present invention. The conditional expression 17 includes a condition number 16 and a condition description 18. The condition number 16 is a number for uniquely distinguishing the conditional expression, and the condition description 18 is a matching condition described by a regular expression.

  FIG. 4 is a diagram showing a configuration of the state transition table generation unit 3 in the present invention. In the figure, a state transition table generation control unit 21 controls an operation procedure for generating a state transition generation table. The NFA state set 22, the NFA state transition set 23, the NFA output description set 24, the state set 25, the output transition set 26, and the state description set 27 are data to which the state transition table generation control unit 21 refers.

  FIG. 5 is a diagram showing an example of the configuration of the state transition table storage unit 4 in the present invention. In the figure, reference numeral 31 denotes a hash value calculation unit, which calculates a hash value 32 from the current state 13 and the input character 14. The hash value 32 is a hash value calculated by the hash value calculation unit 31. The state transition hash pointer 33 is a table that stores a plurality of pointers of the state transition hash chain 34. The state transition hash chain 34 is a set of a pointer to the state transition hash chain 34, a current state 13, an input character 14, and a next state 15. The state transition hash table 35 has a data structure including a state transition hash pointer 33 and a state transition hash chain 34. The comparison unit 36 compares the set of the current state 13a and the input character 14a input from the outside with the current state 13b and the input character 14b stored in the state transition hash table 35, and outputs the next state 15.

  FIG. 6 is a diagram showing the configuration of the output table storage unit 5 in the present invention. The condition number index 41 stores a plurality of pointers to the condition number chain. The condition number chain 42 is a set of a pointer to the condition number chain 42 and the condition number 16.

  FIG. 7 is a diagram showing an example of the collation result 10 in the present invention. The matching result 10 includes a condition number 16 that has been successfully matched with the input document 6.

  Next, prior to describing the operation of the present invention, terms and symbols used in the following description will be described.

As described in Non-Patent Document 1 and the like, a conventionally known deterministic finite automaton with output is given by a set of (Q, Σ, Δ, δ, λ, q ₀ ). Where Q is a state set. Σ is an input alphabet and includes the empty character ε. Δ is an output alphabet, δ is a transition function (Q × Σ → Q), λ is an output function (Q → Δ), and q ₀ is an initial state.

The SDFA automaton 8 of the present invention is given by a set of (Q _s , Σ _s , Δ _s , δ _s , λ _s , q ₀ ).

Here, Q _s is the state set 25, which corresponds to the state set Q of the conventional finite automaton with output.

Δ _s is an output alphabet, and is a set of sets of condition numbers 16 in this embodiment.

δ _s is a state transition function realized by the state transition table storage unit 4. Hereinafter, when the current state 13 is q _s and the input character 14 is σ _s , the next state 15 is q _d .
δ _s (q _s , σ _s ) = q _d
Is written.

q ₀ is the initial state, the meaning is the same as the output with deterministic finite automaton conventionally known.

Σ _s is an extended input alphabet obtained by adding an arbitrary character σ _any and an excluded character σ _other to an input alphabet Σ of a conventional finite automaton with output. That is,
Σ _s = Σ∪ {σ _any , σ _other }
And

Also,
δ _s (q _s , σ _s ) = q _d
When the state transition 11 is t,
t = trans (q _s , q _d , σ _s )
It shall be written as

A set of state transitions 11 in which the next state 15 exists with respect to the set of the current state 13 input characters 14 is referred to as a state transition set 26 and is denoted as T. Also, for the state transition t = trans (q _s , q _d , σ _s ), q _s is called the starting point, q _d is the end point, and σ _s is called the transition character. Furthermore, the function that gives the starting point of state transition t is called Source, the function that gives the end point is called Destination, and the function that gives the transition character is called Char.
q _s = Source (t)
q _d = Destination (t)
σ _s = Char (t)
Is written.
Also,
λ _s (q _s ) = r
If the output description 18 is d,
d = desc (q _s , r)
It shall be written as

A set of output descriptions 12 whose output alphabet r is not empty with respect to the current state 13 is called an output description set 27 and is denoted as D. For the output description d = desc (q _s , r), q _s is called the output state and r is called the output result. Furthermore, the function that gives the output state of the output description d is called State, and the function that gives the output result is called Result.
q _s = State (d)
r = Result (d)
Will be written.

NFA is generated in the process of generating the SDFA automaton 8 of the present invention. NFA is (Q _(NFA) , Σ _s , Δ _s , δ _{s (NFA)} , λ _{s (NFA)} , q _{0 (NFA)} ) Represented by a pair of

Here, Q _(NFA) represents the NFA state set 22. The state set 25 is a set of NFA state sets. That is, when the state set 25 is expressed as Q, there is a relationship of Q = ^{2Q (NFA)} . Hereinafter, in order to distinguish the NFA state from the DFA state, it is denoted as q _(NFA), and the initial state of _NFA is denoted as q _{0 (NFA)} .

δ _{s (NFA)} is an NFA state transition function, where the current NFA state 13 is q _(NFA) and the input character 14 is a function from σ _s to the next state set.

λ _{s (NFA)} is an output function of NFA, or when the current NFA state 13 is q _(NFA) , the output alphabet becomes r∈Δ _s λ _{s (NFA)} (q _(NFA) ) = r
Is written.

Δ _s is a set of condition numbers 16, Σ _s is an extended input alphabet, and its meaning is the same as the SDFA automaton 8 of the present invention.

The NFA state transition is defined as follows for the NFA state. NFA state q _{d (NFA)} is
q _{d (NFA)} ∈δ (q _{s (NFA)} , σ _s )
When NFA state transition 32 is
t _(NFA) = trans (q _{s (NFA)} , q _{d (NFA)} , σ _s )
It shall be written as A set of NFA state transitions is called an NFA state transition set and is denoted as T _(NFA) . Similarly, for state transition t _(NFA) = trans (q _{s (NFA)} , q _{d (NFA)} , σ _s ), q _{s (NFA)} is the starting point, q _{d (NFA)} is the ending point, σ _s Is called a transition character. Furthermore, the function that gives the start point of state transition t _(NFA) is called Source, the function that gives the end point is called Destination, and the function that gives the transition character is called Char.
q _{s (NFA)} = Source (t _(NFA) )
q _{d (NFA)} = Destination (t _(NFA) )
σ _{s (NFA)} = Char (t _(NFA) )
Will be written.

A set of the NFA state q _(NFA) and the output alphabet r is called an NFA output description.
λ _s (q _(NFA) ) = r
If the NFA output description 34 is d _{(NFA)
d (NFA) = desc (q} s (NFA), r)
It shall be written as

A set of NFA output descriptions for which the output alphabet p is not empty for the NFA state 13 is called an NFA output description set 24 and is denoted as D _(NFA) . For the output description d _(NFA) = desc (q _(NFA) , r), q _(NFA) is called the output state and r is called the output result. Furthermore, the function that gives the output state of the output description d is called State, and the function that gives the output result is called Result.
q _(NFA) = State (d _(NFA) )
r _(NFA) = Result (d _(NFA) )
Will be written.

  This is the end of the explanation of terms and symbols. Next, the operation will be described.

  FIG. 8 shows the operation of the character string matching device 1 of the present invention.

  First, the character string matching device 1 of the present invention receives the matching condition 2 and generates a state transition 11 and an output description 12 by the state transition table generating unit 3, that is, executes a procedure for compiling the matching condition (step S51). ).

  Next, a procedure for receiving the input document 6 and outputting the collation result 10 while referring to the state transition 11 and the output description 12 by the input character reading unit 7 and the SDFA automaton 8 is sequentially executed (step S52).

  In this example, “comparison of input document” is performed once for “compilation of collation conditions”, but it is generated by a procedure of “compilation of collation conditions”. Using the state transition 11 and the output description 12, “input document matching” may be performed on a plurality of input documents.

  Next, the procedure “Compilation of collation conditions” will be described with reference to FIG.

  First, an NFA including an ε transition is generated from a regular expression by a procedure of “Generation of NFA including an ε transition” (step S101).

  Next, the ε transition (transition by an empty character) included in the NFA is removed by the procedure of “removing ε transition” (step S102).

  Next, a transition to the initial state when the verification fails is added by the procedure “addition of transition to the initial state” (step S103).

  Next, non-deterministic transitions are removed by the procedure of “removing non-deterministic transitions” (step S104).

  Next, the state unnecessary in the previous procedure is removed by the procedure of “removal of unused state” (step S105).

  Next, the redundant state and redundant state transition are removed by the procedure “reduction of the number of states” (step S106).

  Next, a state transition table and an output table are generated from the state set by a procedure of “generation of state transition table and output table” (step S107).

  The procedure “Compilation of collation conditions” can be executed by the above procedure.

  For the procedure of “Generation of NFA including ε transition” in step S101, a known procedure shown in Non-Patent Document 1 or the like can be used.

However, as shown in FIG. 24, the metacharacter “.” Representing an arbitrary character included in the regular expression is replaced with σ _any . Hereinafter, in FIG. 24 to FIG. 31, the state q _(NFA) is simply expressed as q.

In addition, as shown in FIG. 25, meta characters “^” that are included in the regular expression and represent a character set other than the specific character set are replaced with σ _other , and a state transition from the corresponding state to the initial state q _{0 (NFA)} is added. Add a procedure to do.

  The procedure of “removing the ε transition” in step S102 is also realized by replacing the ε transition (transition by an empty character) with a transition to the transition destination set by a known procedure shown in Non-Patent Document 1 or the like. can do.

  Next, FIG. 10 shows the procedure of “add failure transition to initial state” in step S103.

First, when there is no transition due to σ _any from the initial state q _{0 (NFA) of} the NFA generated in step S102, the process proceeds to step S202. In cases other than that described here, process flow proceeds to Step S203 (Step S201).

The procedure of “add failure transition to initial state (when there is no transition destination by σ _any from the initial state)” is executed and the process ends (step S202).

If there is a transition due to σ _any from the initial state q _{0 (NFA)} of NFA in step S201, the procedure “add failure transition to initial state (when there is a transition destination due to σ _any from the initial state)” is executed. And the process ends (step S203).

In the present embodiment, the process is divided from the initial state q _{0 (NFA) of NFA} depending on whether or not a transition destination by σ _any exists, but the process of step S202 can be substituted by step S203. . The processing in step S202 is limited to the case where the applicable range is “the transition destination by σ _any exists from the initial state” as compared with step S203, but the number of state transitions can be further reduced. The fact that there is a transition destination with σ _any from the initial state means that an arbitrary character “.” Is specified at the beginning of the regular expression that serves as a matching condition, but such a specification is rarely used in practice. Therefore, in many cases, the number of state transitions can be reduced by applying the process of step S202.

  According to the above procedure, the procedure “add failure transition to initial state” can be performed.

Next, the procedure of “addition of failed transition to initial state (when there is no transition destination by σ _any from the initial state)” in step S202 will be described.

In general, in NFA or DFA that performs continuous character string matching, character string matching from any character position can be realized by adding an ε transition from all states to the initial state. FIG. 26 shows an example in which an ε transition is added to the NFA corresponding to the regular expression (a | b | c (d | e)) f. In SDFA automaton 8 of the present invention, in order to perform re-state transition back to the initial state q ₀ without advancing an input character on failure to transition, it can be omitted ε transition to the initial state q _0. That is, the NFA shown in FIG. 27 may be configured, and the total number of state transitions can be greatly reduced. However, when it is possible to transition from the initial state q _{0 (NFA)} to the state q _{1 (NFA)} by the transition character σ, there is a transition from the state q _(NFA) by the transition character σ, or the state q _{(NFA )} Is one of the transition characters σ starting from σ _any, it is possible to transition to q _{1 (NFA)} by σ even when the transition by σ is successful, so from the state q _(NFA) to σ Add a state transition to state q _{1 (NFA)} by. FIG. 28 shows an example of a regular expression (a | b | c (d | e)) f. In the example of FIG. 28, it is possible to transition from the initial state q _{0 (NFA)} to the state q _{1 (NFA)} by the transition character a, and there is a transition by the transition character a from the state q _{3 (NFA)} . _Is added with a transition from state q _{3 (NFA)} to state q _{1 (NFA)} by transition character a. Although the transition from the state q _{3 (NFA)} by the transition character a becomes a non-deterministic state transition, this non-deterministic transition is finally removed by the subsequent procedure of “removing non-deterministic transition”. Procedures "Add failed transition to initial state (when there is no transition destination by σ _any from the initial state)" and "Add failed transition to initial state (when there is a transition destination from the initial state by σ _any )" Is for performing this processing.

With reference to FIG. 11, the procedure of “addition of failed transition to initial state (when there is no transition destination by σ _any from the initial state)” in step S202 will be described.

First, t _{0 (NFA)} is set as the first state transition starting from the initial state q _{0 (NFA)} of the NFA generated in step S102, and the process proceeds to step S302 (step S301).

When the processing of all the state transitions t _{0 (NFA)} starting from the initial state q _{0 (NFA)} of the _NFA is completed, the procedure ends. In cases other than that described here, process flow proceeds to Step S303 (Step S302).

σ is set to Char (t _{0 (NFA)} ), and the process proceeds to step S304 (step S303).

Let q _(NFA) be the first NFA state included in the NFA state set Q _(NFA) , and go to step S305 (step S304).

When the processing of all the NFA states q _(NFA) included in the NFA state set Q _(NFA) is completed, the process proceeds to step S313. In cases other than that described here, process flow proceeds to Step S306 (Step S305).

When the NFA state q _(NFA) is the initial state q _{0 (NFA)} , the process proceeds to step S312. In cases other than that described here, process flow proceeds to Step S307 (Step S306).

_Let t _(NFA) be the first NFA state transition starting from q _(NFA) , and proceed to step S308 (step S307).

When the processing of all NFA state transitions t _(NFA) starting from q _(NFA) is completed, the process proceeds to step S312. In cases other than that described here, process flow proceeds to Step S309 (Step S308).

If either condition of Char (t _(NFA) ) = σ and Char (t _(NFA) ) = σ _any is satisfied, the process proceeds to step S310. If neither condition is satisfied, the process proceeds to step S311 (step S309).

If the NFA state transition trans (q _(NFA) , Destination (t _{0 (NFA)} ), σ) is not included in the NFA state transition set T _(NFA) , the process proceeds to step S311 (step S310).

t _(NFA) is set as the next NFA state transition starting from the NFA state q _(NFA) , and the process proceeds to step S305 (step S311).

If at step S306 NFA state q _(NFA) is the initial state q ₀ of NFA _(NFA), or when treated with q all NFA state transitions t originating from the _{(NFA) (NFA)} in step S308 The NFA state q _(NFA) is set as the next NFA state included in the NFA state set Q _NFA , and the process proceeds to step S305 (step S312).

If all NFA states q _(NFA) have been processed in step S305, t _{0 (NFA)} is set as the next NFA state transition starting from the initial state q _{0 (NFA)} of NFA, and the process proceeds to step S302. (Step S313).

The procedure “Addition of failed transition to initial state (when there is no transition destination by σ _any from the initial state)” can be performed according to the above procedure.

Next, the procedure of “add failure transition to initial state (when there is a transition destination by σ _any from the initial state)” in step S203 will be described. The purpose of this procedure is the same as that of step S202, but the transition destination q _{1 (NFA)} by σ _any from the initial state is changed as in the regular expression (. | B | c (d | e)) f shown in FIG. If it exists, since it is possible to transition to q _{1 (NFA)} even if the transition is successful for all states, the initial state q _{0 (NFA)} of _NFA is excluded as shown in FIG. A transition to q _{1 (NFA)} is added to all NFA states q _(NFA) and all transition characters σ.

With reference to FIG. 12, the procedure of “add failure transition to initial state (when there is a transition destination by σ _any from the initial state)” in step S203 will be described.

First, t _{0 (NFA)} is set as the first NFA state transition starting from the initial state q _{0 (NFA)} of NFA generated in step S102, and the process proceeds to step S352 (step S351).

When the processing of all NFA state transitions t _{0 (NFA)} starting from the initial state q _{0 (NFA)} of the NFA is completed, the procedure is terminated. In cases other than that described here, process flow proceeds to Step S353 (Step S352).

σ is set to Char (t _{0 (NFA)} ), and the process proceeds to step S354 (step S353).

Let q be the first NFA state included in the NFA state set Q _(NFA) , and proceed to step S355 (step S354).

When the processing of all NFA states q included in the NFA state set Q _(NFA) is completed, the process proceeds to step S359. In cases other than that described here, process flow proceeds to Step S356 (Step S355).

When the NFA state q _(NFA) is the initial state q _{0 (NFA)} , the process proceeds to step S356. In cases other than that described here, process flow proceeds to Step S357 (Step S306).

If NFA state transition trans (q _(NFA) , Destination (t _{0 (NFA)} ), σ _(NFA) ) is not included in NFA state transition set T _(NFA) , in addition to T _(NFA) , go to step S355 (Step S358).

If all NFA states q _(NFA) have been processed in step S355, t _{0 (NFA)} is set as the next NFA state transition starting from the initial state q _{0 (NFA)} of NFA, and the process proceeds to step S352. (Step S359).

The procedure “Addition of failed transition to initial state (when transition destination by σ _any exists from initial state)” can be performed according to the above procedure.

Next, the procedure of “remove non-deterministic transition” in step S104 will be described. This procedure removes non-deterministic transitions contained in NFA and generates deterministic transitions. An example is shown in FIG. Since q ₁ and q ₂ exist from the state q _source as the state transition by the transition character a, that is, they are non-deterministic, the transition destination states are merged, that is, the transition destination NFA state set q _{1 as} shown in FIG. and the state transition to the union of _{q 2 q n = q 1 ∪q} 2. This procedure is similar to the procedure of generating DFA which are basically shown in Non-Patent Document 1, in the example of the present embodiment, further, as shown in FIG. 34, the state transition q ₂ by any character sigma _any In the case of including the end point q ₁ of the state transition by the transition character a, in addition to the state transition to the state q _n = q ₁ ∪q ₂ for the non-deterministic transition character a, the transition to the state q ₁ by the exclusion character σ _other Is generated (FIG. 35).

The procedure of “removing non-deterministic transition” in step S104 will be described with reference to FIG. The variable Retry is used in this procedure and the procedure of “removing non-deterministic transitions related to σ _other ”. The variable Retry can take either TRUE or FALSE values.

  First, a “state set initialization” procedure is performed, and the process proceeds to step S402 (step S401).

  Next, the variable Retry is initialized to FALSE, and the process proceeds to step S401 (step S402).

  Next, the procedure of “removing non-deterministic transitions regarding Σ” is performed. The process proceeds to step S404 (step S403).

Next, the procedure of “removal of non-deterministic transition related to σ _other ” is performed, and the process proceeds to step S405 (step S404).

  If the variable Retry is TRUE, the process proceeds to step S403. If FALSE, the process ends (step S405).

  According to the above procedure, the procedure “removing non-deterministic transition” can be performed.

Next, the procedure of “initialization of state set” in step S401 will be described. This procedure is for initializing the necessary set of states in order to generate DFA states. For all NFA states q _(NFA) , DFA states {q _(NFA) } are related. It is intended to generate with state transitions.

  The procedure of “initialization of state set” in step S401 will be described with reference to FIG.

  First, the state set Q is initialized to empty, and the process proceeds to step S502 (step S501).

Next, q _(NFA) is set as the first NFA state included in the NFA state set Q _(NFA) , and the process proceeds to step S502 (step S502).

If all the NFA states q _(NFA) included in the NFA state set Q _(NFA) have been processed, the process proceeds to step S506. In cases other than that described here, process flow proceeds to Step S504 (Step S503).

The NFA state set (ie, DFA state) {q _(NFA) } is added to the state set Q, and the process proceeds to step S505 (step S504).

Let q _(NFA) be the next NFA state included in the NFA state set Q _(NFA) , and proceed to step S503 (step S505).

When all the NFA states _(NFA) included in the NFA state set Q _(NFA) are processed in step S503, the state transition set T is emptied and the process proceeds to step S507 (step S506).

Next, t _(NFA) is set as the first NFA state transition included in the NFA state transition set T _(NFA) , and the process proceeds to step S508 (step S507).

When all the NFA state transitions t _(NFA) included in the NFA state transition set T _{( NFA)} are processed, the process proceeds to step S511. In cases other than that described here, process flow proceeds to Step S509 (Step S508).

Trans ({Source (t _(NFA) )}, {Destination (t _(NFA) )}, Char (t _(NFA) )) is added to the state transition set T, and the process proceeds to step S510 (step S509).

Let t _(NFA) be the next NFA state transition included in the NFA state transition set T _(NFA) , and proceed to Step S508 (Step S510).

When all the NFA state transitions t _(NFA) included in T _(NFA) are processed in step S508, the output description set D is emptied and the process proceeds to step S512 (step S511).

Next, d _(NFA) is set as the first NFA output description included in the NFA output description set D _(NFA) , and the process proceeds to step S508 (step S512).

When all the NFA output descriptions d _(NFA) included in the NFA output description set D _{( NFA)} have been processed, the process ends. In cases other than that described here, process flow proceeds to Step S509 (Step S513).

Add desc ({State (d _(NFA) )}, Result (d _(NFA) )) to the output description set D, and proceed to Step S510 (Step S514).

d _(NFA) is set as the next NFA output description included in the NFA output description set D _(NFA) , and the process proceeds to step S508 (step S515).

  The procedure “initialization of state set” can be performed by the above procedure.

  Next, the procedure of “removing non-deterministic transition regarding Σ” in step S403 will be described. In this procedure, as shown in FIG. 32 and FIG. 34, when there are a plurality of transition destinations for one transition character σ∈Σ, each transition character is replaced by a transition to a new state. The purpose is to uniquely determine the transition destination by σ.

  The procedure of “removal of non-deterministic transition regarding Σ” in step S403 will be described with reference to FIG. This procedure uses the variable Found. The variable Retry can take either TRUE or FALSE values.

  First, Found is initialized to FALSE, and the process proceeds to step S602 (step S601).

  Next, q is set as the first state transition in the state set Q, and the process proceeds to step S603 (step S602).

  If all the states q in the state set Q have been processed, the process proceeds to step S616. In cases other than that described here, process flow proceeds to Step S604 (Step S603).

  Let σ be the first alphabet in the input alphabet Σ, and go to step S605 (step S604).

  If all alphabets σ in the input alphabet Σ have been processed, the process proceeds to step S610. In cases other than that described here, process flow proceeds to Step S606 (Step S605).

There are multiple state transitions t starting with q and σ as a transition character, or there are state transitions t with σ as a transition character and state transition t with σ _any as a transition character. If is non-deterministic, the process proceeds to step S607. In cases other than that described here, process flow proceeds to Step S609 (Step S606).

The parameter “q _source ” is set to “q” and σ _t is set to “σ”, and the procedure “generation of new state” is performed, and the process proceeds to step S608 (step S607).

  TRUE is set in the variable Found, and the process proceeds to step S609 (step S608).

  Let σ be the next alphabet in the input alphabet Σ, and proceed to step S605 (step S609).

  When all the alphabets σ in the input alphabet Σ are processed in step S605, t is set as the first state transition starting from the state q, and the process proceeds to step S611 (step S610).

  If all the state transitions t starting from the state q have been processed, the process proceeds to step S615. In cases other than that described here, process flow proceeds to Step S612 (Step S611).

If the transition character Char (t) of t is σ _any , the process proceeds to step S613. In cases other than that described here, process flow proceeds to Step S614 (Step S612).

The transition character of t is replaced with σ _other , that is, t is replaced with (Source (t), Destination (t), σ _other ), and the process proceeds to step S614 (step S613).

  Let t be the next state transition starting from state q, and proceed to step S611 (step S614).

  When all the state transitions t starting from the state q are processed in step S611, q is set as the next state transition in the state set Q, and the process proceeds to step S603 (step S615).

  When all the states q in the state set Q are processed in step S603, if the variable Found is TRUE, the process proceeds to step S601. Otherwise, the process ends (step S616).

  With the above procedure, the procedure “removal of non-deterministic transitions regarding Σ” can be performed.

Next, the procedure “removal of non-deterministic transition related to σ _other ” in step S404 will be described. In this procedure, as shown in FIG. 36, there is a state transition from the state q _source to a plurality of states q ₁ and q ₂ for the transition by the transition character σ _other generated by “removing the non-deterministic transition regarding Σ”. In this case, non-deterministic transitions due to the transition character σ _other are removed as shown in FIG. 37 by replacing them with transitions to the state of the union q _n = q ₁ ∪q ₂ .

The procedure “removal of non-deterministic transition related to σ _other ” in step S404 will be described with reference to FIG. This procedure uses the variable Found. The variable Found can take either TRUE or FALSE values. In this procedure, the variable Counter is used. The variable Counter can take an integer value of 0 or greater.

  First, FALSE is set in the variable Found, and the process proceeds to step S702 (step S701).

  Let q be the first state in the state set Q, and go to step S703 (step S702).

  If all the states q in the state set Q have been processed, the process proceeds to step S714. In cases other than that described here, process flow proceeds to Step S701 (Step S703).

  The variable Counter is set to 0, and the process proceeds to step S705 (step S704).

  The first state transition starting from q is set in t, and the process proceeds to step S706 (step S705).

  If all the state transitions t starting from q have been processed, the process proceeds to step S710. In cases other than that described here, process flow proceeds to Step S707 (Step S706).

If the transition character Char (t) of t is σ _other , the process proceeds to step S708. In cases other than that described here, process flow proceeds to Step S709 (Step S707).

  1 is added to the variable Counter, and the process proceeds to step S709 (step S708).

  The next state transition starting from q is set in t, and the process proceeds to step S705 (step S709).

  When all the state transitions t starting from q are processed in step S706, if the value of the variable Counter is 2 or more, the process proceeds to step S711. In cases other than that described here, process flow proceeds to Step S713 (Step S710).

The parameter q _{source is set} to q and σ _t is set to σ _other , the procedure “create new state” is called, and the process proceeds to step S712 (step S711).

  TRUE is set in the variable Found, and TRUE is set in the variable Retry, and the process proceeds to step S713 (step S712).

  Let q be the next state transition in the state set Q, and go to step S703 (step S713).

  If all the states q in the state set Q have been processed in step S703, the process proceeds to step S701 if the variable Found is TRUE. Otherwise, the process ends (step S714).

By the above procedure, it is possible to perform the procedure "remove nondeterministic transition relates sigma _other".

Next, the procedure “procedure of new state” in steps S607 and S711 will be described. This procedure is a procedure for removing individual non-deterministic state transitions related to the transition character σ _t from the state q _source .

The procedure “procedure of new state” in steps S607 and S711 will be described with reference to FIG. The procedure “create new state” takes q _source and σ _t as parameters.

First, a set of states that can be transitioned by σ _t is obtained starting from the state q _source , and a union related to these NFA states is obtained to obtain a state q _n . If q _n is included in the state transition set T, the process proceeds to step S817. In cases other than that described here, process flow proceeds to Step S802 (Step S801).

Next, the state q _n is added to the state set Q, and the process proceeds to step S803 (step S802).

The t as the first state transitions starting from the q _source, the process proceeds to step S804 (step S803).

If all the state transitions t starting from q _source have been processed, the process proceeds to step S817. In cases other than that described here, process flow proceeds to Step S805 (Step S804).

If Char (t) = σ _t or Char (t) = σ _any , the process proceeds to step S806. In cases other than that described here, process flow proceeds to Step S816 (Step S805).

Let t ₁ be the first state transition starting from Destination (t), and go to step S807 (step S806).

If all the state transitions t ₁ starting from Destination (t) have been processed, the process proceeds to step S812. Otherwise, the process proceeds to step S808 (step S807).

If state transition trans (q _n , Destination (t ₁ ), Char (t ₁ )) ∈ T, add trans (q _n , Destination (t ₁ ), Char (t ₁ )) to T and add to step S809 Proceed (step S808).

If Char (t ₁ ) = σ _other , the process proceeds to step S810. Otherwise, the process proceeds to step S811 (step S809).

The procedure “correction of state transition by σ _other ” is called, and the process proceeds to step S811. At this time, (q _source , q _n , t ₁ ) is given as a parameter (step S810).

The t _1, the next state transition which starts the Destination (t), the process proceeds to step S807 (step S811).

If all state transitions t ₁ starting from Destination (t) are processed in step S807, d is set as the first output description in which Destination (t) is the output state, and the process proceeds to step S813 (step S812).

  If all output descriptions d having Destination (t) as an output state have been processed, the process proceeds to step S816. In cases other than that described here, process flow proceeds to Step S814 (Step S813).

If the output description desc (q _n , Result (d)) εD is not satisfied, desc (q _n , Result (d)) is added to D, and the process proceeds to step S815 (step S814).

  d is set as the next output description in which Destination (t) is the output state, and the process proceeds to step S813 (step S815).

The t as the next state transitions starting from the q _source, the process proceeds to step S804 (step S 816).

If q _n in step S801 is not included in the state transition set T, and the case of processing all the state transitions t which starts q _source in step S804, the first state transition to a start point q _source t Then, the process proceeds to step S818 (step S817).

If all the state transitions t starting from q _source have been processed, the process proceeds to step S822. In cases other than that described here, process flow proceeds to Step S819 (Step S818).

For Char _(t) = σ t proceeds to step S820. In cases other than that described here, process flow proceeds to Step S821 (Step S819).

  The state transition t is deleted from the state transition set T, and the process proceeds to step S821 (step S820).

The t as the next state transitions starting from the q _source, the process proceeds to step S818 (step S821).

If all state transitions t starting from q _source are processed in step S818, unless state transitions trans (q _source , q _n , σ _t ) ∈T, trans (q _source , q _n , σ _t ) Is added and the process ends (step S822).

  With the above procedure, the procedure “generation of a new state” can be performed.

Next, the procedure “correction of state transition by σ _other ” in step S811 will be described. As shown in FIG. 38, this procedure is performed when a state transition based on the transition character σ _other is included in the state transitions starting from the states q ₁ and q ₂ to be merged when removing the non-deterministic transition. The procedure for merging states that are the end points of state transitions starting from states q ₁ and q ₂ is shown. When the states q ₁ and q ₂ in FIG. 38 are merged, the result is as shown in FIG. 39, and the end point of the state transition by the state q _n = q ₁ ∪q ₂ transition character b is q ₃ ∪q ₅ . Similarly, when the states q ₁ and q ₂ shown in FIG. 40 are merged, as shown in FIG. 41, the end points of the state transitions by the transition characters b, c, and σ _other are q ₃ ∪q ₆ and q ₄ ∪q ₅ , respectively. Q ₄ ∪q ₆ .

The procedure “correction of state transition by σ _other ” in step S811 will be described with reference to FIG. In this procedure, a set of extended input alphabets σ∈Σ _s , CharSet is used.

  First, CharSet is initialized to empty, and the process proceeds to step S902 (step S901).

Next, t is the first state transition starting from Source (t _other ), and the process proceeds to step S903 (step S902).

If you are processing all the state transitions starting from the Source (t _other), the process proceeds to step S907. In cases other than that described here, process flow proceeds to Step S904 (Step S903).

If Char (t) is included in the input alphabet Σ, ie, Char (t) ≠ σ _any and Char (t) ≠ σ _other , the process proceeds to step S906. In cases other than that described here, process flow proceeds to Step S905 (Step S904).

  Char (t) is added to CharSet, and the process proceeds to step S906 (step S905).

Let t be the next state transition starting from Source (t _other ), and go to step S903 (step S906).

If all state transitions starting from Source (t _other ) are processed in step S903, t is the first state transition starting from state q, and the process proceeds to step S908 (step S907).

  When all the state transitions q starting from the state q have been processed, the process ends. In cases other than that described here, process flow proceeds to Step S909 (Step S908).

If Destination (t) ≠ Source (t _other ), the process proceeds to step S910. In cases other than that described here, process flow proceeds to Step S916 (Step S909).

The t ₁ as the first state transition which starts the Destination (t), the process proceeds to step S911 (step S910).

If all the state transitions t ₁ starting from Destination (t) have been processed, the process proceeds to step S916. In cases other than that described here, process flow proceeds to Step S912 (Step S911).

If Char (t ₁ ) is included in the input alphabet Σ, that is, Char (t ₁ ) ≠ σ _any and Char (t ₁ ) ≠ σ _other , the process proceeds to step S913. In cases other than that described here, process flow proceeds to Step S915 (Step S912).

If Char (t ₁ ) is included in CharSet, the process proceeds to step S915. In cases other than that described here, process flow proceeds to Step S914 (Step S913).

If state transition trans (q _n , Destination (t _other ), Char (t ₁ )) ∈T is not true, add trans (q _n , Destination (t _other ), Char (t ₁ )) to T, and go to step S915 (Step S914).

The t ₁ as the next state transition which starts the Destination (t), the process proceeds to step S911 (step S915).

If not a Destination (t) ≠ Source (t other) at step S909, or if processing all the state transitions t ₁ originating from the Destination (t) at step S911, starting from the state q the t And the process proceeds to step S908 (step S916).

With the above procedure, the procedure “correction of state transition by σ _other ” can be performed.

  Next, the procedure of “removing unused state” in step S105 will be described. In this procedure, a state that has occurred as a result of the processing so far and that does not become the end point of state transition, that is, a state that never reaches any input is deleted.

  The procedure of “removing unused state” in step S105 will be described with reference to FIG. This procedure uses the variable Found. The variable Found can take either TRUE or FALSE values.

  First, the variable Found is set to FALSE, and the process proceeds to step S1002 (step S1001).

  Next, q is set as the first state included in the state set Q, and the process proceeds to step S1003 (step S1002).

  If all the states q included in the state set Q have been processed, the process proceeds to step S1017. In cases other than that described here, process flow proceeds to Step S1004 (Step S1003).

If q is the initial state q ₀ , the process proceeds to step S1016. In cases other than that described here, process flow proceeds to Step S1005 (Step S1004).

  If there is a state transition with q as the end point, the process proceeds to step S1016. In cases other than that described here, process flow proceeds to Step S1006 (Step S1005).

  The variable Found is set to TRUE, and the process proceeds to step S1007 (step S1006).

  t is the first state transition starting from q, and the process proceeds to step S1008 (step S1007).

  If all the state transitions t starting from q have been processed, the process proceeds to step S1011. In cases other than that described here, process flow proceeds to Step S1009 (Step S1008).

  The state transition t is removed from the state transition set T, and the process proceeds to step S1010 (step S1009).

  Let t be the next state transition starting from q, and go to step S1008 (step S1010).

  When all the state transitions t starting from q are processed in step S1008, d is set as the first output description in which q is the output state, and the process proceeds to step S1012 (step 1011).

  If all output descriptions d having q as the output state have been processed, the process proceeds to step S1015. In cases other than that described here, process flow proceeds to Step 1013 (Step 1012).

  The output description d is deleted from the output description set D, and the process proceeds to step S1014 (step S1013).

  Let d be the next output description with q as the output state, and proceed to step S1012 (step S1014).

  When all output descriptions d having q as the output state are processed in step S1012, the state q is deleted from the state set Q, and the process proceeds to step S1012 (step S1015).

  Let q be the next state included in the state set Q, and go to step S1003 (step S1016).

  If all the states q included in the state set Q are processed in step 1003, the variable Found is checked, and if Found is TRUE, the process proceeds to step S1002. Otherwise, the process ends (step 1017).

  With the above procedure, the procedure “removal of unused state” can be performed.

  The procedure “removing unused state” is intended to reduce the memory capacity required to store the state transition table. Therefore, even if this procedure is omitted, the procedure “collation of input document” can be executed. By omitting this procedure, the time required for “compilation of collation conditions” can be shortened.

Next, referring to FIG. 20, the “redundancy removal” procedure in step S106 will be described. In this procedure, two kinds of unnecessary states are removed. The first case is a state transition having the same end point state as the state transition by σ _other . An example is shown in FIG. In FIG. 42, the operation of the SDFA automaton is the same even if the state transition whose end point is q ₃ by the transition character b is deleted. By deleting this state transition, the total number of state transitions is reduced, and the state transition table It is possible to reduce the memory capacity required to store the data. The second case is a combination of a plurality of states with the same end point of the state transition for all the transition characters. An example is shown in FIG. In FIG. 43, since the end points of the state transitions for all the transition characters are the same in the state q ₁ and the state q ₂ , the state q ₁ and the state q ₂ can be replaced by q _n = q ₁ ∪q ₂ . If a state or state transition is deleted by one of these two cases, it may be possible to delete the state or state transition by the other, so these will not be deleted until the state or state transition is no longer deleted. Repeat two types.

  The procedure of “redundant state removal” in step S106 will be described with reference to FIG. This procedure uses the variables StateRemoved and TransitionRemoved. The variables StateRemoved and TransitionRemoved can take either TRUE or FALSE values.

  First, TRUE is set in StateRemoved, and the process proceeds to step S1102 (step S1101).

  The procedure “delete redundant state transition” is called, and the process proceeds to step S1103. In this procedure, TransitionRemoved is set (step S1102).

  If TransitionRemoved = FALSE and StateRemoved = FALSE, the process ends. Otherwise, go to step S1104 (step S1103)

  The procedure “merging redundant states” is called, and the process proceeds to step S1105 (step S1104).

  If TransitionRemoved = FALSE and StateRemoved = FALSE, the process ends. In cases other than that described here, process flow proceeds to Step S1102 (Step S1105).

  The procedure “removing the redundant state” can be performed by the above procedure.

  Note that the procedure “removing the redundant state” is intended to reduce the memory capacity necessary for storing the state transition table. Therefore, even if this procedure is omitted, the procedure “collation of input document” can be executed. By omitting this procedure, the time required for “compilation of collation conditions” can be shortened.

Next, the procedure “Deletion of redundant state transition” in step S1102 will be described. This procedure corresponds to the first case of “redundant state removal”, and deletes the state transition having the same end point state as the state transition by σ _other as shown in FIG.

  The procedure “delete redundant state transition” in step S1102 will be described with reference to FIG.

  First, the variable TransitionRemoved is set to FALSE, and the process proceeds to step S1202 (step S1201).

  Next, t is set as the first state transition included in the state transition set T, and the process proceeds to step S1203 (step S1202).

  When all the states t included in the state transition set T have been processed, the process ends. In cases other than that described here, process flow proceeds to Step S1204 (Step S1203).

If Char (t) is σ _other , the process proceeds to step S1205. In cases other than that described here, process flow proceeds to Step S1210 (Step S1204).

Next, the t ₁ as the first state transitions starting from the Source (t), the process proceeds to step S1206 (step S1205).

If all the state transitions t ₁ starting from Source (t) have been processed, the process proceeds to step S1210. In cases other than that described here, process flow proceeds to Step S1207 (Step S1206).

If t ₁ ≠ t and Destination (t ₁ ) = Destination (t), the process proceeds to step S1208. In cases other than that described here, process flow proceeds to Step S1209 (Step S1207).

Remove the state transition t ₁ from the state transition set T, the processing flow advances to step S1209 (step S1208).

The t ₁ as the next state transition starting from the Source (t), the process proceeds to step S1206 (step S1209).

If Char (t) is not σ _other in step S1204, or if all state transitions t ₁ starting from Source (t) are processed in step S1206, t is included in the state transition set T. The next state transition is made, and the process proceeds to step S1203 (step S1210).

  With the above procedure, the procedure “delete redundant state transition” can be performed.

  Next, the procedure “merging redundant states” in step 1104 will be described. This procedure merges a plurality of states having the same end point of the state transition for all the transition characters as shown in FIG. 43 in the second case of “redundant state removal”.

  The procedure “merging redundant states” in step 1104 can be executed by a known procedure described in Non-Patent Document 1, for example. This procedure sets the variable StateRemoved to TRUE when one or more states are merged. Otherwise, set the variable StateRemoved to FALSE.

  Next, “generation of state transition table and output table” in step S107 will be described with reference to FIG. In this procedure, the state transition 11 and the output description 12 are extracted from the state set 25, the state transition set 26, and the output description set 27, and stored in the state transition table storage unit 4 and the output table storage unit 12, respectively.

  First, when the total number of states included in the state set Q is N, a unique state number StateId (q) of 0 to N-1 is associated with each state q (step S1301).

  The first state transition t is extracted from the state transition set T (step S1302).

  When all the state transitions t included in the state transition set T have been processed, the process proceeds to step S1307. In cases other than that described here, process flow proceeds to Step S1304 (Step S1303).

  The hash value calculation unit 31 calculates the hash value 32 for the set of the current state Source (t) and the input character Char (t), and the process proceeds to step S1305 (step S1304).

  A state transition hash chain 34 composed of a combination of the current state Source (t), the input character Char (t), and the next state Destination (t) is added to the state transition hash pointer 33 using the hash value 32 as an offset, and the process proceeds to step S1306. (Step S1305).

  t is set as the next state transition included in the state transition set T, and the process proceeds to step S1303 (step S1306).

  When all the state transitions included in the state transition set T are processed in step S1303, the first output description d is extracted from the output description set (step S1307).

  When all output descriptions d included in the output description set D have been processed, the process ends. In cases other than that described here, process flow proceeds to Step S130 (Step S1308).

  Result (d) is added as a condition number chain 42 to the condition number index 41 corresponding to the state number StateId (State (d)) of State (d), and the process proceeds to step S1310 (step S1309).

  Let d be the next output description included in the output description set D, and go to step S1308 (step S1310).

  The procedure “Generate state transition table and output table” can be executed by the procedure described above.

  Next, the procedure “collation of input document” will be described with reference to FIG.

First, to set the initial state q ₀ state q, the flow proceeds to step S2002 (step S2001).

  The output table storage unit 5 is searched, and the condition number 16 related to q is output. This procedure is realized by sequentially searching for a pointer to the condition number chain 42 corresponding to the state number StateId (q) of the current state 13 of the condition number index 41. When the search for all the condition number chains 42 is completed, the process proceeds to step S2003 (step S2002).

  Exit when all input is complete. In cases other than that described here, process flow proceeds to Step S2004 (Step S2003).

  The next input character 14 is received from the input character reading unit 7 and set to σ, and the process proceeds to step S2005 (step S2004).

Searching a state transition table storing unit 4, whether a transition destination q _d from the state q by the transition character sigma is present, i.e., checked trans (q, q _d, sigma) whether ∈T becomes q _d is present The process proceeds to step S2006 (step S2005).

When the transition destination q _d exists, the process proceeds to step S2007. In cases other than that described here, process flow proceeds to Step S2008 (Step S2006).

Set q _d to q, and proceed to step S2002 (step S2007).

If q _d is not present in step S2006 determines whether a transition destination q _d by transition character sigma _other from the state q exists, _{i.e., trans (q, q d,} σ other) is ∈T becomes q _d are present Check whether or not to proceed to step S2009 (step S2008).

When the transition destination q _d exists, the process proceeds to step S2007. In cases other than that described here, process flow proceeds to Step S2010 (Step S2009).

Set q ₀ to state q, and proceed to step S2006.

  With the above procedure, the procedure “verification of input document” can be performed.

  Hereinafter, the operation of the character string collating apparatus of the present embodiment will be described by taking the case of the collation condition 2 shown in FIG. 44 and the input character string shown in FIG. 54 as an example. The collation condition of FIG. 44 is a collation condition logically equivalent to FIG. 52, in which the collation condition notation of FIG. 52 shown in Patent Document 3 is matched with the format of this embodiment.

  By executing the procedure “Compilation condition compilation” shown in step S51, the state transition table shown in FIG. 45 and the output table shown in FIG. 46 are generated from the comparison condition 2 in FIG. Note that the initial state number is 0.

  The state transition table of FIG. 45 is a table representing the current state 13 and the next state 15 for the input character 14 stored in the state transition table storage unit 4. For example, the state number of the current state is 6 and the input character “ "d" indicates that the next state is 10. In the drawing, “-” indicates that the next state 15 does not exist, and the state transition table storage unit 4 can store such a combination without consuming memory. In the conventional finite state automaton with output, as shown in FIG. 53, 90 combinations are required, whereas in the example of FIG. 45, information may be stored for 46 state transitions, and a state transition table is stored. This has the effect of reducing the memory required to do this.

  The output table of FIG. 46 is a table representing a set of condition numbers 16 corresponding to the current state 13 stored in the output table storage unit 5. For example, when the current state is 4, condition number 0 is output. It is shown that.

As an example, FIG. 47 shows an operation when the input character string “aaca” is input. First, the state q is set to the initial state (state 0). Next, the first character “a” is read, and a transition is made to state 2 which is the transition destination of character “a” in state 0. Since the transition destination by the character “a” is defined, it is not necessary to refer to σ _other , and this is expressed as (unnecessary) in FIG. Next, the second character “a” is read, and the state transitions to state 5 which is the transition destination of character “a” in state 2. Next, the third character “c” is read, and a transition is made to state 9, which is the transition destination of character “c” in state 5. Next, the character “a” of the fourth character is read, and the state transitions to the state 12 that is the transition destination of the character “a” in the state 9. The input ends here.

  In the conventional technique described in Patent Document 3, it is necessary to refer to the state transition table seven times for the same condition. However, in the present embodiment, only four times are required. As described above, according to the present invention, the number of times of reference to the state transition table due to transition failure is set to 2 or less per character, performance degradation due to performance degradation due to repeated transition failure is prevented, and a high-speed character string The purpose is to enable verification.

  The collation conditions in FIG. 44 are described for comparison with the technique described in Patent Document 3, but have a special condition that all characters can be transitioned from the initial state. . The purpose of the collation condition in FIG. 44 is to make it possible to detect the character string even if one character of the collation target character string “abcd” changes, but the purpose and the meaning of the collation condition are almost changed. Without being converted to the collation conditions shown in FIG. In such a case, the memory required for the state transition can be further reduced, and the effects of the present embodiment can be further exhibited.

  By executing the procedure “Compilation Condition Compilation” shown in Step S51, the state transition table shown in FIG. 49 and the output table shown in FIG. 50 are generated from the collation condition 2 in FIG. Note that the initial state number is 0.

  The meaning of the state transition table of FIG. 49 is the same as that of FIG. 45, and the number of state transitions that need to be stored in the state transition table storage unit 4 is reduced to 23, and the effect of further reducing the necessary memory is obtained. ing.

FIG. 51 shows the operation for the input “xabxd” with respect to the matching condition 2 in FIG. Since the transition destination of the first character “x” is not defined in state 0, σ _other is referred to. However, since the transition destination is still not defined, the next state is the initial state, that is, state 0. At this time, the state transition table is referred to twice. Subsequently, the state transitions to state 3 with the second character “a” and to state 6 with the third character “b”. In the fourth character “x”, since the transition destination of “x” is not defined, σ _other is referred to and the next state 1 is obtained. At this time, the state transition table is referenced twice. Further, the state transits to the state 2 at the fifth character “d”. In the state 2, since the number 0 exists as the condition number 16 to be output, this is output. In this case, the reference count of the state transition table is 7 times.

  Note that, as described in Non-Patent Document 1, etc., a general DFA that does not have an output can be regarded as a special Moore machine that outputs two types of information, “accepted” and “not accepted” as output alphabets. it can. Also in this example, it is possible to configure a character string collating apparatus that outputs two types of information “accepted” and “not accepted” by simply determining whether or not a condition number exists in the collation result.

  In the above embodiment, the target of input and verification is “character”. However, “character” is not limited to a human-readable character string, and can be applied to any symbol string or data string. Is possible. For example, the present invention may be applied to identification of data measured by a gene sequence or a sensor.

  In the above embodiment, the state transition table storage unit 4 uses the state transition hash table 35. However, any data structure capable of logically expressing a two-dimensional table such as an array or a tree structure. It may be realized by.

  In addition, data structures such as arrays with high access speed are used for frequently used states such as the initial state, and hash tables and tree structures with high memory capacity are used for infrequently used states. Data structures may be used together.

  In the above embodiment, the condition number index 41 is used for the output table storage unit 5. However, any data structure that can logically express a one-dimensional table such as a tree structure or a hash table is used. It may be realized.

  The present invention can be applied to a character string matching device.

Claims (17)

A state transition table generating unit that generates a state transition table based on a matching condition described in a regular expression;
With an automaton that transitions based on the state transition table generated by the state transition table generation unit,
The state transition table generation unit sets an excluded character that transitions to a predetermined state when there is no next transition destination state for the set of the current state and the input character in the state transition table generated based on the matching condition A character string collating apparatus characterized by creating a state transition table.   The state transition table generation unit creates a state transition table that transitions to an initial state when there is no next transition destination state for the pair of the current state and the input character in the state transition table generated based on the matching condition The character string collating apparatus according to claim 1, wherein: When a plurality of collation conditions are input, the state transition table generation unit performs output description merging based on state merging, creates an output table,
The character string collating apparatus according to claim 1, wherein the automaton is output based on the output table. The state transition table generation unit replaces the arbitrary character with the exclusion character when the transition destination q ₁ by an arbitrary character and the transition destination q ₂ by a predetermined transition character σ exist from a certain state, and the predetermined transition character σ _2. The character string matching apparatus according to claim ₁ , wherein the transition destination is replaced with a state where q1 and q2 are merged.   The character string matching device according to claim 1, wherein the state transition table generation unit replaces a plurality of states with a merged state when there are a plurality of transition destination states by the excluded character from a certain state. When the state transition table generation unit merges the predetermined states q ₁ and q ₂ , and changes state from q ₂ to q ₃ by the exclusion character, the state transition destination and q ₃ starting from q ₁ The character string collating device according to claim 1, wherein the character string collating devices are merged.   The character string matching device according to claim 1, wherein the state transition table generating unit generates a state transition in which a transition by “.” Is an arbitrary character transition when generating the NFA. 2. The character string matching according to claim 1, wherein the state transition table generation unit replaces the transition by [^ a] with the transition by the character a and the transition by the excluded character σ _other when generating the NFA. apparatus. The character string matching device according to claim 1, wherein the state transition table generation unit deletes a state transition caused by a normal character when the normal character and the excluded character σ _other transition to the same state. When the normal character and the excluded character σ _other transition to the same state, after deleting the state transition by the normal character, merge multiple states that transition to the same state for all characters, and , After merging multiple states that transition to the same state for all characters, when the normal character and the excluded character σ _other transition to the same state, delete the state transition by the normal character The character string matching device according to claim 9.   The character string matching device according to claim 1, wherein the state transition table generation unit configures the state transition table with a hash table.   In the state transition table generated based on the matching condition, the automaton transitions to the initial state without reading the input character when there is no next transition destination state for the set of the current state and the input character. The character string matching device according to claim 1, wherein When a plurality of collation conditions are input, the state transition table generation unit performs output description merging based on state merging, creates an output table,
13. The character string matching apparatus according to claim 12, wherein the automaton is output based on the output table. When generating the state transition table, the state transition table generating unit is capable of transitioning from the initial state q ₀ to the predetermined state q ₁ with a predetermined transition character σ, and from the arbitrary state q to the predetermined transition character 13. The character string matching apparatus according to claim 12, wherein when the transition is possible by σ, a state transition from the arbitrary state q to the predetermined state q ₁ by the predetermined transition character σ is added.   The character string matching device according to claim 12, wherein the state transition table generation unit configures the state transition table by a hash table. A program for controlling a computer and checking character strings,
The state transition table generation unit generates a state transition table based on the matching condition described by the regular expression, and in the state transition table generated based on the matching condition, the next state for the set of the current state and the input character When there is no transition destination state, a process for creating a state transition table by setting an exclusion character to transition to a predetermined state;
A character string matching program that causes the computer to execute a process of making a transition based on a state transition table generated by the state transition table generating unit.   Transition is performed based on the state transition table generated by the state transition table generation unit, and in the state transition table generated based on the collation condition, there is no next transition destination state for the set of the current state and the input character The character string matching program according to claim 16, further comprising: causing the computer to execute a process of transitioning to an initial state without reading an input character.

Images