JP4118363B2 - Apparatus and method for collating multiple symbol strings based on sparse state transition table - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a collation apparatus and method for collectively judging whether or not at least one symbol string exists in given text data in a character string search apparatus or the like.
[0002]
[Prior art]
Today, in a document processing apparatus such as a word processor, it is required to collectively determine whether or not a set of a plurality of symbol strings exists as a search term in a text. Here, the symbol string means a sequence of characters and other symbols, and the character string is also a kind of symbol string. Such a determination function is often called multiple character string matching or multiple character string search.
[0003]
As a more efficient multi-string search device, Aho and MJ Corasick (“Afficient String Matching: An Aid to Bibliographic Search” CACM Vol.18 No. .6 1975), a method that constructs a Deterministic Finite Automaton (DFA) and a FAST (Flying Algorithm for Searching Terms) method proposed by Uraya (“Fast String Matching Algorithm” FAST "Information Processing Society Journal Vol.30 No.9 1989 Published Patent Publication No. 64-74619 Patent Application No. 62-231091).
[0004]
In the following, first, a multi-character string collation algorithm obtained by converting the AC method and the AC method into DFA will be described, and then a multi-character string collation algorithm of the FAST method will be described.
[0005]
The AC method is a method of matching character strings by configuring a finite state machine called a PMM (Pattern Matching Machine) for an input key set.
The collation operation in the AC method is as follows. First, the state number is set to “1” as an initial setting. Next, a symbol is read from the input text one character at a time, and it is determined by this input symbol which state is to be changed from the current state. If the transition by the input symbol is not defined for the current state, it is assumed that the verification has failed and the transition is made to the fail destination of the current state. If the transition by this input symbol is not defined for the state of the failure destination, the transition to the failure destination is repeated.
[0006]
Since transitions are defined for all symbols in the initial state “1”, the transition due to the failure stops in the initial state at the worst. Thus, the transition is repeated for the input symbol of the text. If a symbol string to be accepted for the state is defined, the symbol string and its position in the text are output.
[0007]
FIG. 64 shows an AC method PMM using three symbol strings {ab, bc, bd} as search keys. The PMM in FIG. 64 is composed of six states “1”, “2”, “3”, “4”, “5”, “6”, the solid line arrows indicate normal transition destinations, and the broken line arrows indicate It points to the failure destination. “^ A, b” represents an input symbol other than a and b. In the states “4”, “5”, “6” (s4, s5, s6), symbol strings “ab” are used as output keywords. "," Bc ", and" bd "are defined.
[0008]
The operation of this PMM for the input symbol string 'cabcz' is as shown in FIG. The initial state is “1”. First, when the symbol “c” is input, this corresponds to an input symbol other than a and b, so that the next state is the same state “1” and no output is generated. Next, when the symbol “a” is input, the state transitions to the next state “2”, and when the symbol “b” is input, the state transitions to the next state “4”. Here, the symbol string “ab” defined in the state “4” is output.
[0009]
However, since the transition destination is not defined in the state “4”, when the symbol “c” is input next, the state transitions to the state “3” of the fail destination, where the transition destination is searched. Then, since the state “5” is defined as the transition destination by the symbol “c”, the state transitions to that state, and the symbol string “bc” is output. Next, when the symbol “z” is input, the state transits to “1”, and the operation ends.
[0010]
Thus, in the AC method, the number of transitions increases by one every time a failure transition occurs due to an input symbol whose transition destination is not defined. Therefore, a finite state machine transition of less than 2n at the maximum is performed for n input symbols. In general, as the number of keys increases, the probability of hitting the first character of the key increases, but with this, the failure transition also increases. Therefore, the verification speed of the AC method gradually decreases as the number of keys increases. .
[0011]
The speed of the AC method is reduced in the failure transition where the transition destination is not defined, but in DFA, the state of the transition destination is uniquely determined for the input symbol. For this reason, finite state machine transitions are always performed n times for n input symbols, and the matching speed is high. Aho et al. Show a method for converting the AC state transition machine to DFA.
[0012]
FIG. 66 shows a finite state machine corresponding to the state transition machine of the AC method for the symbol string {ab, bc, bd}. In FIG. 66, “state” represents the current state, and “next” represents the transition destination state when the symbol described in “input” is input. The states s1, s2, s3, s4, s5, and s6 correspond to the states “1”, “2”, “3”, “4”, “5”, and “6”, respectively. For example, the notation “¬a, b” represents a symbol other than a and b.
[0013]
The operation of the finite state machine for the input symbol string 'cabcz' is as shown in FIG. The initial state is 1. In the state transition shown in FIG. 67, there is no failure transition as shown in FIG. 65, and the number of state transitions coincides with the number of symbols 5 included in the input symbol 'cabcz'.
[0014]
Also in the FAST method known as a high-speed collation method, as in the AC method, character strings are collated by configuring a PMM for the input key set.
The collating operation in the FAST method is as follows. First, the state number is set to “0” as an initial state. Also, the shortest key length in the input key set is set as the shortest key length, and the collation start position in the input text is set at a position separated from the head of the text by the shortest key length.
[0015]
Next, the symbols are read one character at a time from the collation start position toward the left of the text, and the state to be transitioned from the current state is determined by the input symbols. If the transition is not defined, the collation start position is shifted rightward by a specified amount corresponding to the input symbol, and collation is resumed.
[0016]
In this way, as long as state transition is possible with respect to the input symbol, the text is scanned from right to left to extract a character string pattern. If transition is impossible, the collation start position is shifted to the right in the text by the shift amount defined for the input symbol.
[0017]
FIG. 68 shows a FAST method PMM using three symbol strings {state, east, smart} as search keys. 68 has 14 states “0”, “1”, “2”, “3”, “4”, “5”, “6”, “7”, “8”, “9”, It consists of “10”, “11”, “12”, “13”, and the solid arrow indicates the transition destination, and the broken arrow indicates the shift destination.
[0018]
The state transition is defined in the reverse order of the sequence of symbols included in each search key, and “Depth” represents the depth of each state in the PMM. In the states “5”, “9”, and “13” (s5, s9, and s13), symbol strings “state”, “east”, and “smart” are defined as output keywords, respectively.
[0019]
FIG. 69 shows a table of transition destinations and shift amounts at the time of inputting symbols for the respective states of the PMM. In FIG. 69, the numbers in the first row represent state numbers, and the symbols in the first column represent input symbols. Here, “(Other)” represents an input symbol other than “a”, “e”, “m”, “r”, “s”, and “t”. In this table, the positive value element represents the state number of the transition destination by the corresponding input symbol, and the negative value element represents the shift amount by the corresponding input symbol.
[0020]
The operation of this PMM with respect to the input symbol string “aasestate” is as shown in FIG. The initial state is “0”. In this case, the shortest of the symbol strings “state”, “east”, and “smart” is “east” and its length is 4, so the shortest key length is 4. Therefore, collation is performed from the right to the left with the position “t” separated from the right end of the input symbol string by the shortest key length 4 as the collation start position.
[0021]
When collation fails, the shift amount defined for the input symbol is multiplied by −1 to obtain the magnitude of the shift amount, and the collation start position is shifted to the right by that amount. Then, the state number is set to “0” and the collation is resumed.
[0022]
When the symbol “t” is input in the initial state “0” indicated at the position of the symbol “t” in the input symbol string of FIG. 70, the state transitions to the state “6” according to the table of FIG. Next, when the symbol “s” is inputted, the state transits to the state “7”, and when the symbol “a” is inputted next, the state transits to the state “8”. Next, when the symbol “e” is input, the state transits to the state “9”, and the symbol string “east” defined in the state “9” is output.
[0023]
Next, when the symbol “s” is input, in the state “9” for this symbol, not the transition destination but the shift amount −7 is defined. Shift. Then, the state returns to the initial state “0”, and the collation is restarted with the position of the symbol “e” as the shift destination as a new collation start position. In the same manner, the collation is continued, and the symbol string “state” is output when the state transitions to the state “5”.
[0024]
The above-described multi-character string search process is used in each device such as a database, a word processor, and a full-text search device.
The full-text search device refers to a device that performs a character string search in order to confirm whether or not a search result is correct in a search using a full-text search index. Here, the full-text search index is not necessarily the correct answer for keywords for which the index itself is entered, such as a signature file or a file that does not have the appearance position of a word in a document (inverted file). It means a search index that doesn't always return.
[0025]
For example, consider a case where a keyword 'John Smith' is searched for an English index. Since the index unit is usually a word between spaces, 'John Smith' becomes the same as 'John AND Smith'. However, when a document is searched under the search condition of 'John AND Smith', even if 'John' and 'Smith' appear apart from each other, they are included in the search result and an excessive result is obtained. In such a case, whether the result is correct can be confirmed by a character string search.
[0026]
[Problems to be solved by the invention]
The problem in the conventional character string matching method described above is the relationship between the speed of the portion corresponding to the state transition of the PMM and the storage capacity.
[0027]
In the AC method, it is possible to reduce the storage capacity by using a list structure to represent the state transition portion. However, in the list structure, pointers must be followed in order, and the access processing is slow, so the collating operation is even slower.
[0028]
The collation speed of the AC method converted to DFA is high, but a table structure as shown in FIG. 66 must be used to represent all state transitions defined for all input symbols. However, this places a great burden on the storage capacity.
[0029]
For example, the number of input symbols is 256 (8-bit code), the number of states is N, and the pointer is 4 bytes. In the tabular format, one state requires 256 pointers to the next state, one pointer to the fail destination, and one pointer to the output symbol string. For this reason, a storage capacity of N * (256 + 1 + 1) * 4 bytes is required.
[0030]
In general, as the number of search keys increases, the number of states N also increases. Therefore, when the number of keys is large, the necessary storage capacity becomes enormous. Therefore, it is not realistic to configure a character string matching device based on the DFA-converted AC method.
[0031]
Similarly, in the FAST method, since state transitions or shifts are defined for all input symbols, a table structure as shown in FIG. 69 must be used. Therefore, when a character string collation apparatus is configured based on the FAST method, a huge storage capacity is still required.
[0032]
An object of the present invention is to provide a multi-symbol string collation apparatus and method capable of performing high-speed collation with a realistic storage capacity.
[0033]
[Means for Solving the Problems]
FIG. 1 is a principle diagram of a collation device according to the present invention. The collation apparatus of FIG. 1 is a collation apparatus in an information processing apparatus that uses a given symbol string as a key and determines whether or not the key exists in a file using a finite state machine, and is a state transition storage Means 1 and collation means 2 are provided.
[0034]
The state transition storage unit 1 is a state transition table that defines a collation operation related to at least one key, and a sparse state transition table in which data representing a predetermined operation is reduced in a compressed array format. Remember.
[0035]
  The collating means 2 performs an operation corresponding to each symbol included in the file while referring to the sparse state transition table, and collates the symbol string in the file with the one or more keys.At this time, the collation means 2 checks whether or not an operation for the input symbol input from the file is defined in the sparse state transition table, and when the operation for the input symbol is not defined, Perform the specified operation.
[0036]
For example, when a state transition table based on the DFA AC method is created, data representing a transition operation from the current state to the initial state and data representing a transition operation from the current state to the next state of the initial state Of these, at least one of the data is removed from the conventional state transition table to reduce the amount of information in the state transition table.
[0037]
Further, when creating a state transition table based on the FAST method, data representing a shift operation for returning the collation position in the file to the direction opposite to the collation direction is excluded from the conventional state transition table as much as possible. Reduce the amount of information in the transition table.
[0038]
As a result, a portion from which data is removed in the state transition table becomes an empty element, and a sparse state transition table in which elements are sparsely generated is generated. By compressing such a sparse state transition table and storing it in an array, a compact finite state machine can be constructed, and the storage capacity is greatly reduced.
[0039]
In addition, since the compressed state transition table is basically created based on the DFA state transition table, it is necessary to check whether transitions are defined, but it is input from the file. It is only necessary to perform one transition operation for each symbol, and the high speed of DFA is maintained.
[0040]
As a file to be collated, a file containing an arbitrary symbol string such as a document file described in text or a file obtained by converting voice data into a digital code can be used.
[0041]
For example, the state transition storage unit 1 in FIG. 1 corresponds to the state transition unit 122 in FIG. 5 of the embodiment, and the matching unit 2 corresponds to the state transition determination unit 121.
[0042]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 2 is a block diagram of a collation system based on the present invention. The collation system in FIG. 2 includes a compression device 101 and a collation device 102. The compression device 101 includes a keyword input unit 103, a sparse array finite state machine creation unit 104, and a state transition machine compression unit 105, and the collation device 102 includes a collation state transition machine unit 106 and a text input unit 107.
[0043]
The keyword input unit 103 accepts the input keyword group to be searched, and the sparse array finite state machine creation unit 104 performs sparse in the array format for the input keyword group for character string matching. The finite state machine is constructed on a binary tree as an intermediate structure. Here, the sparse array means an array containing almost no elements. The state transition machine compression unit 105 converts the intermediate structure created by the sparse array finite state machine creation unit 104 into a compressed array format with high collation speed.
[0044]
The matching state transition machine unit 106 performs matching between the text input from the text input unit 107 and the keyword group using a compressed array type finite state machine.
[0045]
FIG. 3 is a flowchart of the operation of the verification system of FIG. When the processing is started in FIG. 3, first, the keyword input unit 103 receives a keyword group to be searched (step ST1).
[0046]
Next, the sparse array finite state machine creation unit 104 determines a character string collating machine having an intermediate structure that has the collation speed of the same order as that of DFA for the input keyword and is sparse in the array format. It builds on (step ST2). The character string matching machine on the binary tree has a structure in which the addition and insertion of elements are easy although the matching speed is low.
[0047]
Next, the state transition machine compression unit 105 converts the character string matching machine on the binary tree into a compressed array format with high speed matching (step ST3). At this time, a character label for confirmation is attached to the portion where the element is present in the array, and the respective arrays are overlapped so that the portions where the elements of the plurality of arrays are not overlapped with each other. Create an array.
[0048]
Then, the verification state transition machine unit 106 performs verification on the input text using an array-type finite state machine.
A finite state machine that has the same collation speed order as the DFA-based AC method or array implementation FAST method for the input keyword group to be searched, but is sparse in the array format and can be compressed By constructing the above, a character string collating apparatus that has a high collation speed and can reduce the storage capacity is constructed.
[0049]
The construction method is summarized as follows.
A string matching machine is constructed as an intermediate structure on a binary tree that allows easy insertion and addition of elements and no storage capacity problems, and this intermediate structure is converted into a compressed array format that is fast and compact for collation. To do. At this time, a label for confirming the element is given to the element, and the array is compressed by overlapping the elements so that there is no overlapping of elements.
[0050]
FIG. 4 is a configuration diagram of an information processing apparatus (computer) that realizes the collation system of FIG. 4 includes a CPU (central processing unit) 111, a memory 112, an input device 113, an output device 114, an external storage device 115, a medium drive device 116, and a network connection device 117, each of which is a bus. 118 are connected to each other.
[0051]
The CPU 111 executes a program stored in the memory 112 to realize each process of the compression apparatus 101 and the collation apparatus 102. In addition to the above-described program, the memory 112 stores data used for processing. As the memory 112, for example, a ROM (read only memory), a RAM (random access memory), or the like is used.
[0052]
The input device 113 corresponds to, for example, a keyboard, a pointing device, and the like, and is used for inputting requests and instructions from the user. The output device 114 corresponds to a display device, a printer, or the like, and is used to output a state transition machine, a collation result, and the like.
[0053]
The external storage device 115 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, or the like. The above-described program and data can be stored in the external storage device 115, and loaded into the memory 112 for use as necessary. The external storage device 115 is also used as a database for storing keywords and text.
[0054]
The medium driving device 116 drives the portable recording medium 119 and accesses the stored contents. As the portable recording medium 119, an arbitrary computer-readable recording medium such as a memory card, a floppy disk, a CD-ROM (compact disk read only memory), an optical disk, a magneto-optical disk, or the like is used. The above-described program and data are stored in the portable recording medium 119 and can be loaded into the memory 112 and used as necessary.
[0055]
The network connection device 117 is connected to an arbitrary communication network such as a local area network (LAN) and performs data conversion associated with communication. The collation system communicates with an external information provider device 120 (such as a database) via the network connection device 117. As a result, the above-described program and data can be received from the apparatus 120 via the network and used by loading them into the memory 112 as necessary.
[0056]
Next, a first embodiment based on the DFA AC method will be described with reference to FIGS.
In the first embodiment, when the transition to the initial state and the transition to the next state of the initial state are not defined with respect to the table structure of the AC method converted to DFA, when the matching to these fails The state transition machine is redefined as a transition from the initial state.
[0057]
As described above, if the transition to the initial state is automatically performed when the transition is not defined, it is not necessary to store the transition to the initial state. In addition, since it is possible to make a transition to the next state after the initial state via the initial state, it is not necessary to store this transition. By omitting these transition definitions, elements of the DFA state transition table can be largely deleted, and a sparse state transition table can be obtained.
[0058]
Next, this state transition machine is made into an array form, and a plurality of arrays are overlapped so that portions where transitions are defined do not overlap. At the same time, in order to check whether or not the element is included in the array, the character is assigned as a label to the character for which the transition is defined.
[0059]
Also, when creating a compressed state transition table, the storage capacity actually used is saved by going through the intermediate form of the binary tree.
FIG. 5 is a configuration diagram of the collation device 102 of FIG. In FIG. 5, the state transition determination unit 121 and the state transition unit 122 correspond to the matching state transition machine unit 106. The text input unit 107 extracts symbols one character at a time from the target text, and the state transition determination unit 121 determines which state to transition to the input symbol.
[0060]
The state transition unit 122 corresponds to the memory 112, for example, and includes a compression state transition unit 123, a confirmation label unit 124, and an output symbol unit 125. The compressed state transition unit 123 stores the state transition table in a compressed array format, and the confirmation label unit 124 stores a label for confirming whether or not a transition is defined along with the compression. The output symbol portion 125 defines a symbol string that is output when a certain state is reached.
[0061]
FIG. 6 shows a first state transition table before compression when three symbol strings {ab, bc, bd} are used as input keywords. When this table is compared with the table of FIG. 66, the transition to the initial state s1 and the transition to the next states s2 and s3 of the initial state s1 are excluded from the transitions in the states s2, s3, s4, s5, and s6. .
[0062]
FIG. 7 shows the transition for each input symbol defined in the state transition table of FIG. In FIG. 7, “˜a, b” represents input symbols other than a and b.
FIG. 8 shows a collating array obtained by compressing the state transition table of FIG. In FIG. 8, index represents an array subscript, GOTO represents a superimposed state transition table stored in the compressed state transition unit 123, and CHECK represents a confirmation label stored in the confirmation label unit 124. OUTPUT represents an array of pointers stored in the output symbol portion 125. These pointers point to output character strings stored in the state transition unit 122.
[0063]
The element “#” in the array CHECK represents a terminal symbol. In addition, states s1 to s6 shown below the array in FIG. 8 indicate how the states in FIG. 6 overlap, and symbols a and b correspond to the input symbols in FIG. Represents the storage location of the transition destination.
[0064]
FIG. 9 shows a first character code conversion table used for explaining the first embodiment. According to this character code conversion table, the character code is converted into an internal code. Here, for the sake of simplicity, the range of the character code is from a to z of the alphabet, but any other symbols such as numbers can be used.
[0065]
Next, the matching process in the first embodiment will be described with reference to FIGS. 10 and 11.
FIG. 10 is a flowchart of character string matching processing based on the AC method. When processing is started in FIG. 10, the state transition determination unit 121 first sets a text pointer indicating the input text at the head thereof, and sets the transition pointer indicating the state transition array in the state transition unit 122 to the initial state. (Step ST11). Next, it is checked whether or not the text pointer points to the end of the text (step ST12). The collation ends when the text pointer points to the end of the text.
[0066]
If the text pointer does not point to the end of the text, the character pointed to by the pointer is taken out (step ST13), the value of the internal code corresponding to the character is added to the value of the transition pointer, and the added value is set as an index. It is checked whether the character label stored in is the same as this character (step ST14).
[0067]
If these characters do not match, no transition is defined for the input character. Therefore, the internal code of the input character is added to the pointer to the initial state, and the transition destination defined at the position where the added value is index is set as a new transition pointer (step ST15).
[0068]
If the label and the input character are the same in step ST14, it is next checked whether an output character string is defined for the state (step ST16).
If the output character string is not defined, the internal code of the input character is added to the current transition pointer, and the transition destination defined at the position where the added value is an index is set as a new transition pointer (step ST18). . Then, the text pointer is advanced by one character (step ST19), and the processes after step ST12 are repeated.
[0069]
If an output character string is defined, the character string is output as a collation result (step ST17), and the processes after step ST18 are performed. In step ST17, the current text pointer value can be output as the position of the collated character string.
[0070]
FIG. 11 shows the collation operation for the input text “cabcz” when the state transition array of FIG. 8 is used. In FIG. 11, “′ c” or the like represents the value of the internal code corresponding to the character c or the like. GOTO [x] represents the number of the transition destination defined at the position of index “x”, CHECK [x] represents the label stored at the position of index “x”, and OUTPUT [x ] Represents an output pointer stored at the position of index “x”. The collation in this case is performed as follows.
[0071]
In FIG. 8, the initial state corresponds to the position of index = 1. The first input symbol is “c” (step ST13). Since transition destinations for all input symbols are defined in the initial state, as indicated by “* 1” in FIG. 11, the check in step ST14 is omitted, and the process proceeds to step ST16. Here, since the output is not defined, the process of step ST17 is not performed.
Next, by subtracting the conversion table of FIG. 9 from the input symbol “c”, 'c = 3 is obtained. Therefore, the result of GOTO [1 + 3] = GOTO [4] is set as a transition destination to be advanced next. In FIG. 8, since the transition destination defined at the position of index “4” is “1”, GOTO [4] = 1. Therefore, the transition destination is again in the initial state corresponding to index “1” (step ST18).
[0072]
Next, when the collation operation is performed on the input symbol “a” in the same manner, GOTO [1 + ′ a] = 26, and the state starts from index “26” (step ST18).
[0073]
Next, for the input symbol “b”, it is necessary to access the array CHECK to check whether or not a transition is defined (step ST14). Therefore, when “b = 2” is added to the value 26 of the current transition pointer, index “28” is obtained. Since the label stored at the position of the index “28” is “b”, CHECK [28 + ’b] = b, and it can be seen that the transition for this input symbol is defined.
[0074]
At this time, since the output symbol string “ab” is defined in OUTPUT [26 + ’b], this is output (step ST17). The next transition destination is GOTO [26 + 'b] = 29.
[0075]
Next, the same processing is performed on the input symbol “c”, the symbol string “bc” is output, and a transition is made to the position of GOTO [29 + ’c] = 5.
Next, when the same processing is performed on the last input symbol “z”, since CHECK [5 + ′ z] is not z, the transition from the state of index “5” fails, and the symbol “z” A transition is defined as a transition from an initial state. As a result, the transition pointer becomes GOTO [1 + 'z] = 1 (step ST15), and since the text is finished, the collation operation is finished.
[0076]
Thus, the symbol strings “ab” and “bc” included in the input text are output as the collation results.
Next, a method for creating a state transition array used for collation will be described. FIG. 12 is a first block diagram of the compression apparatus of FIG. In FIG. 12, a binary tree conversion unit 131 and a transition addition unit 132 correspond to the sparse array finite state machine creation unit 104, and a conversion unit 133 corresponds to the state transition machine compression unit 105.
[0077]
The keyword input unit 103 receives the specified keyword group. The binary tree conversion unit 131 converts the accepted keyword group into a binary tree structure in the direction from the left to the right of each key. The transition adding unit 132 adds, to the created binary tree structure, a transition to the state other than the initial state and the next state when collation fails. The conversion unit 133 converts the binary tree structure output from the transition addition unit 132 into a compressed array format.
[0078]
The procedure for creating such an array for collation is as follows: creation of a binary tree from an input keyword group, addition of a fail destination node to a node of the resulting binary tree, and direct transition within a failure transition This includes the addition of a “gotoin / gotoout” destination node list that defines a thing, and the conversion of the finally obtained binary tree into an array format.
[0079]
FIG. 13 is a flowchart of the binary tree creation process. When the processing is started in FIG. 13, the binary tree conversion unit 131 first creates a binary tree for the input key (step ST21). Next, the corresponding input key is added as an output symbol string to the node that has received the key (step ST22), and the process ends.
[0080]
For example, when a binary tree is created for the keyword group {ab, bc, bd} corresponding to the state transition diagram of FIG. 7, the result is as shown in FIG. In FIG. 14, a rectangular box represents one node, and a character label added to each node corresponds to a symbol appearing in the keyword. The horizontal pointer represents a pointer to a node having the same depth on the binary tree, and the downward pointer represents a pointer to a deeper node. Here, output symbol strings “ab”, “bc”, and “bd” are added as outputs to the nodes “4”, “5”, and “6”, respectively.
[0081]
FIG. 15 is a flowchart of processing for assigning a transition destination of a failure transition to a created binary tree node. When the process is started in FIG. 15, the transition adding unit 132 first initializes the node queue Q that stores the node to be processed (step ST31).
[0082]
Next, a node that can transition from the root node of the binary tree (a node that becomes the next node with respect to the root node and a node that has the same depth as that node) is placed in the queue Q (step ST32).
[0083]
Next, the failure destination of the node in the queue is set as the root node (step ST33), and it is determined whether Q is empty (step ST34). If Q is empty, the process ends. If Q is not empty, then one node is taken out from the queue Q, set to r (step ST35), and the taken out node is removed from the queue Q (step ST36).
[0084]
Next, the node queue J is initialized (step ST37), and a node that can transition from r (a node that becomes the next node with respect to the node r and a node that has the same depth as that node) is placed in the queue J (step ST38). . Then, it is determined whether J is empty (step ST39).
[0085]
If J is empty, the process after step ST34 is repeated. If J is not empty, then one node is extracted from the node queue J, set to s (step ST40), and the extracted node is removed from the queue J (step ST41). Next, s is put in the queue Q (step ST42), the failure destination of the node r is set to t (step ST43), and whether or not the transition from the node t by the character label attached to the node s is defined. Is performed (step ST44).
[0086]
If no such transition is defined, the failure destination of t is set to t (step ST45), the determination of step ST44 is performed again, and a loop is repeated until the determination result is YES. Eventually, the loop can be exited in the initial state.
[0087]
If the transition is defined in step ST44, the destination of transition with the label from t to s is set as the failure destination of s (step ST46). Then, the output character string at the s fail destination is added as the output (output character string) of s (step ST47), and the processes after step ST39 are repeated.
For example, when the failure transition of the binary tree of FIG. 14 is calculated and assigned to each node of the binary tree, the result is as shown in FIG. The procedure of failure calculation in this case will be described along the flow of FIG.
[0088]
First, nodes “2” and “3” that can transition from the root node “1” are put in the queue Q (step ST32), and the fail destination of those nodes is set as the root node. The subsequent processing is as shown in FIG. In FIG. 17, for example, the notation of goto (1, b) represents a transition from the node “1” defined for the symbol “b”. Such a procedure is the same as the procedure for creating the failure function in the general AC method.
[0089]
FIG. 18 is a flowchart of processing for adding a transition to a state other than the initial state and the next state to the node of the binary tree as a gain / out node list for the added failure transition. When the process is started in FIG. 18, the transition adding unit 132 first initializes the node queue Q (step ST51).
[0090]
Next, a node that can transition from the root node of the binary tree is put in the queue Q (step ST52), and it is determined whether or not the queue Q is empty (step ST53). If the queue Q is empty, the process ends.
[0091]
If the queue Q is not empty, one node is taken out from the queue Q, set to r (step ST54), and the node is removed from the queue Q (step ST55). Next, all possible input symbols are entered in the queue X (step ST56), and it is determined whether the queue X is empty (step ST57). If the queue X is empty, the processes after step ST53 are repeated.
[0092]
For example, when an 8-bit code is used, the number of symbol codes to be placed in the queue X in step ST56 is 256 from 0 to 255.
If the queue X is not empty, one character is extracted from the queue X, set to s (step ST58), and at the same time, this character is removed from the queue X (step ST59). Then, it is determined whether or not the transition from r to the next state is possible by the symbol s (step ST60). If transition is possible, the transition destination of r by the symbol s is added to the queue Q (step ST65), and the processing after step ST57 is repeated.
[0093]
If the transition is not possible in step ST60, it is next determined whether the failure destination of the node r is the root node (step ST61). If it is the root node, the processing from step ST57 is repeated as it is.
[0094]
If the failure destination of the node r is not the root node in step ST61, it is next determined whether or not the transition by the symbol s is possible from the failure destination of r (step ST62). If the transition is impossible, the process after step ST57 is repeated. If transition is possible, next, a node that can be transitioned with symbol s from the fail destination of r is added to r's goout (step ST63), and node r is added to the gotoin of that node (step ST64). The processes after step ST57 are repeated.
[0095]
Now, assume that the processing target node r is r = A, the failure destination node of the node A by the symbol s is B, and a set of nodes that can transition from the node B is C. At this time, in the loop processing from step ST57 to step ST65 in FIG. Then, a list of nodes in the set C is added to the node A as “goout”, and the node A is added as “gotoin” to each node in the set C.
[0096]
FIG. 19 shows the values of goout and gotoin for the binary tree shown in FIG. The binary tree in FIG. 19 corresponds to an intermediate structure finite state machine that the transition adding unit 132 finally outputs.
[0097]
In the above-mentioned calculation of the gooutout list and the gotoin list, the transition can actually be defined only when the fail destination is a node other than the initial state. In FIG. 16, only the node “4” clearly satisfies this condition, so the processing for this node will be described.
[0098]
In FIG. 16, the failure destination of the node “4” is the node “3”. From node “3”, it is possible to transition to node “5” by character “c” and transition to node “6” by character “d”. Therefore, the goout of the node “4” is the node “5” and the node “6”. Also, the “goin” of the nodes “5” and “6” are both the node “4”. In FIG. 19, these gooutout and gotoin are represented in the form of labeled nodes.
[0099]
In the case of the conventional DFA, “goout” is defined for all nodes of the binary tree. However, in the present invention, “goout” is defined only for a node where a failure transition to a node other than the root node is defined. Is done. In this case, it is necessary to additionally define “gotoin”, but the failure transition of many nodes having the root node as the failure destination is deleted, which contributes to a reduction in storage capacity.
[0100]
20 and 21 are flowcharts of processing for converting the binary tree output by the transition adding unit 132 into a state transition machine in a compressed array format. When the processing is started in FIG. 20, the conversion unit 133 first initializes the arrays GOTO, CHECK, and OUTPUT to 0 (step ST71), and initializes the member index of the binary tree node to 0 (step ST72). .
[0101]
This index is provided independently of the index of the state transition array in order to store the correspondence between each node of the binary tree and the index of the state transition array as shown in FIG.
[0102]
Next, among the possible input symbols, those that cannot be transitioned from the root node to other nodes are put in the queue R (step ST73), and it is determined whether or not the queue R is empty (step ST74). If the queue R is not empty, one character is extracted from the queue R, set to s (step ST75), and removed from the queue R (step ST76).
[0103]
Next, GOTO [1 + 's] is set to 1 (step ST77), CHECK [1 +' s] is set to a character label of s (step ST78), and the processes after step ST74 are repeated. “′ S” represents the internal code in the array for s, but this may be the character code as it is.
[0104]
When the queue R becomes empty, the queue Q is then initialized (step ST79), and Pn = root node, Cn = next node of the root node, and Pp = 1 (step ST80).
[0105]
Here, the next node of the root node means a node having a minimum character label among a plurality of nodes (node string) that can be transitioned from the root node. In the case of a binary tree as shown in FIG. 19, the node put in Cn matches the node pointed by the pointer down from the root node.
[0106]
Next, a triplet of [Pn, Cn, Pp] is added to the queue Q (step ST81), and it is determined whether the queue Q is empty (step ST82). If the queue Q is empty, the process ends.
[0107]
If the queue Q is not empty, then a triplet is extracted from the head of the queue Q, set to s (step ST83), and the triplet is removed from the queue Q (step ST84). Next, the position on the array GOTO, CHECK, and OUTPUT where the node connected to the goout of the node Pn in s and the node connected to the node Cn in s and having the same depth as Cn can be inserted is obtained. Set to point (step ST85).
[0108]
At this time, the position of the point of the already inserted node is not overlapped with the position of the point of the newly inserted node. If a new insertable position is already used as a point, for example, it is shifted by one and set to a point.
[0109]
Next, GOTO [s Pp] = point is set (step ST86), a node connected to the goout of Pn of s is placed in the queue tmp (step ST87), and it is determined whether or not the queue tmp is empty (step ST88). .
[0110]
If the queue tmp is not empty, then one node is extracted from the queue tmp and set to i (step ST89), and the node is removed from the queue tmp (step ST90). Then, it is determined whether GOTO [i index] is 0 (step ST91).
[0111]
Here, the index of the position i on the state transition array in which the node i is stored, or 0 is stored in the index of the node i of the binary tree. If this is 0, node i of the binary tree has not yet been moved onto the state transition array.
[0112]
If GOTO [i index] is 0, the process of step ST88 is repeated. If it is not 0, the transition to the goout destination has been moved on the array, so the transition based on the goout is mapped on the array (step ST92), and the processes after step ST88 are repeated.
[0113]
In step ST92, transitions are mapped by setting GOTO [label of 'point +' i] = GOTO [index of i] and OUTPUT [label of 'point +' i]] = output of i. Here, “a label of“ i ”represents an internal code of a character label of the node i, and“ output ”of i represents an output symbol string defined in the node i. Thus, the output of node i is copied onto the array.
[0114]
If the queue tmp becomes empty in step ST88, next, a node connected to the gain of Pn in s is placed in the queue tmp (FIG. 21, step ST93), and it is determined whether or not the queue tmp is empty (step ST94).
[0115]
If the queue tmp is not empty, one node is taken out from the queue tmp, set to i (step ST95), and i is removed from the queue (step ST96). Then, it is determined whether GOTO [i index] is 0 (step ST97).
[0116]
If GOTO [i index] is 0, the process of step ST97 is repeated. If it is not 0, the transition of gotoout is mapped onto the array via gotoin (step ST98), and the processes after step ST97 are repeated.
[0117]
In step ST98, mapping is performed by setting GOTO [GOTO [index of i] + label of 'i] = point. If an output is defined for the node Pn in s, the output of Pn is copied onto the array as OUTPUT [GOTO [index of i] + 'i label] = Pn output.
[0118]
If the queue tmp becomes empty in step ST94, next, the character label of the node connected to goout of Pn and the character label of the node connected to Cn at the same depth are put in the queue chtmp (step ST99). Then, it is determined whether or not the queue chtmp is empty (step ST100).
[0119]
If chtmp is not empty, then one character label is taken out from the queue chtmp, set to j (step ST101), and the label is removed from the queue chtmp (step ST102). Then, CHECK [point + 'j] = j is set, the label of the node insertion destination is set (step ST103), and the processing after step ST100 is repeated.
[0120]
When chtmp becomes empty in step ST100, next, Cn in s and a node having the same depth are placed in the queue tmp (step ST104), and it is determined whether the queue tmp is empty (step ST105).
[0121]
If the queue tmp is not empty, a node is taken out from the queue tmp, set to i (step ST106), and the node is removed from the queue (step ST107). Next, the value of (point + 'i label) is set in the index of node i (step ST108), and it is checked whether or not output is defined for node i (step ST109).
[0122]
If there is no output in the node i, the process after step ST105 is repeated. If there is an output, the output of the node i is copied to the array OUTPUT (step ST110), and the processes after step ST105 are repeated. In step ST110, OUTPUT [label + 'i label] = i is output.
[0123]
If tmp becomes empty in step ST105, next, a node having the same depth as Cn is placed in the queue tmp (step ST111), and it is determined whether tmp is empty (step ST112).
[0124]
If the queue tmp is not empty, one node is extracted from the queue tmp, set to i (step ST113), and the node is removed from the queue tmp (step ST114). Then, it is determined whether or not the transition to the next state can be made by any symbol from the node i (step ST115).
[0125]
If the transition is impossible, the processes after step ST112 are repeated. If transition is possible, the triplet is added to the queue Q as the first node of the transition destination of Pn = i, Cn = i, and the index of Pp = i (step ST116), and the processing after step ST112 is repeated. Here, the leading node at the transition destination of i means a node having the smallest character label in a node string that can transition from the node i.
[0126]
If tmp becomes empty in step ST115, the process returns to step ST82 in FIG. 20, and the subsequent processing is repeated.
Next, a procedure for converting the binary tree of FIG. 19 into the array format as shown in FIG. 8 will be described along the flow of FIGS.
[0127]
First, since all transitions are defined for the initial state, this is defined. This process corresponds to the loop of steps ST74 to ST78 in FIG. As a result, an array format as shown in FIG. 22 is obtained.
[0128]
Thereafter, nodes on the binary tree are inserted into the array in order from the root node of the binary tree. This process corresponds to the loop from step ST79 in FIG. 20 to step ST116 in FIG. In this insertion process, the conversion unit 133 accumulates the triple data [Pn, Cn, Pp] for the nodes of the binary tree in a queue and inserts them into the array one by one.
[0129]
First, although inserted from the root node, in FIG. 20, the triples are stacked in the queue Q in steps ST79, ST80, and ST81, and these are taken out in steps ST13 and ST14. Here, Pn = root node “1”, Cn = node “2”, and Pp = 1. In step ST85, a place where insertion is possible is searched. This processing is as follows for the root node "1".
[0130]
In this case, Pn's goout is empty. Nodes having the same depth as Cn are node "2" and node "3". The character labels of these nodes are “a” and “b”, respectively. The operation of searching for a place on the array where the node string can be inserted corresponds to searching for a place corresponding to the pattern as shown in FIG. 23 on the arrays GOTO, CHECK, and OUTPUT.
[0131]
In the pattern of FIG. 23, the upper row represents a pattern on the current array CHECK, and the lower row represents a pattern to be inserted. “0” indicates that the area is empty, and “*” indicates that the area may be empty or an element may be included.
[0132]
When a position where such a pattern can be inserted is searched for on the array of FIG. 22, the pattern of FIG. 23 can be overlapped as shown in FIG. 24. Therefore, the value of point indicating the insertion position of the node is 1. Become. This position corresponds to the position immediately before the insertion position of the label “a” of the node “2”.
[0133]
In step ST86, GOTO [Pp] = point, so that GOTO [1] = 1. Thereby, the transition from the node “1” of the binary tree to the nodes “2” and “3” is moved onto the array. However, since the position of index = 1 corresponding to the root node “1” is GOTO [1] = 1 in advance, there is no change.
[0134]
Since the processing from step ST87 in FIG. 20 to step ST98 in FIG. 21 is processing when the node Pp has “outout / gotoin”, it is not related to the root node.
[0135]
The process from step ST99 to ST103 in FIG. 21 is a process for inserting a character label into the array. This is equivalent to setting the character label possessed by each node to the CHECK portion at the location on the previously secured array. By this operation, an array format as shown in FIG. 25 is obtained. In FIG. 25, the labels “a” and “b” with underlining correspond to the inserted labels.
[0136]
The processes from steps ST104 to ST110 in FIG. 21 are a process for setting the position on the array to a node of the binary tree and a process for copying the output of the node onto the array. As a result, 2 is set to the index of the node “2”, and 3 is set to the index of the node “3”.
[0137]
In addition, the loop processing from step ST111 to ST116 is processing for loading a node that can transition from Pp into the queue Q. Here, since the node “2” and the node “3” can transition from the root node, [Pn = node “2”, Cn = node “4”, Pp = 2] and [Pn = node “3”. , Cn = node “5”, Pp = 3] are put on the queue. Then, the process returns to step ST82 in FIG.
[0138]
This time, when Pn = node “2”, Cn = node “4”, and Pp = 2, the same processing is performed to find a position where the node can be inserted, and the position of point = 26 is found. Therefore, when the processing from step ST85 in FIG. 20 to step ST111 in FIG. 21 is performed in the same manner, an array format as shown in FIG. 26 is obtained.
[0139]
In FIG. 26, the label “b” of the node “4” is inserted at the position of index = 28, but the label “b” is converted into the internal code 2 when the conversion table of FIG. Therefore, a value 26 obtained by subtracting 2 from 28 is defined as a point. As a result, in step ST18 of the collation process of FIG. 10, it is possible to move from the position of index = 26 to the position of index = 28 by the input symbol “b” and transition to the transition destination represented by the element.
[0140]
Also, “ab” that is the output of the node “4” is copied to OUTPUT [28].
Next, since the node subsequent to the node “4” is the node “7”, [Pn = node “4”, Cn = node “7”, Pp = 28] is loaded on the queue Q. Then, the process returns to step ST82 in FIG.
[0141]
Next, what is taken out from the queue Q is [Pn = node “3”, Cn = node “5”, Pp = 3]. For this, it is only necessary to find an insertion location that satisfies the pattern as shown in FIG. 27. However, if the label “c” of the node “5” is matched with the index = 29 position, the internal code is 3. point = 26. Since this value has already been set to point once, in order to avoid duplication, one point is shifted to point = 27.
[0142]
In this way, when the processing from step ST85 in FIG. 20 to step ST111 in FIG. 21 is performed in the same manner, an array format as shown in FIG. 28 is obtained. In this case, “bc” that is the output of the node “5” is copied to OUTPUT [30], and “bd” that is the output of the node “6” is copied to OUTPUT [31].
[0143]
Next, nodes “8” and “9” follow the nodes “5” and “6”, respectively. Therefore, [Pn = node “5”, Cn = node “8”, Pp = 30], [Pn = node “6”, Cn = node “9”, Pp = 31] are accumulated in the queue Q. Then, the process returns to step ST82 in FIG.
[0144]
This time, the same processing is performed with [Pn = node “4”, Cn = node “7”, Pp = 28]. A search is made for a position where a node can be inserted. Since transitions by characters “c” and “d” are defined as goout in the node “4”, the place where the node can be inserted is a place that satisfies the pattern shown in FIG. It becomes. The value of point corresponding to this place is 29.
[0145]
At this time, the processing from step ST85 in FIG. 20 to step ST111 in FIG. 21 is performed in the same way. However, in this case, since “out” is defined for the node “4” to be Pn, the step in FIG. The loop processing from ST87 to ST92 is started. However, since GOTO [index of node “5”] and GOTO [index of node “6”] are both undefined, the process of step ST92 is not performed. Eventually, the resulting array is as shown in FIG.
[0146]
Since there is no node that can transition from the node “7”, no more nodes are loaded in the queue Q, and the process returns to step ST82 in FIG.
Next, [Pn = node “5”, Cn = node “8”, Pp = 30] is extracted from the queue Q. The place where insertion is possible is a place that satisfies the pattern as shown in FIG. The value of point corresponding to this place is 5.
[0147]
At this time, the processing from step ST85 in FIG. 20 to step ST111 in FIG. 21 is performed in the same way. In this case, however, node “4” is defined as “gotoin” in node “5” serving as Pn. 21 enters the loop processing from steps ST93 to ST98 in FIG.
[0148]
In the condition determination in step ST97, GOTO [index of node “4”] = 29, so in step ST98, GOTO [29 + ’c] = GOTO [32] = 5. In addition, since “bc” is defined as output in the node “5” which is Pn, this is also copied, and OUTPUT [32] = bc. As a result, an arrangement as shown in FIG. 32 is obtained.
[0149]
Finally, when the node “9” is processed in the same manner as the node “8”, the result shown in FIG. 8 is obtained.
In this example, in order to increase the compression ratio of the state transition array, the minimum allowable value is used as the point. However, an index larger than that value may be used as the point.
[0150]
Next, an embodiment of a character string replacement device in which a character string replacement function is added to the collation device of the first embodiment will be described. Here, an example in which the input keywords {ab, bc, bd} are replaced with {aaa, bbb, ccc}, respectively.
[0151]
This character string replacement function is a function for storing the location where the keyword is detected instead of outputting the keyword detected from the input text and replacing the keyword after processing the text.
[0152]
FIG. 33 shows a state transition array for character string replacement for the input key. In FIG. 33, the array GOTO stores an overlaid state transition table, the array CHECK stores a confirmation label, the array SUBST stores a pointer to a replacement character string, and the array LENGTH is a verified replacement. Stores the length of the previous string.
[0153]
FIG. 34 shows the initial state of the text offset storage array used for the replacement process. This array stores a text offset indicating the position of the character string to be replaced in the text, a SUBST pointer indicating the corresponding replacement character string, and the length of the replacement target character string.
[0154]
The text offset for each symbol in the input text 'cabcz' is as shown in FIG. Also, after pattern matching is performed on this input text, the text offset storage array is as shown in FIG. The collation process at this time is the same as in FIG. As a result of the collation, the positions of the keywords “ab” and “bc” in the text are stored as text offsets “1” and “2”, respectively.
[0155]
In the replacement process, the character string replacement device sets a pointer in each of the text and the text offset storage array. Then, text characters are output unless there is a pointer to the text in a section corresponding to the replacement target character string length starting from the position of the text offset stored in the text offset array. When the pointer to the text is at the text offset position in the text offset array, the corresponding replacement character string is output.
[0156]
FIG. 37 is a flowchart of this replacement process. When the processing is started in FIG. 37, the character string replacing device first sets the text pointer t at the head of the text (step ST121), and sets the replacement pointer p at the head of the text offset storage array (step ST122). Then, it is determined whether or not the pointer t points to the end of the text (step ST123). If the pointer t points to the end of the text, the process ends.
[0157]
If the pointer t does not point to the end of the text, the value of the text offset stored at the position pointed to by the pointer p is compared with the pointer t (step ST124). If they do not match, the character in the text pointed to by the pointer t is output (step ST125), the pointer t is advanced by one character (step ST126), and the processing after step ST123 is repeated.
[0158]
If the two match in step ST124, then the SUBST pointer stored at the position pointed to by pointer p is taken out, and the replacement character string pointed to by it is output (step ST127). Next, the pointer t is advanced by the length of the replacement target character string stored at the position indicated by the pointer p (step ST128), and the pointer p is advanced by one (step ST129).
[0159]
Then, it is determined whether the value of the pointer t is larger than the text offset at the position pointed to by the pointer p and whether the position pointed to by the pointer p is not the end of the text offset storage array (step ST130). If the determination result is YES, the processing after step ST129 is repeated, and if it is NO, the processing after step ST123 is repeated.
[0160]
As a result of such replacement processing, the text “cabcz” is converted to “caacz”.
Next, a second embodiment based on the FAST method will be described with reference to FIGS.
[0161]
In the state transition table of the FAST method, the data defining the operation for the input symbol of each state seems to be easily separated into two parts, default shift and transition, but this is not necessarily the case. For example, in the state transition table of FIG. 69, the shift amount for the input symbol “a” in the state “6” is −2 (size 2), but the default shift amount for the other input symbols is −5 (size 5). ). The former small shift amount is determined during calculation of the FAST method shift amount. For this reason, it is impossible to make the table format compact by separating the table format into the default shift amount and the transition to the next state.
[0162]
Therefore, in the second embodiment, first, operations using FAST method input symbols are divided into three types. The first operation is a default shift of state for general input symbols. The next operation is a shift for a particular input symbol. The last operation is a transition to the next state by an input symbol.
[0163]
In the collation operation, it is first determined whether or not a transition or shift is defined for the input symbol. If neither shift nor transition is defined, the pointer is shifted to the right according to the default shift. If an input symbol is accepted but a shift is defined instead of a transition, the pointer is shifted to the right according to the shift. If a transition is defined, transition to the next state.
[0164]
Such an operation is represented in an array format for each state, and the arrays are overlapped so that the portions where transition / shift are defined do not overlap. At the same time, in order to confirm whether or not an element is included in the array, the character is assigned as a label to a character for which transition / shift is defined.
[0165]
Also, when creating a compressed table format, the storage capacity actually used is saved by going through the intermediate structure of the binary tree.
The configuration of the collation device in the second embodiment is the same as that in FIG. In this case, the state transition determination unit 121 determines which state is to be transitioned with respect to the input symbol, or how much is to be shifted in the text.
[0166]
FIG. 38 shows a second state transition table before compression when three symbol strings {smart, east, state} are input keywords. When this table is compared with the table of FIG. 69, typical shift amounts frequently used in each state are combined into one as a default shift (default shift), and only a specific shift amount or a transition destination for a specific input symbol. Is defined for each input symbol.
[0167]
FIG. 39 shows the transition for each input symbol defined in the state transition table of FIG. The state transition diagram of FIG. 39 is composed of 14 nodes of nodes “0” to “13”. Data a1, a2, a3, and a4 assigned to each node represent a state number (node number), a default shift size, a specific character, a specific shift size, and an output character string, respectively. Yes.
[0168]
FIG. 40 shows a collation array obtained by compressing the state transition table of FIG. In FIG. 40, index represents an array subscript, GOTO represents a superimposed state transition table stored in the compressed state transition unit 123, and CHECK represents a confirmation label stored in the confirmation label unit 124. OUTPUT represents an array of pointers stored in the output symbol portion 125. These pointers point to output character strings stored in the state transition unit 122.
[0169]
If the element of the array GOTO is less than 0, it represents the shift amount, and otherwise represents a transition to the next state.
Further, states s0 to s13 shown below the array in FIG. 40 indicate how the states “0” to “13” in FIG. 38 overlap. Symbols a, e, and the like represent storage locations of access destination data corresponding to the respective input symbols in FIG. 38, and symbol D represents a storage location of access destination data corresponding to the default shift.
[0170]
FIG. 41 shows a second character code conversion table used for explaining the second embodiment. According to this character code conversion table, the character code is converted into an internal code. In the conversion table of FIG. 41, “1” is set as the internal code in the first column in order to set the default shift code at the top of each superimposed array. For this reason, the internal code for each input symbol is 2 or more, and is set in the second column and thereafter.
[0171]
The result obtained by adding 1 to the character code value of the input symbol may be used as the internal code.
Next, collation processing in the second embodiment will be described with reference to FIGS. 42 and 43.
[0172]
FIG. 42 is a flowchart of character string matching processing based on the FAST method. When the processing is started in FIG. 42, the state transition determination unit 121 first sets a text pointer indicating the input text at a position where the shortest key length is added to the head of the text pointer, and the state transition in the state transition unit 122 A transition pointer pointing to the array is set to an initial state (step ST131). Next, it is checked whether or not the text pointer points to the end of the text (step ST132). The collation ends when the text pointer points to the end of the text.
[0173]
If the text pointer does not point to the end of the text, the character pointed to by the pointer is taken out (step ST133), the value of the internal code corresponding to the character is added to the value of the transition pointer, and the added value is set as an index. It is checked whether the character label stored in is the same as this character (step ST134).
[0174]
If these characters do not match, then no transition or shift is defined for the input character. Therefore, the text pointer is advanced by the default shift for the state (step ST135), the transition pointer is set to the initial state (step ST136), and the processes after step ST132 are repeated.
[0175]
If the label and the input character are the same in step ST134, it is next checked whether or not a shift by the input character is defined (step ST137). If the input character is shifted, in the current state, the text pointer is advanced by the amount of shift defined for the input character (step ST138), and the transition pointer is set to the initial state (step ST139). ), The process after step ST132 is repeated.
[0176]
In step ST137, when the transition by the input character is defined instead of the shift, it is checked whether the output character string is defined for the state (step ST140).
[0177]
If the output character string is not defined, the internal code of the input character is added to the current transition pointer, and the transition destination defined at the position where the added value is an index is set as a new transition pointer (step ST142). . Then, the text pointer is returned by one character (step ST143), and the processes after step ST132 are repeated.
[0178]
If an output character string is defined, the character string is output as a collation result (step ST141), and the processes after step ST142 are performed. In step ST141, the current text pointer value can be output as the position of the collated character string.
[0179]
FIG. 43 shows a collation operation for the input text ‘aseastasterr’ using the state transition array of FIG. 40. First, the text pointer is set at a position that is the shortest keyword length 4 from the beginning of the text, and the transition pointer is set to the index value “1” corresponding to the initial state.
[0180]
For the first input symbol “e”, CHECK [1 + ′ e] = e and GOTO [1 + ′ e] = 3> 0, so the transition pointer = 3 (step ST142) and the next state is entered. Transition.
[0181]
Thereafter, the collating operation as shown in FIG. 43 is performed in the same manner. Here, first, whether or not the transition destination or the shift amount by the input symbol is defined is checked by accessing the array CHECK, and if the transition is not defined, the shift is performed by the default shift.
[0182]
When a shift smaller than the default shift is defined for a specific input symbol, the shift is performed according to the shift amount. In FIG. 43, such a specific shift occurs when the symbol “r” is input in the final initial state “1”. For the symbol “r”, the text pointer is shifted rightward by a shift amount 1 which is smaller than the default shift amount 4, but this moves to the end of the text, so the processing ends. Yes.
[0183]
Thus, the symbol strings “east” and “state” included in the input text are output as the collation result.
Next, a method for creating a state transition array used for collation will be described. FIG. 44 is a second block diagram of the compression apparatus of FIG. 44, a binary tree conversion unit 141, a preprocessing unit 142, and a shift amount calculation unit 143 correspond to the sparse array finite state machine creation unit 104, and the conversion unit 144 corresponds to the state transition machine compression unit 105.
[0184]
The keyword input unit 103 receives the specified keyword group. The binary tree conversion unit 141 converts the accepted keyword group into a binary tree structure in the direction from the left to the right of each key. The preprocessing unit 142 sets the length of the input keyword having the shortest length, the depth of each node, and the terminal node for each keyword. The shift amount calculation unit 143 calculates the shift amount for each state. Further, the conversion unit 144 converts the binary tree structure output from the shift amount calculation unit 143 into a compressed array format.
[0185]
The procedure for creating such an array for collation includes the creation of a binary tree from the input keyword group, the addition of the node depth to the resulting binary tree node, and the setting of the last node for the keyword , Calculation of shift amount, creation of binary tree of intermediate structure, and conversion from intermediate structure to array format.
[0186]
FIG. 45 is a flowchart of the binary tree creation process. When the processing is started in FIG. 45, the binary tree conversion unit 141 first creates a binary tree in the reverse order of the key sequence for the input key (step ST151). Next, the corresponding input key is added as an output symbol string to the node that has received the key (step ST152), and the process ends.
[0187]
FIG. 46 is a flowchart of the preprocessing for calculating the shift amount for the binary tree. When the processing is started in FIG. 46, the preprocessing unit 142 first sets the distance (depth) from the root node for each node (step ST161). Next, the shortest input keyword length is obtained (step ST162), the last node for each keyword is obtained (step ST163), and the process ends.
[0188]
For example, when a binary tree is created for the keyword group {smart, east, state} corresponding to the state transition diagram of FIG. 39 and pre-processing is performed, a binary tree as shown in FIG. 47 is obtained. In FIG. 47, the meanings of the downward pointer and the horizontal pointer are the same as those in FIG.
[0189]
Data d given to each node represents the depth of the node, and output represents an output character string. In this case, output symbol strings “state”, “east”, and “smart” are added as outputs to the nodes “5”, “9”, and “13”, respectively.
[0190]
Further, 4 is set as the shortest pattern length of these output symbol strings (keywords), and the nodes “5”, “9”, “13” are the termination nodes for the keywords “state”, “east”, “smart”, respectively. "Is set.
[0191]
48, 49, and 50 are flowcharts of the calculation process for obtaining the shift amount for the binary tree.
FIG. 48 shows a process for setting a shift by a specific character and a failure transition to each node. When the processing is started in FIG. 48, the shift amount calculation unit 143 first initializes a node queue Q that stores nodes to be processed (step ST171). Next, a node that can transition from the root node of the binary tree is placed in the queue Q (step ST172).
[0192]
Next, the failure destination of the node in the queue is set as the root node (step ST173), and it is determined whether Q is empty (step ST174). If Q is empty, the process ends. If Q is not empty, then one node is taken out from queue Q, set to r (step ST175), and the taken out node is removed from queue Q (step ST176).
[0193]
Next, the node queue J is initialized (step ST177), and a node that can transition from r is put in the queue J (step ST178). Then, it is determined whether J is empty (step ST179).
[0194]
If J is empty, the process after step ST174 is repeated. If J is not empty, then one node is extracted from the node queue J, set to s (step ST180), and the extracted node is removed from the queue J (step ST181). Next, s is put in the queue Q (step ST182), the failure destination of the node r is set to t (step ST183), and whether or not the transition from the node t by the character label attached to the node s is defined. Is performed (step ST184).
[0195]
If such a transition is defined, then the destination of transition from t to s is the destination of s (step ST188), and the processing after step ST179 is repeated.
[0196]
If no transition is defined in step ST184, then whether or not the shift amount at the label of s is not defined at node t, or whether or not the shift amount already defined is greater than the current depth of s. Is determined (step ST185).
[0197]
If the shift amount is undefined or the defined value is larger than the current depth of s, then the shift amount at the label of node s relative to node t is set as the depth of s (step ST186). The shift amount for such a specific label is represented as a specific list added to each node of the binary tree. Next, the failure destination of t is set to t (step ST187), and the processes after step ST184 are repeated.
[0198]
In step ST185, if a shift amount equal to or smaller than the current depth of s is defined for the label of s, the processing after step ST187 is performed.
FIG. 49 is a flowchart of processing for adding a default shift amount, which is the maximum shift amount, to each node. When the processing is started in FIG. 49, the shift amount calculation unit 143 sets a value obtained by adding the shortest key length to the node depth as the default shift amount for each node (step ST191), and ends the processing. .
[0199]
For example, in the case of the binary tree in FIG. 47, first, the shift amount calculation unit 143 places nodes “1” and “6” that can transition from the root node “0” into the queue Q by the processing in step ST172 in FIG. Next, the failure transition destinations of the nodes “1” and “6” are set to the root node by the process of step ST173.
[0200]
Next, the node “1” is taken out from the queue Q and set to r, and the node “2” is loaded on the node queue J by the processing of steps ST175 to ST178. Since only the node “2” is loaded on J, the process goes through the loop of steps ST179 to ST183 only once.
[0201]
Here, the node “2” is set to s by the process of step ST180, and the fail destination of the node r (node “1”), that is, the root node is set to t by the process of step ST183.
[0202]
Since it is possible to transition from the root node with the label of s (the label “t” of the node “2”), the process proceeds to step ST188, and the failure destination of s is the label “t” of the node “2” from the root node. The node “6” that can be transitioned by is obtained.
[0203]
Next, the node “6” is taken out from the queue Q. The same processing is performed on the node “7” that can be transitioned from the node “6”. In this case, r = node “6”, s = node “7”, and t = node “1” (fail destination of node “6”). However, transition from t to the label “s” of the node “7” is impossible.
[0204]
Therefore, in the node “1”, the depth of the node “6”, that is, 1 is set as the specific shift amount with respect to the input symbol “s”. The shift amount “1” corresponding to this symbol “s” is represented by a list structure as a specific list.
[0205]
If such a process is repeated in the same manner, a failure transition and a provisional shift amount are assigned to all nodes. Thereafter, according to the process of FIG. 49, a default shift amount that is the maximum shift amount in each node is assigned to each node. In this way, a binary tree as shown in FIG. 51 is obtained.
In FIG. 51, a broken-line arrow represents a failure transition, and a failure destination of a node for which no failure transition is displayed is a root node “0”. Data D added to each node represents a default shift amount. Furthermore, a specific character label and a specific shift amount corresponding thereto are set in the specific list connected to the nodes “0” and “6”.
[0206]
FIG. 50 is a flowchart of processing for reducing the shift amount of each node as necessary and assigning the final shift amount. When the processing is started in FIG. 50, the shift amount calculation unit 143 first loads the input keyword on the queue Q (step ST201), and determines whether the queue Q is empty (step ST202).
[0207]
If the queue Q is not empty, the keyword popped from the queue Q is put into j (step ST203). Next, the last node corresponding to the keyword j is set to jst (step ST204), the fail destination of jst is set to bst (step ST205), and jlen = jst depth (step ST206). Then, it is determined whether or not bst is a root node (step ST207). If bst is the root node, the processes after step ST202 are repeated.
[0208]
If bst is not the root node, the depth of bst is subtracted from jlen, and the result is set to slen (step ST208). Next, bst is put in the queue N (step ST209), and it is determined whether or not the queue N is empty (step ST210). If the queue N is empty, the processing after step ST207 is repeated with bst = bst as the fail destination (step ST211).
[0209]
If the queue N is not empty, the node popped from the queue N is put in r (step ST212), and it is determined whether or not the default shift of r is larger than (slen + r depth) (step ST213). If the default shift of r is equal to or less than (slen + r depth), the processes after step ST210 are repeated.
[0210]
If the default shift of r> (slen + r depth), then the default shift of r = slen + r depth (step ST214), nodes that can transition from r are loaded in the queue N (step ST215), and after step ST210 Repeat the process.
[0211]
If the queue Q is empty in step ST202, loop processing from step ST216 to ST220 is performed. This process is a process in which, when the shift size by a specific character defined for the node is larger than the default shift size, the shift size is cut to the same size as the default shift.
[0212]
Here, first, all nodes are placed in the queue Q (step ST216), and it is determined whether or not the queue Q is empty (step ST217). If the queue Q is empty, the process ends.
[0213]
If the queue Q is not empty, the node popped from the queue Q is put in j (step ST218), and whether or not the shift size for the specific character defined in j is larger than the default shift size. Is determined (step ST219).
[0214]
If the determination result is NO, the processes after step ST217 are repeated. If the determination result is YES, assuming that the shift size for the specific character = the default shift size (step ST220), the processing after step ST217 is repeated.
[0215]
When the shift amount set in the binary tree of FIG. 51 is shaped according to the processing of FIG. 50, the following is obtained. First, by the processing of steps ST201 to ST203, a keyword set for verification {smart, east, state} is loaded on the queue Q, and the keyword popped from the queue is set to j.
[0216]
Assuming that this is “state”, jst = node “5”, bst = node “7”, and jlen = 5 by the processing of steps ST204 to ST206. At this time, since bst is not a root node, the process proceeds to step ST208. In step ST208, slen = jlen-bst depth = 5-2 = 3. In step ST209, the node “7” is placed in the queue N, and in step ST212, it is popped from the queue N and set to r.
[0217]
In the condition determination in step ST213, the default shift of r (default shift of node “7”) is 6, which is larger than slen. Therefore, the process moves to step ST214, where r default shift = slen + r depth = 3 + 2 = 5.
[0218]
Thereafter, the node “8” is loaded in the queue N by the processing in step ST215, and the same processing is performed. As a result, the branches below the node “8” are processed in order, and the size of the default shift of each node is changed.
[0219]
As described above, in the node that is the fail destination of the terminal node of the keyword and the deeper node connected to the node, the size of the default shift is (slen + r depth) by the processing in steps ST213 and ST214. It is suppressed to the following.
[0220]
When all the nodes are finally processed, a binary tree as shown in FIG. 52 is obtained. In FIG. 52, it can be seen that the default shifts of the nodes “7” and “1” which are the fail destinations of the keyword end nodes “5” and “9” are smaller by 1 than in FIG. Further, the default shift of each node connected below the nodes “7” and “1” is also decreased by one.
[0221]
FIGS. 53 and 54 are flowcharts of processing for converting the binary tree output from the shift amount calculation unit 143 into a compressed state transition machine in an array format. When the processing is started in FIG. 53, the conversion unit 144 first initializes the arrays GOTO, CHECK, and OUTPUT to 0 (step ST221), and initializes the member index of the binary tree node to 0 (step ST222). .
[0222]
This index is provided independently of the index of the state transition array in order to store the correspondence between each node of the binary tree and the index of the state transition array as shown in FIG.
[0223]
Next, the queue Q is initialized (step ST223), and Pn = root node, Cn = next node of the root node, and Pp = 1 (step ST224).
Here, the next node of the root node means a node having a minimum character label among a plurality of nodes (node string) that can be transitioned from the root node. In the case of a binary tree as shown in FIG. 52, the node placed in Cn matches the node pointed by the pointer down from the root node.
[0224]
Next, the triplet [Pn, Cn, Pp] is added to the queue Q (step ST225), and it is determined whether or not the queue Q is empty (step ST226). If the queue Q is empty, the process ends.
[0225]
If the queue Q is not empty, then a triple is popped from the head of the queue Q and set to s (step ST227). An array GOTO, CHECK, in which a node connected to the specific list of the node Pn in s, a node connected to the node Cn in s and having the same depth as Cn, and the shift amount of the default shift with respect to s can be inserted. The position on OUTPUT is obtained and set to point (step ST228).
[0226]
At this time, the position of the point of the already inserted node is not overlapped with the position of the point of the newly inserted node. If a new insertable position is already used as a point, for example, it is shifted by one and set to a point.
[0227]
Next, GOTO [s Pp] = point (step ST229), and CHECK [point + 1] = 1 (step ST230). Also, the magnitude of the default shift of the node Pn in s is multiplied by −1 to make a negative value, and this is put into GOTO [point + 1] (step ST231). As a result, a default shift value is set at a position of index = point + 1 in the array.
[0228]
Next, a node connected to the Pn specific list of s is put in the queue tmp (step ST232), and it is determined whether or not the queue tmp is empty (step ST233). If the queue tmp is not empty, the node popped from the queue tmp is set to j (step ST234).
[0229]
Then, as a shift amount for the character label of node j, a value obtained by multiplying the corresponding shift magnitude by -1 is set in GOTO [point + 'j label] (step ST235). Here, “label” is an internal code on the array corresponding to the character label, and “label ≧ 2” when the conversion table of FIG. 41 is used.
[0230]
Next, the character label of j is set as a confirmation label in CHECK [point + 'j label] (step ST236), and the processing after step ST233 is repeated.
[0231]
When the queue tmp becomes empty in step ST233, next, Cn in s and a node having the same depth are placed in the queue tmp (FIG. 54, step ST237), and it is determined whether or not the queue tmp is empty (step ST238).
[0232]
If the queue tmp is not empty, the node popped from the queue tmp is set to i (step ST239), and the value of (point + 'i label) is set to the index of the node i (step ST240). It is checked whether or not output is defined for the node i (step ST241).
[0233]
If node i has an output, the output of node i is copied to array OUTPUT (step ST242). Here, OUTPUT [label + 'i label] = output of i. Next, as a label of CHECK [point + 'i label] = i (step ST243), the processes after step ST238 are repeated. If there is no output in the node i, the process after step ST243 is performed.
[0234]
If tmp becomes empty in step ST238, next, a node having the same depth as Cn is placed in the queue tmp (step ST244), and it is determined whether tmp is empty (step ST245).
[0235]
If the queue tmp is not empty, the node popped from the queue tmp is set to i (step ST246), and it is determined whether or not it is possible to make a transition to the next state with any symbol from the node i (step ST247). If the transition is impossible, the processes after step ST245 are repeated.
[0236]
If transition is possible, the triplet is added to the queue Q as the leading node of the transition destination of Pn = i and Cn = i, and the index of Pp = i (step ST248), and the processing after step ST245 is repeated. Here, the leading node at the transition destination of i means a node having the smallest character label in a node string that can transition from the node i.
[0237]
If tmp becomes empty in step ST245, the process returns to step ST226 in FIG. 53, and the subsequent processing is repeated.
Next, the procedure for converting the binary tree of FIG. 52 into the array format as shown in FIG. 40 will be described with reference to the flowcharts of FIGS. In this conversion process, the relationship between the preceding node and the node that can be shifted from it or the node that can be shifted is mapped from the binary tree to the array format.
[0238]
FIG. 55 shows a summary of transitions by input symbols, specific shifts by input symbols, and default shifts for each node in FIG. In FIG. 55, for example, “e: 1” in the first row and second column indicates that the transition destination by the input symbol “e” for the node “0” is the node “1”, and in the first row and fourth column. “A: 2” indicates that the magnitude of the shift by the input symbol “a” for the node “0” is 2.
[0239]
In the process from step ST227 of FIG. 53 to step ST248 of FIG. 54, the conversion unit 144 searches for a vacant place on the array where the symbol of the next transition / shift can be defined for the preceding state, and the data is obtained. Insert and map transition relationships. This data insertion method is basically the same as the method described in the first embodiment.
[0240]
Here, the mapping operation will be described using the root node “0” as an example. The pattern of data to be inserted with respect to the root node is as shown in FIG. FIG. 56 shows that there are six input symbols {a, e, m, r, s, t} that can define transitions other than the default shift for the root node.
[0241]
Initially, all the elements of the state transition array are empty. Therefore, when the place where the pattern of FIG. 56 can be inserted is searched on the array, the index of the smallest possible array is “1”. Therefore, when the point is set to “1”, the character label is written, and the shift amount of the default shift or specific shift is set by the processing from step ST229 to ST236 in FIG. 53, the pattern shown in the line s0 in FIG. Is mapped. At this stage, the portions corresponding to the labels “e” and “t” remain undefined.
[0242]
Since the portions of the labels “e” and “t” correspond to the next node of the root node “0”, they are set when these nodes are inserted into the array by the processing from step ST237 to ST248 in FIG. Is done. Here, the next node to be processed next to the root node is the node “1”.
[0243]
Since the only character that can transition from the node “1” is “t”, the insertion location of the node “1” is a location corresponding to the pattern as shown in FIG. In FIG. 57, the upper row represents a pattern on the current array CHECK, and the lower row represents a pattern to be inserted.
[0244]
When a place where such a pattern can be inserted is searched on the array, the index of the minimum array that can be set to “point” is “3”. Therefore, in order to define that the node “1” is the transition destination from the node “0”, the value 3 of this point is set to GOTO [7], and the transition from the node “0” to the symbol “e” is performed. It is defined that the destination data starts from the position of index = 3.
[0245]
Then, this node “1” is inserted at the position of index = 3, and character labels, default shift amounts, etc. are set by the processing from step ST229 to ST236. As a result, the pattern shown in the row of s1 in FIG. 40 is mapped.
[0246]
Such processing is performed in the order in which nodes are stacked in the queue, that is, nodes “0”, “1”, “6”, “2”, “7”, “10”, “3”, “8”, “11”. By performing in the order of “4”, “9”, “12”, “5”, “13”, a state transition array as shown in FIG. 40 is finally obtained.
[0247]
The nodes “14”, “15”, and “16” in FIG. 52 are also loaded in the queue and subjected to the same processing, and strictly speaking, transitions by the terminal symbol “#” are defined. However, since the terminal symbols are not included in the output character string, the processing of these nodes is omitted here, and the transition to these nodes is also omitted in FIG. It is also possible not to define a transition for the terminal node by setting the value of the label of the terminal symbol in the array to 0.
[0248]
Next, an embodiment of a character string replacement device in which a character string replacement function is added to the collation device of the second embodiment will be described. Here, an example in which the input keyword {smart, east, state} is replaced with {SMART, EAST, STATE}, respectively.
[0249]
FIG. 58 shows a state transition array for character string replacement for an input key. In FIG. 58, array GOTO stores a superimposed state transition table, array CHECK stores a confirmation label, array SUBST stores a pointer to a replacement character string, and array LENGTH stores characters before replacement. Stores the length of the column. The initial state of the text offset storage array used for the replacement process is the same as that shown in FIG.
[0250]
The text offset for each symbol in the input text 'aceast' is as shown in FIG. Also, after pattern matching is performed on this input text, the text offset storage array is as shown in FIG. The collation process at this time is the same as in FIG. As a result of the collation, the position of the keyword “east” in the text is stored as the text offset “3”.
[0251]
The replacement process is performed in the same manner as in FIG. As a result of this replacement processing, the text “aaseast” is converted to “aasEAST”.
In the first and second embodiments described above, since a state transition array in which a DFA having a table structure is compressed is basically used, collation can be completed with the same number of transition operations as the number of input symbols. The high speed of DFA is maintained. However, the storage capacity required to store the state transition array is much smaller than that of the conventional state transition table.
[0252]
By the way, in the above-mentioned example, the text of the English character is input, but the present invention can be applied to the text written in the 2-byte character code such as Japanese by the following two methods.
[0253]
In the first method, first, a state transition array is created for each 1-byte keyword group. When one of the keyword patterns is detected in the text, the process returns to the line feed symbol preceding the pattern. Next, it is confirmed whether or not the detected pattern is shifted by 1 byte in the sentence starting from the position next to the line feed symbol.
[0254]
In the second method, a state transition array is created using the input 2-byte character code as one unit, and collation processing is performed.
For text in which 1-byte characters such as ASCII (American Standard Code for Information Interchange) and 2-byte characters such as Japanese EUC (Extended UNIX Code) are mixed, the 2-byte characters are 1 byte when shifted in the second embodiment. There is a possibility that collation is performed with a deviation.
[0255]
Therefore, for such text, it is necessary to go back to the beginning of the sentence containing the pattern at the stage of accepting the pattern, and check whether the detected pattern is misaligned with reference to the beginning position. is there.
[0256]
【Example】
Hereinafter, a comparison result of the storage capacity and search speed at the time of character string search between the conventional collation apparatus and the collation apparatus of the present invention will be described.
[0257]
First, the case of keyword search based on the AC method will be compared. FIG. 61 shows a change in memory usage in the conventional method and the method of the present invention. In FIG. 61, the vertical axis of the graph represents the memory usage, and the horizontal axis represents the number of keywords. “AC + ours” corresponds to the collation apparatus according to the first embodiment of the present invention based on the DFA AC method, and “DFA of AC” corresponds to the collation apparatus using the conventional state transition table. .
[0258]
As can be seen from FIG. 61, as the number of keywords increases, the memory usage greatly increases in the conventional method, but does not increase so much in the method of the first embodiment of the present invention.
[0259]
FIG. 62 shows a change in search speed of each collation device when a keyword search is performed on a 75 Mbyte text. In FIG. 62, the vertical axis represents the time (seconds) required for the search, and the horizontal axis represents the number of keywords. Here, for comparison, the search speed of the conventional AC method is added as “AC”.
[0260]
As shown in FIG. 62, as the number of keywords increases, the search speed decreases in the conventional AC method. However, in the conventional AC method converted to DFA and the method of the first embodiment, the search speed is kept high. You can see that Furthermore, the difference in search time between “AC + ours” and “DFA of AC” is slight.
[0261]
Next, a comparison will be made regarding the case of keyword search based on the FAST method. The storage areas required in the collation apparatus according to the second embodiment of the present invention are a binary tree area, a shift quantity list area for a specific character, and a compressed state transition array area. In addition, the necessary storage area in the conventional FAST method collation apparatus is the number of pointer areas determined based on the number of possible input symbols and the number of states.
[0262]
FIG. 63 shows changes in memory usage in the conventional method and the method of the present invention. In FIG. 63, the vertical axis of the graph represents the memory usage, and the horizontal axis represents the number of keywords. “NORMAL” corresponds to a collation device using the conventional FAST method, and “COMPRESS” corresponds to the collation device of the second embodiment of the present invention.
[0263]
As can be seen from FIG. 63, as the number of keywords increases, the memory usage greatly increases in the conventional method, but hardly increases in the method of the second embodiment of the present invention.
[0264]
【The invention's effect】
According to the present invention, it is possible to create an efficient collation table on both sides of speed and storage capacity for a given keyword group. Further, by using the table, it is possible to efficiently collate / search a plurality of keywords specified for the digitized document.
[0265]
In addition, with such a character string collation process, the functions of the word processor such as character string search and character string replacement can be made more efficient. In addition, functions such as character string search of full-text search index in full-text search device and character string replacement such as parenthesis processing at the display stage, and functions such as character string search in database and character string replacement at the display stage are also improved. be able to.
[Brief description of the drawings]
FIG. 1 is a principle diagram of a collation device of the present invention.
FIG. 2 is a configuration diagram of a collation system.
FIG. 3 is a flowchart of the operation of the collation system.
FIG. 4 is a configuration diagram of an information processing apparatus.
FIG. 5 is a configuration diagram of a collation device.
FIG. 6 is a diagram showing a first state transition table before compression;
FIG. 7 is a diagram illustrating a first state transition.
FIG. 8 is a diagram illustrating a compressed first state transition table.
FIG. 9 is a diagram showing a first character code conversion table;
FIG. 10 is a flowchart of first collation processing.
FIG. 11 is a diagram illustrating a first collation operation.
FIG. 12 is a configuration diagram of a first compression device.
FIG. 13 is a flowchart of first binary tree creation processing;
FIG. 14 is a diagram illustrating a first binary tree.
FIG. 15 is a flowchart of processing for adding failure to a binary tree;
FIG. 16 is a diagram illustrating a first binary tree with a failure.
FIG. 17 is a diagram illustrating an example of failure calculation;
FIG. 18 is a flowchart of processing for adding “gotoin” and “gooutout” to a binary tree;
FIG. 19 is a diagram illustrating a first binary tree with “gotoin” and “goout”.
FIG. 20 is a first flowchart of first conversion processing;
FIG. 21 is a flowchart (part 2) of the first conversion process;
FIG. 22 is a diagram showing a first arrangement.
FIG. 23 is a diagram showing a first pattern.
FIG. 24 is a diagram showing a second arrangement.
FIG. 25 is a diagram showing a third arrangement.
FIG. 26 is a diagram showing a fourth arrangement;
FIG. 27 is a diagram showing a second pattern.
FIG. 28 is a diagram showing a fifth arrangement.
FIG. 29 is a diagram showing a third pattern.
FIG. 30 is a diagram showing a sixth arrangement.
FIG. 31 is a diagram showing a fourth pattern.
FIG. 32 is a diagram showing a seventh arrangement.
FIG. 33 is a diagram showing a first state transition table for character string replacement.
FIG. 34 is a diagram showing a text offset storage array.
FIG. 35 is a diagram showing a text offset of the first text.
FIG. 36 is a diagram showing a first text offset storage array after pattern matching.
FIG. 37 is a flowchart of a replacement process.
FIG. 38 is a diagram showing a second state transition table before compression;
FIG. 39 is a diagram illustrating a second state transition.
FIG. 40 is a diagram illustrating a compressed second state transition table.
FIG. 41 is a diagram illustrating a second character code conversion table.
FIG. 42 is a flowchart of second collation processing.
FIG. 43 is a diagram illustrating a second collation operation.
FIG. 44 is a configuration diagram of a second compression device.
FIG. 45 is a flowchart of second binary tree creation processing;
FIG. 46 is a flowchart of pre-processing for shift calculation.
FIG. 47 is a diagram illustrating a second binary tree.
FIG. 48 is a flowchart of first shift amount calculation processing;
FIG. 49 is a flowchart of second shift amount calculation processing;
FIG. 50 is a flowchart of third shift amount calculation processing;
FIG. 51 is a diagram illustrating a second binary tree that is shifted with failure.
FIG. 52 is a diagram illustrating a final second binary tree.
FIG. 53 is a flowchart (part 1) of a second conversion process;
FIG. 54 is a flowchart (part 2) of the second conversion process;
FIG. 55 is a diagram showing information on a binary tree.
FIG. 56 is a diagram showing a fifth pattern.
FIG. 57 is a diagram showing a sixth pattern.
FIG. 58 is a diagram showing a second state transition table for character string replacement.
FIG. 59 is a diagram showing a text offset of the second text.
FIG. 60 is a diagram showing a second text offset storage array after pattern matching.
FIG. 61 is a diagram showing changes in memory usage in the AC method;
FIG. 62 is a diagram showing a change in speed in the AC method.
FIG. 63 is a diagram showing changes in memory usage in the FAST method.
FIG. 64 is a diagram illustrating an example of an AC method pattern matching machine;
FIG. 65 is a diagram illustrating an operation example of an AC method.
FIG. 66 is a diagram showing an AC method DFA.
FIG. 67 is a diagram illustrating an operation example of AC method DFA;
FIG. 68 is a diagram illustrating an example of a FAST method pattern matching machine.
FIG. 69 is a diagram showing transitions and shifts in the FAST method.
FIG. 70 is a diagram illustrating an example of collation by the FAST method.
[Explanation of symbols]
1 State transition storage means
2 verification means
101 Compressor
102 Verification device
103 Keyword input section
104 Sparse array finite state machine creation part
105 State transition machine compression unit
106 State transition machine for verification
107 Text input part
111 CPU
112 memory
113 Input device
114 output device
115 External storage device
116 Medium drive device
117 Network connection device
118 bus
119 Portable recording media
120 Information provider devices
121 State transition determination unit
122 State transition part
123 Compression state transition part
124 Confirmation label
125 Output symbol part
131, 141 Binary tree conversion unit
132 Transition addition part
133, 144 conversion unit
142 Pre-processing section
143 Shift amount calculation part

Claims (12)

A verification device that uses a given symbol string as a key and determines whether or not the key exists in a file by using a finite state machine,
A state transition table of the finite state machine in which a collation operation relating to at least one key is defined, wherein the sparse state transition table excluding data representing a transition operation from the current state to the initial state is a compressed array State transition storage means for storing in a form;
With reference to the sparse state transition table, an operation corresponding to each symbol included in the file is performed, and collation means for collating a symbol string in the file with the one or more keys,
The collation means checks whether or not an operation for an input symbol input from the file is defined in the sparse state transition table, and when an operation for the input symbol is not defined, the initial state is entered . A collation device characterized by performing a transition operation.   The state transition storage unit stores the sparse state transition table excluding data representing a transition operation from the current state to the next state of the initial state, and the collating unit stores the sparse state transition table in the sparse state transition table. The collation apparatus according to claim 1, wherein when a transition for an input symbol is not defined, a transition operation to the next state of the initial state is performed via the initial state. The collation apparatus according to claim 1, storage means for storing an output symbol array for storing data for specifying a key corresponding to an input symbol included in the one or more keys, and data in the output symbol array And an output means for outputting a key corresponding to the input symbol. Collating apparatus according to claim 1, wherein the one or more in response to the input symbols in the key storage means for storing replacement symbol sequence for storing data identifying the replacement symbol string output in place of the key And an output means for outputting a replacement symbol string corresponding to the input symbol based on data in the replacement symbol array.   A word processor device comprising the collation device according to claim 1 and performing at least one of a symbol string search function and a symbol string replacement function using the collation device.   A database system comprising the collation device according to claim 1 and performing at least one of a symbol string search function and a symbol string replacement function using the collation device.   A full-text search device comprising the collation device according to claim 1, wherein the full-text search device performs at least one of a symbol string search function and a symbol string replacement function using the collation device. A collation device that uses a given symbol string as a key and creates a finite state machine for determining whether or not the key exists in a file,
A binary tree data representing at least one or more keys is created, and an intermediate structure finite state machine corresponding to a sparse state transition table excluding data representing a transition operation from the current state to the initial state is represented by the binary tree. Sparse finite state machine creation means for creating based on tree data;
State transition machine compression means for converting the intermediate structure finite state machine into a compressed array format,
With reference to the sparse state transition table converted into the array format, it is checked whether or not an operation for an input symbol input from the file is defined, and an operation for the input symbol is not defined A collation apparatus in which a transition operation to the initial state is performed. The sparse finite state machine forming means, said one or more and the binary tree data generation means for expressing the key, the initial state and initial state the next state and the two information failure transition destinations except quadtree A transition adding means for adding data, and the state transition machine compressing means creates a state transition array compressed so that a plurality of elements do not overlap each other based on the binary tree data received from the transition adding means The collation apparatus according to claim 8, wherein: A verification device that uses a given symbol string as a key and determines whether or not the key exists in a file by using a finite state machine,
A binary tree data representing at least one or more keys is created, and an intermediate structure finite state machine corresponding to a sparse state transition table excluding data representing a transition operation from the current state to the initial state is represented by the binary tree. Sparse finite state machine creation means for creating based on tree data;
State transition machine compression means for converting the intermediate structure finite state machine into a compressed array format;
State transition storage means for storing the sparse state transition table converted into the array format;
With reference to the sparse state transition table, an operation corresponding to each symbol included in the file is performed, and collation means for collating a symbol string in the file with the one or more keys,
The collation means checks whether or not an operation for an input symbol input from the file is defined in the sparse state transition table, and when an operation for the input symbol is not defined, the initial state is entered . A collation device characterized by performing a transition operation. A recording medium recording a program for a computer that uses a given symbol string as a key and determines whether or not the key exists in a file using a finite state machine,
A state transition table of the finite state machine that defines a collation operation relating to at least one or more keys, with reference to the sparse state transition table excluding data representing a transition operation from the current state to the initial state, When an operation corresponding to each symbol included in the file is performed and a symbol string in the file is collated with the one or more keys, an operation on an input symbol input from the file is added to the sparse state transition table. Check whether it is defined, when the operation for the input symbol is not defined, computer-readable recording medium a program for realizing a function of performing a transition operation to the initial state to the computer . A recording medium recording a program for a computer that creates a finite state machine for determining whether or not the key exists in a file using a given symbol string as a key,
A function of creating binary tree data representing at least one or more keys;
A function for creating a finite state machine having an intermediate structure corresponding to a sparse state transition table excluding data representing a transition operation from a current state to an initial state based on the binary tree data;
With reference to the sparse state transition table, it is checked whether or not an operation for an input symbol input from the file is defined. When an operation for the input symbol is not defined , the transition to the initial state is performed. A computer-readable recording medium storing a program for causing the computer to realize a function of converting the finite state machine having the intermediate structure into a compressed array format so that an operation is performed .

Images