JPH11161677A - Dictionary collating device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily discriminate between a symbol and a symbol array so as to apply a simple arranging method to state transition wherein not only the symbol, but also the symbol array is allowed for a transition label. SOLUTION: The device consists of an input part 1 for a symbol array, a dictionary 2 comprising a set of symbol arrays represented in a state transition table, and a dictionary collating part 3 which detects all dictionary elements appearing in the input symbol array. When the dictionary collating part 3 makes a state transition on the basis of symbols in the input symbol array or the symbol array and state transition table, codes derived from the code of the head symbol through mathematical operation is made to correspond to the symbol array. The symbols and symbol array are discriminated through the mathematical operation with small operation quantity, so that high-speed dictionary collating is realized, while the state transition number is reduced with the symbol array label to improve storage efficiency.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a dictionary matching device, and more specifically, a set of symbol strings (dictionary) is represented by a state transition table, collated with an input symbol string, and appears in an input symbol string. A device for detecting all dictionary elements is applicable to a translation device, a text search device, a spell checker, a kana-kanji conversion device, a speech recognition device, a character recognition device, and the like.

[0002]

2. Description of the Related Art The task of matching a symbol string to be input with a set (dictionary) of symbol strings and detecting all dictionary elements appearing in the input symbol string (dictionary matching) is not limited to language processing, but is a wide field. It is done in. In general, high-speed dictionary matching that is proportional to the input length is realized by effectively expressing a dictionary as a graph, reading input symbol strings one by one, and deterministically performing state transitions based on the symbol type (code). it can. FIG. 10 shows an example of an effective graph called TRIE, and FIG. 11 shows an example of an effective graph called DAWG.
A set of symbol strings ca｝ is expressed.

[0003] The above-described graph shows a state si- (x)-.
集合 sj represents a set of state transitions. As a technique for compactly expressing a set of state transitions, there is a single arrangement method. FIG. 12 shows the state transition by the single arrangement method. In FIG. 12, each state s has its own position base (s) in the array, and state transition si- (a)-
† sj is the base (si) and the internal code a ′ of the symbol a.
From the base value base (sj) of the state s2
(Si) Obtained by the value of the base array at the position of + a '. The reason that a 'is stored in the label array at the position of base (si) + a' is to uniquely fix the original state transitioned by a 'at that position.

[0004]

FIG. 13 is a diagram showing an example of a DAWG. The graph shown in FIG.
A set of symbol strings of aa, aba, adcaｃ is represented. The difference from the example shown in FIG. 11 is that there is no state s4, and instead a symbol string dc is used as a label. In order to apply the single constellation method to a state transition that allows not only a symbol but also a symbol string to a transition label, the symbol and the symbol string must be easily distinguishable. The invention of claim 1 has been made for the purpose of solving this problem.

[0005] The base array of the single arrangement method needs to be able to store an arbitrary position in the array. Therefore, when the array length is long, the storage amount of each cell in the base array increases. For example, 1 byte can represent positions from 0 to 255, but if the array length exceeds 255, 2 bytes are required,
The amount of storage required for the base array is doubled at a stroke. The inventions of claims 2 and 3 have been made to solve this problem.

DAWG has a smaller number of states and state transitions than TRIE. In particular, in the set of symbol strings, the end of each symbol fits into one state (called the end state). Therefore, when the DAWG is expressed by the single arrangement method, a transition from a distant position in the array to this unique terminal state occurs,
The required storage amount of each cell in the base array is large. The invention of claim 4 has been made for the purpose of solving this problem.

In TRIE and DAWG, if not only symbols but also symbol strings are allowed for transition labels, subsequent symbol strings can be predicted if the beginning of the symbol string matches the input. The invention of claim 5 has been made to provide such a prediction function to dictionary matching.

[0008]

According to the first aspect of the present invention, there is provided a symbol string input section, a dictionary comprising a set of symbol strings represented by a state transition table, and all dictionary elements appearing in the input symbol string. When performing state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary collation section, an arithmetic operation is performed on the symbol string from the code of the first symbol of the symbol string. The feature is to correspond to the code derived by, and by distinguishing symbols and symbol strings by an arithmetic operation with a small amount of computation, the number of state transitions is reduced by symbol string labels while enabling high-speed dictionary matching This improves the storage efficiency.

According to a second aspect of the present invention, there is provided a symbol string input unit, a dictionary including a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string. When performing a state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, the destination number of the transition destination is calculated by an arithmetic operation from the identification number (state number) of the transition source state. It is characterized by assigning a state number, whereby the state number of the transition destination can be assigned by an arithmetic operation from the identification number (state number) of the state of the transition source, so that the array capacity in the single arrangement method is reduced. It is.

A third aspect of the present invention provides an input unit for a symbol string, a dictionary composed of a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string. When performing a state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, the state transition is divided into a plurality of parts, and the state number of the transition source state and the transition destination state are divided. Is characterized in that the difference between the state numbers is always kept within a fixed value, thereby reducing the array capacity in the single arrangement method by dividing the state transition into a plurality.

According to a fourth aspect of the present invention, there is provided a symbol string input section, a dictionary comprising a set of symbol strings represented by a state transition table, and a dictionary matching section for detecting all dictionary elements appearing in the input symbol string. When the dictionary matching unit performs a state transition based on a symbol or a symbol string in an input symbol string and a state transition table, a plurality of identification numbers (terminal states) are assigned to a state (terminal state) having no transition from the state transition table. State numbers are assigned, and by providing a plurality of terminal states, a state transition at a long distance in the single arrangement method is suppressed, and the array capacity is reduced.

According to a fifth aspect of the present invention, there is provided a symbol string input unit, a dictionary comprising a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string. When a state transition is performed based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, the transition label of the state transition table is the longest common of the partial symbol strings expressed in each state. If the longest common prefix of the partial symbol string set corresponding to the state is used as the transition label, the longest character string can be predicted when performing dictionary matching from left to right.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a schematic diagram of a main part for explaining a basic configuration of the present invention. In the drawing, reference numeral 1 denotes a symbol string input unit, and 2 denotes a symbol string represented by a state transition table. (Dictionary) 3 is a dictionary matching unit that detects all dictionary elements appearing in the input device sequence, and converts the symbol sequence input to the input unit 1 into
The dictionary collating unit 3 collates with the dictionary (set of symbol strings) 2 as described in claims 1 to 5 below.

(Invention of Claim 1) FIG. 2 is a diagram for explaining the operation of the invention of Claim 1, and the invention of Claim 1 is represented by a symbol string input unit 1 and a state transition table. And a dictionary matching unit 3 for detecting all dictionary elements appearing in the input symbol string. The dictionary matching unit 3 determines the state of the symbol or symbol string in the input symbol string. When performing a state transition based on a transition table, a code derived from an arithmetic operation from a code of the first symbol is made to correspond to a symbol string, as shown in FIG.
In DAWG and TRIE, there is at most one transition starting from the symbol a from a certain state. Therefore, as shown in FIG.
Focusing on the fact that a symbol having the same leading symbol and a symbol string do not co-exist, a symbol string starting with a has a ′ + sig
The code ma is made to correspond. Here, sigma is the maximum code value in the dictionary. This ensures that the code of the symbol string does not overlap with other symbols. Note that the transition source cannot be uniquely determined only from the values of the label array where the codes overlap. The algorithm of dictionary matching is as shown in FIG.

FIG. 5 is a diagram showing an input part of a symbol string, a dictionary composed of a set of symbol strings represented by a state transition table, and a dictionary appearing in the input symbol string. And a dictionary matching unit for detecting all dictionary elements to be changed. When the dictionary matching unit performs a state transition based on a symbol or a symbol string in an input symbol string and a state transition table, an identification number of a transition source state It is characterized by assigning the state number of the transition destination by (arithmetic operation) from (state number). As shown in FIG.
The required size of the base array element is suppressed by inserting a relative distance from the state number of the transition source without directly storing the value. In the case of FIG. 5, offset + base (s
The next base value is determined according to i). FIG. 6 is a diagram showing an algorithm of dictionary matching according to the second aspect of the present invention.

According to a third aspect of the present invention, there is provided an input unit for a symbol string, a dictionary composed of a set of symbol strings represented by a state transition table, and all dictionaries appearing in the input symbol string. When the state transition is performed based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary collation section, the state transition is divided into a plurality of parts to determine the transition source. The feature is that the difference between the state number of the state and the state number of the transition destination is always kept within a certain value. If there is no transition destination within a certain distance from the current base value, the difference is within the certain range. Search for an unused array position (a position that is not a base in any state) at
A new state having “se” is provided, and ε transition from which no input symbol is consumed is continued until the state reaches the initial transition destination state.

In the state transition of s1- (a)-> s2, if the distance between base (s1) and base (s2) exceeds a predetermined value max, the state transition is chained as shown in FIG. However, in each transition, the distance between the transition source and the transition destination does not exceed the fixed value max, and the input symbol string is not consumed in the state transition by the symbol e.

According to a fourth aspect of the present invention, there is provided an input unit for a symbol string, a dictionary comprising a set of symbol strings represented by a state transition table, and all dictionaries appearing in the input symbol string. When a state transition is performed based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary comparison section, a state where there is no transition from the dictionary matching section (terminal state) Means multiple identification numbers (state numbers)
S1- (a) F
In the state transition, F is the terminal state.
If the distance between (s1) and base (F) exceeds a predetermined value max, a new terminal state F2 is provided near base (s1) where the distance does not exceed max, and s1- (a) is used instead.
The state transition becomes F2.

According to a fifth aspect of the present invention, there is provided an input unit for a symbol string, a dictionary comprising a set of symbol strings represented by a state transition table, and all dictionaries appearing in the input symbol string. When performing state transition based on a symbol or symbol string in an input symbol string and a state transition table in the dictionary collation section, a transition label of the state transition table is expressed in each state. This is characterized by using the longest common prefix of the partial symbol string. For example, consider a state transition by a symbol as shown in FIG.

The graph of FIG. 8 is represented by {abcde, abcd
f, efcde, efcdf represents a set of symbol strings. As shown in FIG. 9, the structure is not shared with respect to the longest common prefix from left to right. Here, the longest common prefix is abcd in terms of {adcde, abcdf}. Thereby, when the input symbol a comes in the state s1, it is predictable (uniquely determinable) that the input symbol a will follow in the dictionary, and the maximum length symbol string abc
d can be obtained.

[0021]

According to the first aspect of the present invention, there is provided a symbol string input unit,
A dictionary composed of a set of symbol strings represented by a state transition table,
A dictionary matching unit for detecting all dictionary elements appearing in the input symbol string, and performing a state transition based on a symbol or symbol string in the input symbol string and the state transition table in the dictionary matching unit. , The code derived from the code of the first symbol by arithmetic operation is made to correspond,
By performing the distinction between the symbol and the symbol string by an arithmetic operation with a small amount of calculation, it is possible to reduce the number of state transitions by the symbol string label and improve the storage efficiency while enabling high-speed dictionary matching.

According to a second aspect of the present invention, there is provided a symbol string input section, a dictionary composed of a set of symbol strings represented by a state transition table, and a dictionary matching section for detecting all dictionary elements appearing in the input symbol string. When performing a state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, the destination number of the transition destination is calculated by an arithmetic operation from the identification number (state number) of the transition source state. Since the state number is assigned, the state number of the transition destination can be assigned by an arithmetic operation from the identification number (state number) of the state of the transition source, and the array capacity in the single arrangement method can be reduced.

According to a third aspect of the present invention, there is provided a symbol string input section, a dictionary composed of a set of symbol strings represented by a state transition table, and a dictionary matching section for detecting all dictionary elements appearing in the input symbol string. When performing a state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, the state transition is divided into a plurality of parts, and the state number of the transition source state and the transition destination state are divided. Since the difference between the state numbers is always kept within a fixed value, the array capacity in the single arrangement method can be reduced by dividing the state transition into a plurality.

According to a fourth aspect of the present invention, there is provided a symbol string input unit, a dictionary comprising a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string. When the dictionary matching unit performs a state transition based on a symbol or a symbol string in an input symbol string and a state transition table, a plurality of identification numbers (terminal states) are assigned to a state (terminal state) having no transition from the state transition table. Since state numbers are assigned, a plurality of terminal states are provided, so that a long-distance state transition can be suppressed and the array capacity can be reduced in the single arrangement method.

According to a fifth aspect of the present invention, there is provided a symbol string input section, a dictionary composed of a set of symbol strings represented by a state transition table, and a dictionary matching section for detecting all dictionary elements appearing in the input symbol string. When a state transition is performed based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, the transition label of the state transition table is the longest common of the partial symbol strings expressed in each state. Since the prefix is used, if the longest common prefix of the partial symbol string set corresponding to the state is used as the transition label, the longest character string can be predicted when performing dictionary matching from left to right.

[Brief description of the drawings]

FIG. 1 is a schematic configuration diagram of a collation device to which the present invention is applied.

FIG. 2 is a diagram for explaining the operation of the invention of claim 1;

FIG. 3 is a diagram for explaining the operation of the invention of claim 1;

FIG. 4 is a diagram showing an algorithm of dictionary matching according to the first embodiment of the present invention.

FIG. 5 is a diagram for explaining the operation of the invention of claim 2;

FIG. 6 is a diagram showing an algorithm of dictionary matching according to the second aspect of the present invention.

FIG. 7 is a diagram illustrating a chain of state transitions according to the invention of claim 3;

FIG. 8 is a diagram for explaining a state transition in the invention of claim 5;

FIG. 9 is a diagram showing a graph with the minimum number of states according to the invention of claim 5;

FIG. 10 is a diagram showing an example of an effective graph of TRIE.

FIG. 11 is a diagram showing an example of an effective graph of DAWG.

FIG. 12 is a diagram illustrating an example of a single arrangement method.

FIG. 13 is a graph showing another example of DAWG.

[Explanation of symbols]

 1. Input unit, 2. Dictionary, 3. Matching unit.

Claims (5)

[Claims] An input unit for a symbol string, a dictionary composed of a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string, When performing a state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, it is necessary to associate a symbol string with a code derived by an arithmetic operation from the code of the first symbol. A dictionary matching device that features. 2. An input unit for a symbol string, a dictionary including a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string, When performing a state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, a state number of a transition destination is calculated by an arithmetic operation from an identification number (state number) of a state of a transition source. A dictionary matching device, which is assigned. 3. An input unit for a symbol string, a dictionary including a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string, When performing a state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, the state transition is divided into a plurality of parts, thereby dividing the state number of the transition source state and the state number of the transition destination state. A dictionary matching apparatus characterized in that the difference is always kept within a certain value. 4. An input unit for a symbol string, a dictionary composed of a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string, When a state transition is performed by the dictionary matching unit based on a symbol or a symbol string in an input symbol string and a state transition table, a plurality of identification numbers (state numbers) are assigned to a state that has no transition from that state (terminal state). A dictionary matching device, which is assigned. 5. An input unit for a symbol string, a dictionary composed of a set of symbol strings represented by a state transition table, and a dictionary matching unit for detecting all dictionary elements appearing in the input symbol string, When performing a state transition based on a symbol or a symbol string in an input symbol string and a state transition table in the dictionary matching unit, a transition label of the state transition table should be a longest common prefix of a partial symbol string represented in each state. A dictionary matching device characterized by the following.