JPH0415503B2 - - Google Patents

Description

[Detailed Description of the Invention] [Technical Field of the Invention] The present invention is applicable to automatic translation, text-to-speech conversion, etc.
The present invention relates to a word identification device that separates individual words constituting a sentence from a sentence expressed in characters.

[Background of the invention]

In automatic translation or text-to-speech conversion,
Analysis of sentences is essential. Particularly in languages such as Japanese, where word boundaries are unclear and there are many homographs and allophones, determining word boundaries and identifying words is important and difficult. For example, in the word structure of ``Livestock Products Price Stabilization Act'', there are several possibilities such as ``Livestock Products, Prices and Stabilization Act'', ``Livestock Products, Prices, and Stabilization Act'', ``Livestock Products, Prices, Low Prices'', and ``Standard Law'', and ``Livestock Products, Prices, and Stabilization Act''. Must be able to judge.

[Prior art to the invention]

Conventionally, to identify words in a sentence, the method of sequential search judgment using longest match and legal connection relationships extracts all possible combinations of candidate string units and applies evaluation functions to each one. A method was considered in which the best combination was selected using the evaluation method. However, the optimal solution may not be obtained in some cases, and the processing is complicated (requires backtracking).

In addition, the number of combinations would be enormous, making it impossible to apply to long character strings.

[Purpose of the invention]

The present invention analyzes sentences such as Japanese with unclear word boundaries, determines the boundaries of character string units such as words that make up the sentence, and furthermore, performs accurate analysis for the task of identifying each character string unit. The purpose is to perform it easily and with less processing.

[Structure of the invention]

In order to achieve the above object, the present invention provides a text analysis device that analyzes a text consisting of a plurality of character string units and identifies the character string units that make up the text.
A character string unit dictionary 1 that has a character code assigned to each character string unit and used for matching each character string unit, and an input sentence is checked against the character code of the character string unit dictionary, and all possible constituent units of the input sentence are A dictionary collation unit 2 extracts candidate character string units, and for all the candidate characters extracted above, a first evaluation point that is an evaluation that does not depend on the surrounding situation, and a total evaluation point up to the candidate character string. A second evaluation score, which is obtained by adding an evaluation based on the grammatical connection relationship of the candidate character string with other candidate character strings, is obtained for each boundary unit each time, and the obtained first evaluation score and the second evaluation score are calculated for each boundary unit.
The identification position between a candidate character string and other candidate character strings is determined each time according to dynamic programming for each boundary unit using the evaluation points, and the DP identifies each character string unit.
It is characterized by comprising a part 3.

This will be explained in detail below with reference to the drawings.

Figure 1 shows the case using the conventional example described above and the dynamic programming method according to the present invention (hereinafter referred to as
This is a comparison with the case using the DP method. The text analysis device stores in advance all conceivable character example units (including conventional word strings and character strings in addition to so-called words). In the illustrated example, one character string is prepared for each of the readings of the character "shima" and "tou".

Furthermore, five types of character strings are prepared for the pronunciation of ``kara'', such as ``kiku'' as a noun, ``kara'' as a noun, and ``from'' as each particle.

Furthermore, there are 6 types of character strings for the single character "ka" (including single character strings, which are referred to as character strings),
For example, there are words such as ``--ka?'' to express a question, ``--ka'' to express a choice, and ``ka'' to express a counter-question. The same applies to other characters.

As described in the conventional example, the part A in the figure shows the number of combinations when all combinations are evaluated, and it can be seen that there are more than 100,000 combinations, which is not practical.

Part B in the diagram shows the case using the DP method of the present invention.
Even if you include the character strings that indicate the beginning and end of a sentence (comma), it only takes 288 processing times.

[Embodiments of the invention]

Figure 2 is a diagram explaining the concept of the DP method of the present invention. Focusing on a certain word boundary, the character strings that end at that boundary are of two types, A and B, and the character strings that start from that boundary are of α, β, The case where there are three types of γ is shown.

If you select a certain character example (let it be X), the total evaluation score up to X is g(X), the evaluation of X that does not depend on the surrounding situation is (X), and the evaluation based on the connection relationship with other character strings Y. Let be C(X,Y).

At this time, when the evaluation proceeds from the left side to the right side of the boundary shown in FIG. 2, the following processing is performed.

g(a)+C(a,α) g(α)=(α)+MAX g(b)+C(b,α) g(a)+C(a,β) g(β)=(α)+MAX g( B)+C(B,β) g(A)+C(B,γ) g(γ)=(α)+MAX g(B)+C(B,γ) Note that MAX { } takes the maximum value within the bracket. It means that.

In this way, for each character string from the left (from the beginning of the sentence), calculate your own evaluation at that location from your own evaluation and the evaluation based on the connection relationship with the previous string. Go to each boundary.

Incidentally, even at a position that is a boundary for only a part of a character string, as indicated by arrow C in FIG. 1, the same processing as described above may be performed for that character string.

Also, depending on how evaluation points are taken, MIN{ } may be used instead of MAX{ }.

In addition to the original character strings, consider character strings that indicate the beginning and end of a sentence (unnecessary if there are commas).

When evaluations are obtained one after another in this way, when obtaining an evaluation for the last character string (comma), several character string candidates immediately before it (in the example in Figure 1) are evaluated.
You can find out which of the 10 candidates) the connection will have the maximum value. Therefore, by sequentially tracing the character string candidates that give the maximum value, the optimal combination of character string units can be obtained.

Next, a specific embodiment for realizing the DP method of the present invention will be described using FIGS. 3 and 4.

FIG. 3 is a schematic block diagram of an embodiment of the present invention, in which 1 is a character string unit dictionary, 2 is a dictionary collation unit, and 3 is a block diagram of an embodiment of the present invention.
is the DP department.

The string unit dictionary 1 contains not only string unit notation for each string unit (character code used for matching), but also connection relationship information used by the DP unit 3 (identification of right and left connection relationships expressed by numbers). ), evaluation points that are determined regardless of surrounding character strings, character string unit numbers, etc. are set in advance.

The dictionary matching unit 2 extracts all candidate character units that can be constituent units of the input text by comparing the input text with the character string unit dictionary 1, and uses the results as
Set in the DP section.

Then, in the DP section, which combination of character string units is the most preferable is determined by the evaluation calculation described in connection with FIG.

Note that the function and configuration of the dictionary collation section 2 may be the same as those of the prior art, so the DP section 3 will be described in detail below.

FIG. 4 is a block diagram of one embodiment of the DP unit 3.

The explanation of each part is as follows.

WM: A memory that stores information in candidate character string units, and consists of the following parts A to P.
By inputting the address in WM and giving a signal to R, the information of each part is output one character string at a time, and by giving a signal to W, G and P
reads and stores information. A, B, V, and N are set by the dictionary matching section. The G section is initialized to 0 by the dictionary checking section.

A: Stores the type of forward connection relationship in character string units (hereinafter abbreviated as words).

B: Stores the type of backward connection relationship of words.

V: Stores the evaluation score (xi) that is determined regardless of the character strings surrounding the word.

N: Stores the word number of the word.

G: Stores the overall evaluation score G(xi) up to that word.

P: Stores the address in WM of the previous word that gives the best evaluation score up to that word.

EWM: A memory accessed as upper and lower addresses of the contents of C3 and C1,
The address in the WM where the information of the word ending at the boundary indicated by C3 is stored is set by the dictionary matching unit.

BWM: Similar to EWM, the address in WM where information on words starting at the boundary indicated by C3 is stored is set by the dictionary matching unit.

C1: When a signal is given to C1U, it increases by 1, and C
A counter that is cleared to 0 when signaled to 1C, indicating the order in the EWM of the ending word at a certain boundary.

C2: When a signal is given to C2C, it increases by 1, and C2
2C is a counter that is cleared to 0 and is signaled to indicate the order in the BWM of the ending word at a certain boundary.

C3: When a signal is given to C3C, it increases by 1, and C3C
This counter is cleared to 0 when the 3C signal is applied, and indicates the boundary number.

_r5 : A register indicating the upper limit of the boundary number for one sentence, and is set by dictionary comparison.

COMP ₄ : A comparator that compares the values of C3 and _r5 and issues a signal of C3E if C3> _r5 .

COPM ₁ : Logic that checks whether the output read from EWM is 0, that is, the code representing the end of a word for one boundary, and if it is 0, it issues a C1E signal.COMP ₃ : Same as COMP ₁ Logic that checks the output from BWM and issues C2E.

r ₄ : Register that temporarily stores the address in WM to read the judgment result.

MPX: WMA to EWM by the signal given to S
Address multiplexer to switch between the outputs of R4 and _R4 .

r ₁ : A register that holds the type of forward connection relationship of words starting from a certain boundary accessed by BWM from A of WM, and is loaded by the r ₁ L signal.

T: A constant memory that determines the score of the connection relationship determined by the forward connection relationship of words starting from a certain boundary and the backward connection relationship of words ending at that boundary, and is accessed by the value of part B of WM accessed by r ₁ and EWM. and outputs one score.

r ₂ : A register that holds the value of the V section of WM accessed by BWM, and is loaded by the r ₂ L signal.

ADD: An adder that adds the output of T, _r2 , and the value of the G part of WM accessed by EWM.

_r6 : A register that holds the maximum value of the ADD output during a series of processing for a certain word starting from a certain boundary, and is cleared by inputting the _r6C signal.

_r3 : A register that holds the address in WM of word information that gives the maximum value of the ADD output during a series of processing for a certain word starting from a certain boundary.

COMP ₃ : Comparator that compares the output of ADD and the output of r _6. If ADD output > r ₆ output,
Output the r ₃₆ L signal, output the ADD to r ₆ , and output the ADD signal to r ₃ .
Load EWM output. The gate inserted in _r36L is for synchronization with the CL signal.

TMG: Enter C1E, C2E, C3E, C1
U, C1C, C2U, C2C, S, R, W,
This is a timing control circuit that outputs r ₁ L, r ₂ L, r ₆ C, r ₄ L, C3C, and C3U, and controls each signal according to the operating procedure described below.

Figure 5 shows an example of the contents of EWM along the example in Figure 1, with X1, Y1~Y2, Z1~Z
6, ZZ1 to ZZ9, etc. mean addresses within WM.
For example, the boundary of C3=0011 is the arrow d in Figure 1.
corresponds to the position of The same applies to BWM, so it will be omitted.

The procedure for analyzing one sentence is shown below.

In this example, the output of the constant table T in Figure 4 is used as the score C (X, Y) based on the connection relationship between words X and Y, and the calculation of g (X ₁ ) + C (X, Y) In the procedure (Y) + max g (X ₂ ) + C (X ₂ , Y) V (Y) + G (X ₁ ) + T (X ₁ , Y) max V (Y) + G (X ₁ ) + T (X ₁ , Y ).

Also, r ₅ , EWM, BWM, A, B, V, N,
It is assumed that G has been initialized by the dictionary matching unit 2 as explained in each section. It is also assumed that B, V, and G that give the smallest possible ADD output are stored at address 0 of WM.

(1) Issue the C3C signal and clear C3 (boundary number) to 0.

(2) Issue the C2C signal and clear C2 (the order in BWM of words starting from that boundary) to 0.

(3) Emit the S signal and switch MPX to BWM output.

(4) Generate the R signal to cause WM to output A and V of the word indicated by C2 starting from that boundary.

(5) Generate r ₁ L, r ₂ L signals and load the outputs of A and V into r ₁ and r ₂ .

(6) Issue the C1C signal and clear C1 (the order in the EWM of words that end at that boundary) to 0.

(7) Emit S signal and switch MPX to EWM output.

(8) Issue the r ₆ C signal and clear r ₆ (the maximum value of ADD output for one word starting from that boundary) to 0.

(9) By generating CIU and CL signals until a signal appears in CIE at a constant cycle, the maximum value of the ADD output for one word starting from the boundary and the address in WM of the word information that gives the maximum value are r ₆ , stored in _r3 .

(10) Emit the S signal and switch the MPX output to the BWM output.

(11) Emit the W signal and convert the contents of r ₆ and r ₃ to G and P.
write to.

(12) Issue the R, r ₄ L signal and load the written contents of P into r ₄ .

(13) Emit C2U signal and increase C2U by 1.

(14) Repeat steps (4) to (13) until the C2E signal appears.

(15) Emit C3U signal and increase C3U by 1.

(16) Repeat steps (2) to (15) until a signal appears on C3E.

(17) Emit the S signal and switch the MPX output to _r4 .

(18) Emit R signal and output N.

(19) r ₄ Emit L signal and output N.

(20) By repeating (18) and (19), the word information N that is the determination result is sequentially read out.

Through the above procedure, the determination results are output in order from the last words in the sentence.

In the above embodiment, each memory, register, etc. is provided as dedicated hardware, but it is also possible to implement it by software using a general-purpose computer. FIG. 6 shows the processing flow.

As for the above-mentioned unique evaluation that does not depend on the surrounding character strings, the number of characters when the candidate character string is written in kana (including the number of beats and syllables when uttered) or the prefix,
An evaluation point corresponding to the number of characters including the suffix can be used.

Alternatively, general (or limited to a field of use) statistical appearance frequency (usage frequency) information of candidate character strings may be used. Furthermore, distinctions between independent words, affix words, etc. may be made using parts of speech. Or it may be a combination thereof.

In addition, the above-mentioned evaluation based on connection relationships includes frequency information of combinations of parts of speech before and after, frequency of connection between stems and endings, frequency of connections with affixes, frequency of connections at the beginning or end of sentences, frequency of connections with numbers and particles, etc. can be used. Alternatively, information such as which side of the entire sentence is most likely to be placed can also be used.

Further, in the above example, the evaluation calculation was performed from the beginning of the sentence to the end of the sentence, but it can also be performed from the end of the sentence to the beginning of the sentence.

Furthermore, processing may be performed in several parts and then integrated as a whole, or processing in both directions may be combined.

〔Effect of the invention〕

As described above, according to the present invention, the DP method can be easily used by expressing the validity of candidate character strings numerically, and therefore the processing is simple and the amount of processing is extremely small, and the optimal solution can be found. I can do it.

[Brief explanation of drawings]

FIG. 1 is an explanatory diagram comparing the present invention with a conventional example,
FIG. 2 is a conceptual diagram of the present invention, FIG. 3 is a schematic block diagram of the present invention, FIG. 4 is a block diagram of an embodiment of the present invention, and FIG. 5 is a diagram showing a specific example of the contents of EWM.
FIG. 6 is a processing flowchart of an embodiment of the present invention. In FIG. 3, 1 is a character string unit dictionary, 2 is a dictionary matching section, and 3 is a DP section.

Claims (1)

[Claims] 1. Analyzing a sentence consisting of a plurality of character string units,
In a text analysis device that identifies character string units constituting a sentence, a character string unit dictionary 1 has a character code assigned to each character string unit and used for matching each character string unit.
a dictionary matching unit 2 that matches the input sentence with the character code of the character string unit dictionary and extracts all candidate character string units that can be constituent units of the input sentence; a first evaluation point, which is an evaluation that does not depend on the situation;
A second evaluation score, which is the sum of the evaluation points up to the candidate character string and an evaluation based on the grammatical connection relationship of the candidate character string with other candidate character strings, is determined for each boundary unit each time, and the obtained result is calculated for each boundary unit. Using the first evaluation point and the second evaluation point, the fixed position between the candidate character string and other candidate character strings is determined each time according to dynamic programming for each boundary, and the character string identify units
A text analysis device characterized by comprising a DP unit 3. 2. The text analysis device according to claim 1, wherein part or all of the first evaluation score is information corresponding to the number of characters when the candidate character string unit is written in kana. 3. The text analysis device according to claim 1, wherein part or all of the first evaluation score is usage frequency information for each candidate character string. 4. The text analysis device according to claim 1, wherein part or all of the first evaluation score is part-of-speech information for each candidate character string. 5. The text analysis device according to claims 1 to 4, wherein a part or all of the second evaluation score is positional information of the candidate character string in the text.