JPH03168863A - Method for constituting connection probability dictionary - Google Patents

Abstract

PURPOSE:To omit useless retrieval by detecting three-character connection probability by the number of three-character connection probability retrievals when the probability is not registered. CONSTITUTION:A two-character connection probability registering part 11 is retrieved by a two-character string Sk<2> key at first in order to find out connection probability, and the number of three-character connection probability retrievals of a corresponding record is '0', the retrieval of the three-character connection probability is inhibited. Thereby the absence of a record corresponding to a three-character string Sr<3> (=Sk<2>Ckm) key in a three-character connection probability registering part 12 can be detected at the time of retrieving the registering part 11. Consequently, useless retrieval can be removed.

Description

[Detailed description of the invention] [Industrial application field] ? The present invention relates to a dictionary construction method for storing the probability of concatenation between characters in order to correct misspellings and omissions in Japanese texts input into a computer to characters with a high probability of concatenation between characters.

[Prior art]

When entering Japanese text into a computer using a word processor or character reading device, typos or omissions may occur. As a method for automatically detecting and correcting input errors, we focus on the variation in the frequency of occurrence of combinations between consecutive characters in a sentence, and select characters with a high probability of concatenation with the correct characters surrounding the typo as correction candidates. method is used.

The concatenation probability is the probability that a specific character will appear next when the previous character string is determined, and is the probability that a specific character will appear next, and n − 1 character string CLC
,...Cn-■ The probability that character C0 appears next to n character string C1C,...Cn is called the n-character concatenation probability concatenation probability, and p ( c n l c ic z...C
Il-, ). The concatenation probability is derived from the frequency of appearance of n character strings and n-1 character strings contained in a large amount of sentences without input errors (hereinafter referred to as original text data) using the formula [''. The derived concatenation probability is registered in the conjunction probability dictionary.

P(CQI C1C2・C,.) (frequency of appearance of n character string C1C,...CI1)? (
CIC,...CI1-■) Here, let the frequency of appearance of the character string S be A (S). When correcting typographical errors, candidate characters are inserted at the location of the typographical error in accordance with the error characteristics of humans and character reading devices. There are usually multiple candidate characters, and it is necessary to narrow down the appropriate candidate characters from among them.

As an evaluation function of evaluation characters of candidate characters, [define the probability of correct characters F in the 2-co expression. Candidate characters with a high probability of being correct are
It is a candidate character that has a high probability of concatenation with adjacent characters, and is likely to be an orthographic character. F =QyuX QzX Q3
...[2] Here, Q,=P (CL /C■
-20yo-1) Q2=P (c..■/C A, CL
)...[3]Qi=P (Ci.2/Ci Cd.
1)? i is a typo, and CL-20■-■ are the two characters immediately before the typo, CL. , Ci. , are the two characters immediately after the typo. That is, the orthographic probability is the product of three three-letter concatenation probabilities including the letter C in the five letters Ci-2CL-, C, C, and Ci and 2.

Generally, the larger the character concatenation length n in equation [1], the more accurately the original text data can be approximated, and the higher the ability to correct typos. On the other hand, as n becomes larger, the appearance frequency of each character string C1C2...co becomes smaller. Even though character strings C1C2...Cn may exist linguistically, they are not included in the limited original text data, so they are often not registered in the conjunctive probability dictionary. If any of the 3-character concatenation probabilities in equation [3] are not registered in the concatenation probability dictionary because the original text data is not large enough, the following 2-character concatenation probabilities are substituted for each.

? E=p (ce-■/Ci2) Qx=p (Cl/ci-)...
[4] Q. (Sufficiently smaller than the connection probability).

Q,=Pd Q,=Pd...[5]Q,=Pd The connection probability dictionary registers the connection probabilities of equations [3] and [4].

Figure 4 shows the structure of a conventional conjunctive probability dictionary. In FIG. 4, 1 is a 2-character concatenation probability registration section, and 2 is a 3-character concatenation probability registration section. 3 is part of Ki, 2 character string Sk
2 (k=1.2, . . . , α) is registered. 4 is a data section in which the probability of two characters concatenating in the two character string Sk2 is registered. The two-character concatenation probability registration unit 1 consists of records consisting of a pair of two-character string Sk'' and its -2-character concatenation probability, and searches for the two-character concatenation probability for the two-character string Sk''. 5 is the key part, and the 3 character string Sr' (r=1.2, ・=
β) is registered. 6 is the data part, 3 character strings Sr
The 3-character concatenation probability of " is registered. The 3-character concatenation probability registration unit 2 consists of a record consisting of a set of a 3-character string Sr' and its 3-character concatenation probability, and searches for the 3-character concatenation probability for the 3-character string Sr'. .

The procedure for determining the connection probability is shown in FIG.

Here, press the 3 character string Sr3 keys in order: CL-2C, C, C, 1C, C. 1, CICI, C. By setting it to 2, Qe, Q2, and Q3 are obtained, respectively. Also, the 3-character string Sr3 key is the 2-character string Sk” and the character Ck.
It is expressed as lI (m = 1.2, -, δlkl, where δ, k. is the number of 3-character string Sr3 keys whose first two characters are ``Sk''). The basic idea of this procedure is as follows. As shown below.■.Search the 3-character concatenation probability registration unit 2 using the 3-character string Sr3 as a key (step 501).If there is a record L corresponding to the 3-character string Sr' key, the 3-character string The concatenation probability of the three character string Sr3 is read from the record L corresponding to the Sr3 key (step 506).

■． If there is no record L corresponding to the 3-character string Sr'' key (step 502), the 2-character concatenation probability registration unit 1 is searched using the 2-character string Sk'' as a key (step 503).
．． Then, the record L corresponding to the 2-character string key “Sk”,
If there is, the record L2 corresponding to the 2-character string Sr3 key
Read the concatenation probability of two character strings Sk” from (Step 5
05).

■． Further, if there is no record corresponding to the two-character string Sk'' (step 504), a default value Pd is set as the concatenation probability (step 507).

[Problems to be solved by the invention]

However, in the conventional concatenation probability dictionary, the 3-character concatenation probability registration section 2 is first searched using the 3-character string key. Generally, the three-character concatenation probability is often not registered, and in this case, it is necessary to find a substitute two-character concatenation probability by searching the two-character concatenation probability registration section 1 using the two-character string key. As a result, there was a problem in that the number of dictionary searches increased and the time required to correct typos increased. The present invention has been made to solve the above problems.

An object of the present invention is to provide a method for configuring a conjunctive probability dictionary that can reduce the number of times the dictionary is searched and can be searched at high speed.

The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

[Means to solve the problem]

In order to achieve the above object, the present invention uses two character strings as keys, and calculates the concatenation probability of the two character strings and . 2 consisting of a record that registers the 3-character concatenation probability pointer and the number of 3-character concatenation probability searches.
It consists of a character concatenation probability registration unit and a record that registers the concatenation probability of the 3-character string using the last character of the 3-character string as a key, and is limited by the 3-character concatenation probability pointer and the number of 3-character concatenation probability searches. A construction method of a conjunctive probability dictionary having a character concatenation probability registration section, in which the two-character concatenation probability registration section is searched using the two character strings as a key, and the number of three-character concatenation probability searches for the corresponding record is ''O lj. If there is, the search for 3-character concatenation probability will not be performed, and the number of 3-character concatenation probability searches will be "0"
If it is not ++, the main feature is to search for a set of records in the 3-letter concatenation probability section.

[Effect]

According to the above-mentioned means, in order to find the connection probability, first,
Search the 2-character concatenation probability registration section using the 2-character string key, and find the number of 3-character concatenation probability searches for the corresponding record.
If it is 7, the search for 3-character concatenation probability is not performed, so 3
It is possible to detect that a record corresponding to a character string key does not exist in the 3-character concatenation probability registration section when searching the 2-character concatenation probability registration section, thereby eliminating unnecessary searches. If the number of 3-character concatenation probability searches is not ``O'', there is a record corresponding to the 3-character string key, and in this case, the 3-character concatenation probability is limited from the 3-character concatenation probability pointer and the 3-character concatenation probability search number. By searching the set of records in the registration section, the range of the 3-character concatenation probability registration section to be searched is limited, so the search time can be reduced.In other words, the 2-character concatenation probability registration section is This is an index that not only registers the concatenation probability but also limits the search range of the three-character concatenation probability registration section.

These make it possible to speed up dictionary searches.

[Embodiments of the invention]

Hereinafter, one embodiment of the present invention will be specifically described using the drawings.

FIG. 1 is a diagram showing the structure of a conjunctive probability dictionary for explaining an embodiment of the conjunctive probability dictionary structure method of the present invention.

In FIG. 1, "" is a two-character concatenation probability registration part, and 1
2 is a three-character concatenation probability registration section. Reference numeral 13 is a key part, in which two character strings Sk2 (k=1.2, . . . , α) are registered. Reference numeral 14 is a data section, in which the 2-character concatenation probability of the 2-character string Sk''' is registered. 15 is a 3-character concatenation probability pointer section, in which the 3-character concatenation probability pointer is registered. 16 is 3.
This is a character concatenation probability search number registration unit that registers the number of three-character concatenation probability searches. 2-character concatenation probability registration part” is 2-character string Sk
2. Consists of records that register sets of 2-character concatenation probability, 3-character concatenation probability pointer, and 3-character concatenation probability search number.

17 is the key part, and the 3 character string Sr3 (=Sk”Ck
ll) Register the key. In general, there are three specific
If a character string exists, two character strings consisting of the first two characters of the three character strings are sure to exist in the original data.

In other words, the 3-character string Sr3 (=Sk”C
k,) key is registered, the 3-character string Sr3(
2-character string Sk” consisting of the first two characters of =Sk2Ck,)
The key is always registered. Therefore, the 3-character string Sr3(
=Sk"Ck.) By registering only the last character Ck. of the key in the key part 17, the memory amount of the key part 17 is reduced. 18 is a data part, and the 3 character string Sr" key is Register the connection probability.

FIG. 2 shows the procedure for searching the conjunction probability dictionary in this embodiment.

First, the two-character concatenation probability registration section " is searched using the first two character strings Sk" of the three-character string Sr3 (=Sk"Ck.) key as a key (step 201). Next, the two-character string Sk" is searched. It is detected whether or not the key exists in the 2-character concatenation probability registration unit (Step 2O2), and if the 2-character string Sk2 key does not exist, that is, if there is no record L2 corresponding to the 2-character string Sk” key, (No). A default value Pd is set as the connection probability (step 203). If the 2-character string Sk” key exists, that is, if there is a record L2 corresponding to the 2-character string Sk2 key (YE
S), read the record L2 (step 204)
, the number of 3-character concatenation probability searches for record L2 is ′゛O′
” (step 205), and the number of 3-character concatenation probability searches for the record L2 is “O” (Y
ES), the concatenation probability of the two character strings Sk2 is read from the record L2 (step 206). Record L2 3
If the number of character concatenation probability searches is not it O #j (No), the record indicated by the 3-character concatenation probability pointer is the first record, and the records corresponding to the number of 3-character concatenation probability searches are searched, and the 3-character string Sr3 key is searched. The three-character concatenation probability registration unit 12 is searched using the last character Ckll as a key (step 207).

Next, it is detected whether or not the character Ck exists in the 3-character concatenation probability registration unit 12 (step 208), and the character Ck,
k, if the key exists (YES), i.e. the character C
k, if there is a record L3 corresponding to the key (YES),
Read that record L3, and 3 from that record L3.
Read the concatenation probability of character string Sr3 (step 209)
. The character Ck. If the key does not exist (No), that is, since there is no record L3 corresponding to the character Ck and the key, the concatenation probability of the two character strings S1 is read from the record L instead (step 210).

As can be seen from the above description, the basic idea of the procedure for searching the conjunctive probability dictionary of the present invention is as follows.

■． 2-character concatenation probability registration unit 1 using the character string “Sk” as a key
Search for engineering.

■． If there is no record L2 corresponding to the 2-character string Sk" key, a default value Pd is set as the concatenation probability. ■.If there is a record L, corresponding to the 2-character string Sk" key, the 3-character concatenation of that record L2 Probability search number is “O”
Only when the character Ck. Using as a key, the 3-character concatenation probability registration unit 12 is searched for a record set that can be limited by the 3-character concatenation probability pointer and the 3-character concatenation probability search number. If the number of 3-character concatenation probability searches is "0 71," the concatenation probability of the 3-character string Sr3 is not registered in the 3-character concatenation probability registration unit 12, so the concatenation probability of the 2-character string Sk'' is obtained from the record L2 instead. read out.

■． Letter Ck. If there is a record corresponding to , the concatenation probability of the 3-character string Sr3 is read from the record L.

If there is no record corresponding to the character Ck and the key, the concatenation probability of the two character strings Sk'' is read from the record L2.

FIG. 3 is a diagram showing a specific example of an embodiment of the conjunctive probability dictionary that is the object of the present invention. ” for each two-character concatenation probability o. s, o.
In this example, g and 0.1 are registered. In the 3-character concatenation probability registration unit 12, the 3-character concatenation probability registration unit 12 stores
The records for “r” and “ho” are each 3 character strings r
Conjunction probabilities of ``International language'', ``International color'', ``International'', and ``International law'' are registered as 0.2, 0.4, 0.3, and O. Wang.These records have 2-letter concatenation probabilities. It is limited by the 3-letter concatenation probability pointer and the number of 3-letter concatenation probability searches (=4) for the 2-character string r international in the registration section. There are 3 records each with the characters ``ka'' and ``r-ha'' as keys.
The concatenation probability of the character strings “individualization” and “individualization” is 0.6, 0.
4 are registered. These records are the 2-character string ``The 3-character concatenation probability pointer on the record whose key is ``2-character concatenation probability registration section'' and the 3-character concatenation probability search number (=
2). The number of 3-character concatenation probability searches for records with the 2-character string ``AC'' as a key in the ``2-character concatenation probability registration section'' is at O u. Therefore, it can be seen that there is no record in the 3-character concatenation probability registration unit 12 that has a 3-character string whose first two characters are the 2-character string "AC" as a key. Next, the search time of the conventional conjunctive probability dictionary construction method and the conjunctive probability dictionary construction method of this embodiment will be evaluated. To simplify the evaluation, we make the following assumptions.

■． Characters of all character types have the same frequency of appearance. ■． The probability of retrieving a record is equal for all records in the connection probability dictionary.

Assuming that the number of character types is A, the number of 2-character string keys is α, and the number of 3-character string keys is β, (Search time of conventional conjunctive probability dictionary construction method) time) + (probability that the key is not registered in the 3-character concatenation probability registration section) x (search time of the 2-character concatenation probability registration section) = log β + (engine - β / A3) Xlog ct"
・[6] (Search time of conjunctive probability dictionary construction method of this embodiment)
cx: (Search time of 2-letter concatenation probability registration section) + (Probability that a key is registered in 3-character conjunctive probability registration section) x (Search time of 3-character conjunctive probability registration section) = 10gα + (β/A'') Xlog(β/α)...
[7] In the specific example of FIG. 3, α=3 and β=6. Also, assume that the types of Japanese characters are A=7000. Substituting these into equations [6] and [7], (search time of conventional conjunctive probability dictionary construction method) (X: 2.6 (search time of conjunctive probability dictionary construction method of this embodiment) ccl, 6. In other words, the search time can be reduced by about 40% (40%).

As can be seen from the above explanation, according to this embodiment, in order to obtain the concatenation probability, first, the 2-character concatenation probability registration field is searched using the 2-character string S1 key, and the number of 3-character concatenation probability searches for the corresponding record is If tg O tp, the 3-character string Sr3 (=Sk
It is possible to detect that the record corresponding to the ``Ck.) key does not exist in the 3-character concatenation probability registration section 12 at the time of searching the 2-character concatenation probability registration section'', thereby eliminating unnecessary searches. If the number of 3-character concatenation probability searches is not ri O tp, there is a record corresponding to the 3-character string Sr3 (=Sk”Ck,) key, and in this case, the 3-character concatenation probability pointer and the 3-character concatenation probability search number By searching the set of records in the 3-character concatenation probability registration section 12 that is limited from , it is possible to speed up dictionary searches.

Although the present invention has been specifically described above based on the embodiments described above, it goes without saying that the present invention is not limited to the embodiments described above, and can be modified in various ways without departing from the gist thereof.

〔Effect of the invention〕

As described above, according to the present invention, the following effects can be obtained when searching for three-character concatenation probability.

■． A case where a 3-character concatenation probability is not registered can be detected by the number of 3-character concatenation probability searches, and unnecessary searches can be omitted, so that the search time can be shortened.

■． Even if the 3-character concatenation probability is registered, the search range can be limited from the 3-character concatenation probability pointer and the 3-character concatenation probability search number, so the search time can be shortened.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing the structure of a conjunctive probability dictionary for explaining an embodiment of the conjunctive probability dictionary construction method of the present invention, and FIG. 2 is an example of the conjunctive probability dictionary construction method of the present invention. FIG. 3 is a diagram showing a specific example of an embodiment of the conjunctive probability dictionary that is the subject of the present invention. FIG. 4 is a diagram showing the structure of the conventional conjunctive probability dictionary. 5 are diagrams showing a conventional search procedure for a conjunctive probability dictionary. In the figure, 1... 2-character concatenation probability registration section, 2... 3-character concatenation probability registration section, 3... Part of Ki, 4... Data section,
5... Ki part, 6... Data section, ``... 2-character concatenation probability registration section, 12... 3-character concatenation probability registration section, 13.
... Ki part, 14... Data part, 15... 3-character concatenation probability pointer, 16... 3-character concatenation probability search number, 1
7...Ki part, 18...Data part.

Claims (1)

[Claims] (1) Using two character strings as keys, the concatenation probability of two character strings, and 3
A 2-character concatenation probability registration section consisting of a record for registering a character concatenation probability pointer and a 3-character concatenation probability search number, and a record for registering the concatenation probability of 3-character strings using the last character of the 3-character string as a key, A construction method of a conjunctive probability dictionary having a 3-character concatenated probability registration section limited from the 3-character concatenated probability pointer and the 3-character concatenated probability search number, wherein the 2-character concatenated probability registration section is searched using the 2-character string as a key. However, if the number of 3-letter concatenation probability searches for the corresponding record is "0", the 3-letter concatenation probability search is not performed, and if the 3-letter concatenation probability search number is not "0", the 3-letter concatenation probability registration section A conjunctive probability dictionary construction method characterized by searching a set of records.