JP3939264B2 - Morphological analyzer - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a morpheme analyzer that divides an input natural language sentence into morphemes (for example, words), and in particular, intends to improve analysis processing time and / or analysis accuracy.
[0002]
[Prior art]
[0003]
[Patent Document 1]
JP-A-5-52543
[0004]
[Non-Patent Document 1]
Mikio Yamamoto, Masakazu Masuyama, “Japanese Morphological Analysis Using Chain Probabilities of Extended Characters Containing Part-of-Speech and Separation Information”, Proc. Of the 3rd Annual Conference of the Language Processing Society, March 1997
Due to the increase in opportunities for creating text with word processors and the spread of Internet-compatible devices, a large amount of digitized natural language sentences has become readily available. For natural language processing systems that handle a large amount of natural language sentences, such as character recognition systems, machine translation systems, information retrieval systems, information extraction systems, etc., morphological analysis processing is common before various systems perform specialized processing. This is an extremely important process for determining morphemes that are semantic units in sentences such as words and phrases.
[0005]
In such morphological analysis processing, high accuracy of word division (morpheme division) is required, and processing speed of processing a large amount of natural language sentences at high speed is also required.
[0006]
As a conventional morpheme analysis method, a morpheme dictionary (word dictionary), a utilization ending table, a part-of-speech connection table, and the like are generally used, and morpheme analysis is performed while accessing these various storage units (Patent Literature). 1).
[0007]
Recently, a morphological analysis method using a character-based probability model has also been proposed (see Non-Patent Document 1, Japanese Patent Application No. 9-68300, and drawings).
[0008]
This morpheme analysis method is most probable among all combinations of whether or not a morpheme boundary immediately follows each character as a morpheme sequence constituting the input sentence when a natural language text is given as an input sentence. Outputs a sequence of morpheme strings.
[0009]
In order to determine whether or not the most probable morpheme sequence is arranged, a probability model (statistical database; analysis execution data database) learned from a large amount of text data (corpus; learning data) is used. A set of analysis execution data stored in the statistical database is, for example, an extended character string of N characters and chain probability data indicating how much the extended character string appears on the corpus. is there. Note that extended characters are different from normal characters such as “I” and “ha”, and include at least morpheme delimiter information (whether or not this character is immediately followed by a morpheme delimiter). Information is added.
[0010]
[Problems to be solved by the invention]
(1) The morpheme analysis method, which has been generally used in the past using a morpheme dictionary, draws a morpheme dictionary so as to determine a morpheme whose length is unknown at the input stage. It took a considerable amount of time to look up the dictionary, and a large amount of documents could not be processed in a short time. That is, the user cannot quickly obtain the morphological analysis result. Hereinafter, in some cases, this type of morphological analysis is referred to as low-speed morphological analysis.
[0011]
(2) On the other hand, the morpheme analysis method using a character-based probability model (statistical database) compares an extended character string of a predetermined number of characters (N) determined from an input sentence with the stored contents of the statistical database. Since the morpheme analysis is basically performed, the morpheme analysis result can be obtained faster than the morpheme analysis method (slow morpheme analysis method).
[0012]
However, in this morphological analysis method, it is necessary to learn and prepare parameters (statistical data; analysis execution data) in advance, and it is difficult to prepare learning data (corpus) for that purpose. Hereinafter, in some cases, this type of morphological analysis is referred to as high-speed morphological analysis.
[0013]
Necessary learning data (corpus) can calculate statistical data consisting of the above-mentioned extended character strings and their chain probabilities, so information on morpheme breaks (and part-of-speech information on morphemes) is added to the text file. It is a thing. Text files are easy to obtain, but files with the above-mentioned information added are rare at present, and humans add the above-mentioned information one by one to the text file to create learning data (corpus) used for learning Was. Alternatively, the learning data (corpus) used for learning is created by manually correcting the result of the low-speed morphological analysis.
[0014]
In high-speed morphological analysis, even if the above learning data is prepared and a statistical database is created, it is not possible to correctly analyze a character string that is not included in the learning data prepared in advance. Even in low-speed morpheme analysis, of course, a character string consisting of morphemes (unknown words) that are not in the dictionary cannot be analyzed correctly, but a normal morpheme analysis dictionary has a dictionary of tens of thousands to hundreds of thousands of words. It is rare to encounter character strings (unknown words) that cannot be parsed correctly. If it is possible to prepare learning data that contains all morphemes in the dictionary for low-speed morpheme analysis at an appropriate frequency and learn high-speed morpheme analysis using it, in principle, low-speed morpheme analysis and High-speed morphological analysis can be analyzed with almost the same accuracy. That is, it can be said that the character strings that cannot be analyzed correctly are the same.
[0015]
However, it is practically impossible to prepare learning data that includes all morphemes in the dictionary for low-speed morpheme analysis at an appropriate frequency. As a result, in high-speed morpheme analysis, the analysis accuracy of sentences in which morphemes that are not in the learning data appear is inferior to that of low-speed morpheme analysis.
[0016]
(3) If there is no way for the user of high-speed morpheme analysis to know what morpheme the learning data adopted by the morpheme analysis method is composed of, If you look at the analysis results and determine that the accuracy of the high-speed analysis results is poor, you must either use low-speed morphological analysis only for the text or manually correct only the text. Absent.
[0017]
If the text you want to analyze is in various fields, checking the analysis results one by one is a tedious task, and if you do not check, the overall accuracy when using high-speed morphological analysis Will get worse.
[0018]
Examples of texts that you want to analyze in various fields include morphemes to create index files for search services by examining the frequency of morphemes that appear by performing morphological analysis on document files on various WWW servers on the Internet. There are cases where analysis is used.
[0019]
(4) By the way, after manually checking the result of high-speed morphological analysis with low accuracy, it is also possible to reflect (feed back) the data to a statistical database.
[0020]
In this way, after reflection processing, it is possible to analyze sentences in the same field as that field with the same degree of accuracy, but it is troublesome because the work of manual check is not eliminated. .
[0021]
Therefore, there is a demand for a morpheme analyzer that takes a short time to obtain a highly accurate morpheme analysis result on average.
[0022]
[Means for Solving the Problems]
In order to solve such a problem, the present invention provides a partial character string consisting of a predetermined number of characters appearing in a natural language sentence, its absolute or relative frequency information, and the character assigned to each character of the partial character string. Analysis execution time data storage means for storing a lot of analysis execution time data, which is a set of data including at least the position information of whether or not the character is the last character of the morpheme, and the above analysis execution time for an unknown sentence A morpheme analyzer having a first morpheme analysis unit that executes morpheme analysis with reference to the stored contents of the data storage unit is characterized as follows. That is, when the frequency information in the analysis execution time data stored in the analysis execution time data storage means applied by the first morpheme analysis means to the unknown sentence is smaller than a threshold and / or When the unknown sentence has a portion to which the analysis execution data stored in the analysis execution data storage means cannot be applied, the accuracy of the morphological analysis result from the first morpheme analysis means is estimated to be low accuracy Accuracy judging means to perform, or whole or part of the unknown sentence among Character of specific character type Is connected more than the specified number of characters Sometimes, accuracy determination means for estimating the accuracy of the morpheme analysis result from the first morpheme analysis means as low accuracy Or an accuracy determination that estimates that the accuracy of the morpheme analysis result from the first morpheme analysis means is low when a character of a specific character type is included in the whole or part of the unknown sentence by a predetermined number or more Have means It is characterized by that.
[0023]
DETAILED DESCRIPTION OF THE INVENTION
(A) First embodiment
Hereinafter, a first embodiment of a morphological analyzer according to the present invention will be described in detail with reference to the drawings.
[0024]
The morpheme analyzer of the first embodiment basically analyzes an input sentence by a high-speed morpheme analysis method, and automatically converts low-speed morpheme analysis results into learning data for high-speed morpheme analysis. It has a great feature that it has a learning function to make it possible to easily use sentences that have not been used as learning data so far as learning data.
[0025]
FIG. 1 is a functional block diagram showing the configuration of the morphological analyzer of the first embodiment. That is, the morphological analyzer of the first embodiment is actually realized on an information processing apparatus such as a workstation or a personal computer having an input / output device, a processing device, a storage device (and a communication device), and the like. However, functionally, it has the structure shown in FIG.
[0026]
In FIG. 1, a morpheme analyzer 10 according to the first embodiment includes a low-speed morpheme analysis unit 11, a low-speed morpheme analysis result storage unit 12, a conversion unit 13, a learning data storage unit 14, a learning unit 15, and an analysis execution time data storage. Means 16 and high-speed morpheme analysis means 17 are provided. Among these components, the low-speed morpheme analysis unit 11, the low-speed morpheme analysis result storage unit 12, the conversion unit 13, the learning data storage unit 14, and the learning unit 15 constitute the analysis execution time data creation device of the first embodiment. is doing.
[0027]
Although the detailed configuration of the low-speed morpheme analysis unit 11 is omitted, the low-speed morpheme analysis unit 11 has the same configuration as a conventional low-speed morpheme analyzer that performs morpheme analysis using a built-in morpheme dictionary. That is, it has the same detailed configuration as the morphological analyzer described in Patent Document 1 described above and a similar device. In the case of the first embodiment, the low-speed morpheme analyzing unit 11 is not provided for performing morphological analysis on each sentence in the unknown document, but the analysis execution time data stored in the analysis execution time data storage unit 16. It is provided as one element in the configuration for creating. A learning document is input to the low-speed morpheme analysis unit 11. In FIG. 1, a block described as a learning document also means a learning document input unit.
[0028]
The low-speed morpheme analysis result storage unit 12 stores the low-speed morpheme analysis result executed by the low-speed morpheme analysis unit 11 for each sentence of the learning document.
[0029]
The conversion unit 13 converts the data format of the low-speed morpheme analysis result stored in the low-speed morpheme analysis result storage unit 12 into a data format as learning data required by the high-speed morpheme analysis device.
[0030]
The learning data storage means 14 stores learning data (corpus) obtained by the conversion means 13 converting.
[0031]
The learning unit 15 creates analysis execution time data that the high-speed morpheme analysis unit 17 refers to at the time of morpheme analysis of an unknown sentence from the learning data stored in the learning data storage unit 14. That is, for example, an analysis execution data (statistical data) consisting of an extended character string having a predetermined number of characters N appearing on the learning data and a chain probability indicating how much the extended character string appears on the learning data Is created from learning data. As a method of creating analysis execution data from learning data, the methods described in Non-Patent Document 2, Japanese Patent Application No. 9-350651, and drawings can be applied.
[0032]
[Non-Patent Document 2]
Satoshi Nagao and Shinsuke Mori, “How to make n-gram statistics and automatic phrase extraction for large-scale Japanese text”, IPSJ Research Report Natural Language Processing 96-1, July 1993
The analysis execution time data storage means (statistical database) 16 stores the analysis execution time data created by the learning means 15. FIG. 11 described later shows a part of the analysis execution time data (when N is 3) stored in the analysis execution data storage means (statistical database) 16. Note that the chaining probability of the analysis execution data is determined such that, for example, the sum of the chaining probabilities of a plurality of N extended character strings having the same N-1 character on the leading side is 1.
[0033]
The high-speed morpheme analysis unit 17 executes morpheme analysis for each sentence by referring to the contents stored in the analysis execution data storage unit 16 when an unknown document or unknown sentence to be analyzed is given. The obtained morpheme analysis result is output. The high-speed morphological analysis means 17 is realized by, for example, the configuration described in Non-Patent Document 1 described above, Japanese Patent Application No. 9-68300 and the drawings, or a configuration similar thereto.
[0034]
Although illustration is omitted, a detailed configuration example of the high-speed morpheme analysis means 17 is as follows. That is, the high-speed morpheme analysis unit 17 includes a score table 17a, an extended character string generation unit 17b, a chain probability calculation unit 17c, and an optimum route search unit 17d.
[0035]
The score table 17a is based on the path of all the extended character strings from the beginning to the end of the sentence of the unknown sentence to be analyzed, and the chain probability of the extended character strings of a predetermined number of characters stored in the analysis execution time data storage means 16. The chain probability corresponding to the obtained extended character string path is stored. The extended character string generation unit 17b generates extended characters for the unknown sentence to be analyzed, and stores all combinations (paths) of the extended characters in the score table 17a. The chain probability calculation unit 17c calculates the chain probability for each path of the extended character string stored in the score table 17a based on the chain probability stored in the analysis execution time data storage unit 16. The optimum path searching unit 17d selects an extended character string satisfying an optimum condition (for example, giving a maximum chain probability) from among the chain probabilities calculated by the chain probability calculating unit 17c. Result).
[0036]
In FIG. 1, a block described as an unknown document also means an input unit for an unknown document, and a block described as a morpheme analysis result also means an output unit for a morpheme analysis result. Yes.
[0037]
Next, the outline of the processing of the morphological analyzer 10 of the first embodiment will be described in detail with reference to the flowchart of FIG. The processing of the morphological analysis device 10 of the first embodiment is divided into a preparation stage processing for executing morphological analysis of an unknown document and a processing for executing morphological analysis of an unknown document. Step 100 in FIG. ˜102 corresponds to the former process, and step 103 corresponds to the latter process.
[0038]
When the learning document is input to the morpheme analyzer 10, the learning document input by the low-speed morpheme analysis unit 11 using the morpheme dictionary is analyzed, and the morpheme analysis result is stored in the low-speed morpheme analysis result storage unit 12. It is written (step 100).
[0039]
At this time, the data format of the stored morpheme analysis result is naturally an output data format by the low-speed morpheme analysis unit 11. Such a low-speed morpheme analysis result is converted by the conversion unit 13 into the data format of the learning data from which the analysis execution time data used by the high-speed morpheme analysis unit 17 is generated and stored in the learning data storage unit 14. (Step 101).
[0040]
The learning data is processed by the learning means 15 to create analysis execution time data, and the created analysis execution time data is stored in the analysis execution time data storage means 16 (step 102).
[0041]
When an unknown document is input after the processing of the preparation stage of the high-speed morpheme analysis process as described above is completed, the high-speed morpheme analysis unit 17 performs an analysis execution time data storage unit 16 for each sentence of the unknown document. The morpheme analysis is performed with reference to the stored contents of the data, and the obtained morpheme analysis result is output (step 103).
[0042]
FIG. 3 shows an example of a learning document input to the low-speed morpheme analyzing unit 11. As shown in FIG. 3, the learning document is natural language text data that does not accompany extended information or tags.
[0043]
The output data format (morpheme analysis result data format) varies depending on the internal configuration of the low-speed morpheme analyzer 11 applied. FIG. 4 shows an output example (output data format example) as a result of low-speed morphological analysis of the first sentence in FIG. Each line in FIG. 4 shows information of one word. The information of one word consists of three pieces of information separated by a space, and each of them is a part of speech, a standard form, and an appearance form. In the case of nouns that are not used, the standard form and the used form are the same.
[0044]
FIG. 5 shows an example of learning data corresponding to FIG. The conversion unit 13 converts the data of the low-speed morpheme analysis result illustrated in FIG. 4 into the learning data as illustrated in FIG.
[0045]
In the example shown in FIG. 5, the data at the time of analysis execution is an extended character string of N characters, and chain probability data indicating how much the extended character string appears on the corpus, This is an example corresponding to the case where the extended character is obtained by adding morpheme separation information as extended information to a normal character such as “I” or “ha”. Note that the extension information including part-of-speech information in addition to the morpheme segmentation information is different from the format shown in FIG.
[0046]
In FIG. 5A, the morpheme boundary (separation) is represented by an extended character string in which “1” indicates a morpheme delimiter between the preceding characters and “0” otherwise. An example is shown. FIG. 5B is an example in which the same contents as FIG. 5A are represented by slashes (/) as morpheme breaks.
[0047]
FIG. 6 is a flowchart showing an example of the flow of conversion processing by the conversion means 13. FIG. 6 corresponds to the case where the data format after conversion is as shown in FIG.
[0048]
First, it is confirmed whether or not the low-speed morpheme analysis result for which the conversion process has not been completed remains in the low-speed morpheme analysis result storage unit 12 (step 200). If not, a series of conversion processes are terminated.
[0049]
On the other hand, if the unprocessed low-speed morpheme analysis result remains in the low-speed morpheme analysis result storage unit 12, the unprocessed low-speed morpheme analysis result is read by one sentence (step 201). Then, appearance item is extracted from the read low-speed morpheme analysis result (step 202), and characters of each appearance form are converted into extended characters to create an extended character string (step 203). Then, the obtained extended character string is stored in the learning data storage means 14 and the process returns to step 200 described above (step 204).
[0050]
Here, in the conversion to an extended character, “1” indicating that it is a morpheme break is given only to the last character of the appearance form, and “0” indicating that it is not a morpheme break is assigned to other characters. To do. For example, as shown in FIG. 7, if the appearance form is “machine translation”, <machine, 0><machine,0><translation,0><translation,1> is obtained as an extended character string for it. .
[0051]
According to the first embodiment, the learning function for automatically converting the low-speed morpheme analysis result into the learning data for creating the analysis execution data used in the analysis by the high-speed morpheme analysis method is provided. It is possible to realize a morpheme analyzer that basically follows a high-speed morpheme analysis method that can easily use a sentence that was not used as data as learning data. As a result, the learning data can be enhanced and the accuracy of the high-speed morphological analysis results for unknown documents can be expected.
[0052]
In addition, since a learning function that automatically converts low-speed morphological analysis results into learning data is provided, the user only has to input a learning document to the device, and create learning data from the learning document. Or creating learning data from the low-speed morpheme analysis result can be made unnecessary.
[0053]
(B) Second embodiment
Next, a second embodiment of the morphological analyzer according to the present invention will be described in detail with reference to the drawings.
[0054]
The morpheme analyzer of the second embodiment discriminates whether the accuracy of the high-speed morpheme analysis result is good or not, and if not, presents the result to the user, and the accuracy is low to the user. The main feature is that it is left to the judgment of the case.
[0055]
FIG. 8 is a block diagram showing a functional configuration of the morphological analyzer 10A according to the second embodiment. Components identical or corresponding to those in FIG. 1 according to the first embodiment described above are denoted by the same reference numerals. It shows.
[0056]
In FIG. 8, a morpheme analyzer 10A according to the second embodiment includes an analysis execution time data storage unit 16, a high-speed morpheme analyzer 17, an accuracy determination unit 18, and an accuracy / analysis result synthesis unit 19.
[0057]
If the method for creating analysis execution data stored in the analysis execution data storage unit 16 is the same as that in the first embodiment, the illustration is omitted, but the low speed morpheme analysis unit 11 and the low speed morpheme analysis are omitted. The result storage means 12, the conversion means 13, the learning data storage means 14, and the learning means 15 are also provided (refer FIG. 1). A description of these components is omitted.
[0058]
The analysis execution data storage unit 16 and the high-speed morpheme analysis unit 17 are the same as those in the first embodiment, and thus the description of their functions is omitted.
[0059]
The accuracy determination means 18 newly provided in the second embodiment is based on the search result when the high speed morpheme analysis means 17 searches the analysis execution time data storage means 16 for the desired analysis execution time data. A character string that is considered to have low accuracy in the morpheme analysis result obtained from the morpheme analysis means 17 is determined. Such accuracy determination results are given to the accuracy / analysis result synthesis means 19.
[0060]
The analysis execution data is created from learning data as described in Non-Patent Document 1, Japanese Patent Application No. 9-68300 and the drawings, and as described in the first embodiment. The Analysis execution data corresponding to the character string appearing in the learning data exists, but naturally, analysis execution data corresponding to the character string not appearing in the learning data does not exist. Even in the analysis execution data corresponding to the character string appearing in the learning data, the value of the chain probability changes depending on the appearance frequency.
[0061]
Therefore, when the analysis execution data storage means 16 is accessed in order to analyze an unknown document for high-speed morpheme analysis, a portion where the corresponding character string does not exist or a portion having a low chain probability even if it exists, It can be said that the accuracy of a part is lower than the other parts. The accuracy determination unit 18 determines such a low-accuracy portion through access to the analysis execution data storage unit 16.
[0062]
The accuracy / analysis result synthesizing unit 19 is given the accuracy judgment result from the accuracy judging unit 18 and the morpheme analysis result from the high-speed morpheme analyzing unit 17. The accuracy / analysis result synthesizing unit 19 synthesizes these pieces of input information, clearly indicates the character string determined by the accuracy determining unit 18 to be insufficient in accuracy, and presents the morphological analysis result to the user.
[0063]
An example of the overall processing flow of the morpheme analyzer 10A of the second embodiment that can be functionalized as described above will be described in detail with reference to the flowchart of FIG.
[0064]
FIG. 9 shows processing for one sentence in the unknown document. Further, in the processing example of FIG. 9, the specification of the low-precision character string portion has one requirement that the corresponding character string does not exist in the analysis execution data storage unit 16. Further, description will be made assuming that the accuracy counter is built in the apparatus 10A (accuracy determination means 18). This accuracy counter is a temporary memory whose initial value is zero. Further, the description will be given on the assumption that the apparatus 10A (accuracy determination means 18) also incorporates a buffer memory for low-accuracy character strings.
[0065]
A character position pointer in the input sentence is provided, and a character string of N characters starting from the character position indicated by the pointer is read (step 300). Then, based on whether or not the character string could not be read by this reading process, it is determined whether or not the last character string has been read and the subsequent process has already been completed (step 301).
[0066]
If not completed, a search character string is created based on the read character string, the analysis execution time data storage means 16 is searched, and whether or not the search character string exists in the analysis execution time data storage means 16. Is determined (steps 302 and 303).
[0067]
Here, a plurality of search character strings are generally created. For example, since the analysis execution time data as shown in FIG. 11 is stored in the analysis execution data storage means 16, each character of the read character string of N (for example, 3) characters is expanded into two types. Since all combinations of the two types of extended characters for each character in the input character string are search character strings, there are generally only 2 N sets of search character strings to be created. If it is determined in step 303 that no search character string exists, “all” search character strings do not exist, and even when one set of search character strings out of 2 N sets of data is stored in the analysis execution data storage means. 16 is present in the determination of step 303.
[0068]
If the determination result in step 303 shows that the search character string does not exist, the value of the precision counter is incremented by 1, the current read character string is stored in the low-precision character string storage area, and the process proceeds to step 309 described later. (Steps 304 and 305).
[0069]
On the other hand, if the result of determination in step 303 is that the search character string exists in the analysis execution data storage unit 16, it is determined whether or not the value of the accuracy counter at that time is equal to or less than the threshold (step 306). ). The threshold is determined according to the value of N. For example, if N is 3, about 1 is appropriate.
[0070]
If an affirmative result is obtained, the value of the precision counter is cleared to 0, and the low-precision character string stored in the low-precision character string storage area is cleared, and the process proceeds to step 309 described later (step 307). ). On the other hand, if the search character string exists in the analysis execution time data storage means 16 and the value of the accuracy counter at that time is larger than the threshold, it is stored in the low-precision character string storage area at that time. The low-accuracy character string that has been recognized is recognized as a part to be specified by the morphological analysis result, and the process proceeds to step 309 described later (step 308).
[0071]
In step 309, a plurality of morpheme analysis results for the character string of the input sentence up to the character string read this time, based on the chain probability for one or more sets of search character strings existing in the analysis execution time data storage means 16 Update the candidate's evaluation value (product of chain probability). In addition, although the handling in the case where a search character string does not exist is arbitrary, what is necessary is just to employ | adopt the method of the existing high-speed morpheme analysis means as it is. For example, when the analysis execution time data stored in the analysis execution time data storage means 16 relates to an extended character string with N characters, it relates to an extended character string with N-1 characters or N-2 characters. Form and process analysis run-time data. In general, a character string that does not exist if the number of characters is long is often present when viewed for each portion having a shorter number of characters.
[0072]
When the processing in step 309 is completed, the character position pointer is incremented by 1, and the process returns to step 300 described above to read a character string of N characters in the input sentence.
[0073]
When the processing loop consisting of steps 300 to 309 is repeated, the character string of the number N of characters at the end of the sentence in the input sentence is read, and the subsequent processing from step 302 to step 309 is also completed. It is determined that the processing of the last character string is also completed.
[0074]
At this time, among the combinations in which each character in the input sentence is replaced with an extended character, the combination with the highest chain probability is taken as the morpheme analysis result, and this morpheme analysis result is clearly shown to the user as a low-precision character string. The process is terminated (step 310).
[0075]
The processing of the morphological analysis apparatus 10A of the second embodiment as described above will be specifically described on the assumption that the sentence “The configuration of the salary calculation system mirage is shown in FIG. 1” shown in FIG. 10 is input. The analysis execution data storage means 16 is assigned a value (chain probability) with three characters appearing continuously in the learning data as one unit, and the contents shown in FIG. 11 are stored. Shall. It is assumed that a character string not shown in FIG. 11 is not assigned a value. For the sake of simplicity, the description will be made assuming that there is no character string value of less than 3 characters in the sentence head, sentence end processing, and analysis execution time data. Further, description will be made assuming that the threshold value for the value of the accuracy counter is 1.
[0076]
The processing shown in FIG. 9 is started after the accuracy counter and the low-precision character string storage area are initialized.
[0077]
First, in step 300, the first three characters “salary meter” are read, and it is confirmed in step 301 that the reading is not completed. When the character string “salary meter” is searched in the analysis execution data storage unit 16 in step 302, Existence is confirmed (representing presence of a chain probability of 0.71), and the process proceeds to step 309 through steps 303, 306, and 307, and the evaluation value (score) of the extended character string candidate up to that character string is Calculated. Therefore, even when the process for the character string “payroller” is completed, the value of the precision counter is 0, and no character is stored in the low-precision character string storage area.
[0078]
The same path processing is executed for the character strings “given calculation”, “calculation system”, “calculation system”, “system”, and “stem”. Therefore, when the processing for the character string “stem” is completed, the value of the precision counter is 0, and no character is stored in the low-precision character string storage area.
[0079]
Next, when the character string “Tem 蜃” is read, since there is no corresponding analysis execution time data in the analysis execution time data storage means 16, the value of the accuracy counter is incremented by 1 in step 304 (thereby “1”). In step 305, “tem 蜃” is stored in the low-precision character string storage area, and then the process proceeds to step 309.
[0080]
Thereafter, the same processing is executed for the character strings “Miraku”, “Miraku”, “Kairou”, and “Circle”. As a result, when the processing for the character string “ro-no-gano” is completed, the character string “tem mirage” is stored in the low-precision character string storage area, and the value of the accuracy counter is “5”.
[0081]
Since the next character string “configuration” includes analysis execution time data corresponding to the analysis execution data storage unit 16, the process proceeds from step 303 to step 306. Since the accuracy counter value “5” at this time is larger than the threshold value “1”, the accuracy / analysis of the low-accuracy character string “Tem Mirage” stored in the low-accuracy character string storage area in step 308 The result is given to the result synthesizing means 19, and then the process proceeds to step 309. When the process for the character string “configuration” is completed, the character string “Tem Mirage's structure” is displayed in the low-precision character string storage area in the same manner as when the process for the previous character string “ro” is completed. The value stored in the accuracy counter is “5”.
[0082]
For the next character string “configuration” to the final character string “show”, the corresponding analysis execution time data of steps 303, 306, 307, and 309 exist in the analysis execution time data storage unit 16, respectively. The process in the general route is executed.
[0083]
When the process for the final character string “show” is completed, there is no character string next, so the process proceeds to step 310, and as illustrated in FIG. 12, the morphological analysis result “salary / calculation / system / mirage / configuration”. / Is shown to the user so that it can be easily compared with the low-precision character string “Tem Mirage” whose accuracy is not confident.
[0084]
According to the second embodiment, the partial character string determined to have poor high-speed morphological analysis accuracy is presented to the user, so that the user can A morpheme analyzer that can input a correct morpheme analysis result can be realized.
[0085]
Since the analysis result of the morphological analysis device is input to the next syntax analysis device or the like, the accuracy is important, and the degree of adverse effects when an incorrect analysis result is passed to the next device is large. Since it is determined by the user that the part is not clear whether it is correct or incorrect, the correct morphological analysis result can be input to the next apparatus.
[0086]
Here, since the accuracy determination is performed based on whether or not the data is stored in the analysis execution time data storage unit, the accuracy determination function hardly increases the processing time.
[0087]
(C) Third embodiment
Next, a third embodiment of the morphological analyzer according to the present invention will be described in detail with reference to the drawings.
[0088]
The morpheme analyzer of the third embodiment discriminates whether the accuracy of the high-speed morpheme analysis result is good or not, and automatically performs the low-speed morpheme analysis on the poor part, and always provides the high-precision morpheme analysis result. The main feature is that the output is made.
[0089]
FIG. 13 is a block diagram showing a functional configuration of the morphological analyzer 10B according to the third embodiment, which is the same as FIG. 1 according to the first embodiment and FIG. 8 according to the second embodiment. Corresponding parts are denoted by the same reference numerals.
[0090]
In FIG. 13, the morpheme analyzer 10B of the third embodiment includes a low-speed morpheme analysis unit 11, an analysis execution data storage unit 16, a high-speed morpheme analysis unit 17, an accuracy determination unit 18, and an analysis result synthesis unit 20.
[0091]
If the method for creating analysis execution data stored in the analysis execution data storage unit 16 is the same as that in the first embodiment, the illustration is omitted, but the low speed morpheme analysis unit 11 and the low speed morpheme analysis are omitted. The result storage means 12, the conversion means 13, the learning data storage means 14, and the learning means 15 are also provided (refer FIG. 1). A description of these components is omitted. In the case of the third embodiment, when the method of creating the analysis execution time data stored in the analysis execution time data storage means 16 is the same as that of the first embodiment, the low speed morpheme analysis means 11 This is used both for data creation processing and for morphological analysis of character strings including low-precision character strings, which will be described later.
[0092]
The functions of the low-speed morpheme analysis unit 11, the analysis execution time data storage unit 16, and the high-speed morpheme analysis unit 17 are the same as those in the first embodiment, and thus description of the functions is omitted. Furthermore, since the function itself of the accuracy determination means 18 is the same as that of the second embodiment, the description of the function is omitted.
[0093]
However, in the case of the third embodiment, the accuracy determination means 18 determines that the high-speed morpheme analysis method has no confidence in the accuracy, and the low-precision character string in the input sentence is given to the low-speed morpheme analysis means 11. ing. The low-speed morpheme analysis means 11 performs a low-speed morpheme analysis process on the character string portion including such a low-precision character string.
[0094]
The analysis result synthesizing means 20 newly provided in the third embodiment uses the low-speed morpheme analysis result obtained by the low-speed morpheme analysis means 11 for the portion corresponding to the low precision character string in the morpheme analysis result from the high-speed morpheme analysis means 17. It replaces with.
[0095]
FIG. 14 is a flowchart showing an example of the overall processing flow of the morphological analyzer 10B of the third embodiment. The same processing steps as those in FIG. 9 according to the second embodiment are denoted by the same reference numerals. Show.
[0096]
In the case of the second embodiment, the confirmed low-precision character string is recognized as an object to be presented to the user in step 308. In the case of this third embodiment, the confirmed low-precision character string is In step 308a, the low speed morpheme analyzing means 11 is provided.
[0097]
In the case of the second embodiment, the high-speed morpheme analysis result is presented to the user in the form of clearly specifying the low-precision character string in step 310. In the case of this third embodiment, the high-speed morpheme is shown. The part corresponding to the low-precision character string in the analysis result is replaced with the low-speed morpheme analysis result, and the replaced morpheme analysis result is presented to the user. The low-speed morpheme analysis is performed on the character string sandwiched between the morpheme break position before the first character of the low-precision character string and the morpheme break position after the last character of the low-precision character string in the high-speed morpheme analysis result. Executed.
[0098]
Except for the above two points, the other processes are the same as those in the second embodiment, and a description thereof will be omitted.
[0099]
Even when the sentence “The structure of the payroll system mirage is shown in FIG. 1” shown in FIG. 10 is input to the morpheme analyzer 10B of the third embodiment, the character string “Tem mirage” Is recognized as a low-precision character string, as in the second embodiment.
[0100]
Now, it is assumed that “mirage” is registered as one morpheme (noun) in the morpheme dictionary built in the low-speed morpheme analyzer 11. The low-speed morpheme analysis means 11 includes a low-precision character string “Tem Mirage's structure” and a high-speed morpheme analysis result “Pay / Calculation / System / Mirage / Configuration // Figure / 1 /// Show / Select /.”. Is given, the character string “system mirage of the system mirage” is sandwiched between the morpheme separation position before the first character of the low-precision character string “Tem Mirage” and the morpheme separation position after the last character of the low-precision character string. “Configuration” executes the analysis as a low-speed morphological analysis target part.
[0101]
Then, as shown in FIG. 15, the low-speed morpheme analyzing unit 11 outputs the analysis result with “system”, “mirage”, “no”, and “configuration” as separate morphemes. Since the corresponding part of the high-speed morpheme analysis result “salary / calculation / system / mirage / configuration //////////” is replaced with this low-speed morpheme analysis result, the final morpheme analysis result As shown in FIG. 16, “salary / calculation / system / mirage /// configuration /////////”.
[0102]
The analysis result in the third embodiment is analyzed with “mirage” and “no” as separate morphemes compared with the analysis result of the second embodiment, and the accuracy is improved.
[0103]
According to the third embodiment, a low-speed morpheme analysis is automatically performed on a partial character string determined to have poor high-speed morpheme analysis accuracy or its vicinity, and replaced with a low-speed morpheme analysis result. Therefore, it is possible to realize a morpheme analyzer that always outputs highly accurate morpheme analysis results.
[0104]
Also in the third embodiment, since the high-speed morpheme analysis is the basic analysis process, the analysis can be executed in a shorter time than the low-speed morpheme analysis of all input sentences.
[0105]
(D) Fourth embodiment
Next, a fourth embodiment of the morphological analyzer according to the present invention will be described in detail with reference to the drawings.
[0106]
The morpheme analyzer of the fourth embodiment discriminates whether the accuracy of the high-speed morpheme analysis result is good or not, and automatically executes the low-speed morpheme analysis on the bad part, and always provides the high-precision morpheme analysis result In addition to outputting, the low-speed morpheme analysis results are learned and reflected in the analysis execution data of the high-speed morpheme analysis. The main feature is that it has been made possible.
[0107]
FIG. 17 is a block diagram showing a functional configuration of the morphological analyzer 10C according to the fourth embodiment. The same or corresponding parts as those in FIGS. 1, 8, and 13 according to the above-described embodiments are shown in FIG. The same reference numerals are given.
[0108]
In FIG. 17, the morpheme analyzer 10C of the fourth embodiment includes a low-speed morpheme analyzer 11, a converter 13, a learning data storage unit 14, a learning unit 15, an analysis execution time data storage unit 16, a high-speed morpheme analysis unit 17, An accuracy determination unit 18 and an analysis result synthesis unit 20 are provided.
[0109]
All the constituent elements of the morphological analyzer 10C of the fourth embodiment have the same functions as the corresponding elements of the respective embodiments described above.
[0110]
However, in the morpheme analyzer 10C of the fourth embodiment, the result obtained by the low-speed morpheme analyzer 11 analyzing the character string including the low-precision character string is given to the converter 13. This is different from the first and third embodiments. The function of each means on the processing path from the conversion means 13 to the analysis execution time data storage means 16 is the same as that of the first embodiment.
[0111]
The low-speed morpheme analysis unit 11, the conversion unit 13, the learning data storage unit 14, the learning unit 15, and the analysis execution time data storage unit 16 perform the processing on the learning document input from the same as in the first embodiment. Of course, it may be the one that bears.
[0112]
FIG. 18 is a flowchart showing an example of the overall processing flow of the morpheme analyzer 10C of the fourth embodiment. The same processing steps as those in FIG. 14 according to the third embodiment are denoted by the same reference numerals. Show.
[0113]
In the morphological analyzer 10C of the fourth embodiment, the processing of steps 311 and 312 is provided after the final processing step 310a of the third embodiment.
[0114]
Step 311 is a process of converting the low-speed morpheme analysis result into data (learning data) for input to the learning means 15 for high-speed morpheme analysis and additionally storing it. Step 312 is a process of creating analysis execution data using all the learning data at that time.
[0115]
For example, when a low-speed morpheme analysis result as shown in FIG. 15 is obtained for a character string including a low-precision character string, the conversion means 13 learns data by the conversion method as shown in FIG. Is converted, an extended character string (learning data) as shown in FIG. 19 is obtained.
[0116]
When such learning data is added to the existing learning data and the learning unit 15 learns the entire learning data after the addition, the data is shown in FIG. 20 as analysis execution time data corresponding to the low-speed morpheme analysis result. Such data is obtained (other analysis execution time data is also obtained naturally) and stored in the analysis execution time data storage means 16. That is, in addition to data as shown in FIG. 11 (the chain probability changes), data as shown in FIG. 20 is newly added.
[0117]
As a result, when a sentence that can be analyzed based on the learned data is input as an analysis target, for example, when “Payroll system mirage is 2000 yen” is input, the previous low-precision character string. The accuracy determination means 18 does not determine that the portion is recognized as low accuracy, and a high-speed morpheme analysis result with the same degree of accuracy as in the third embodiment can be obtained without starting the low-speed morpheme analysis unit 11 each time. Become.
[0118]
According to the fourth embodiment, when the accuracy of the high-speed morpheme analysis is not good, the low-speed morpheme analysis is automatically executed, and the result is used as data for learning of the high-speed morpheme analysis. Can realize a morpheme analyzer that can accurately and rapidly analyze a sentence including the same morpheme as a sentence with poor accuracy.
[0119]
(E) Other embodiments
In each of the above-described embodiments, the analysis execution data indicates one category, but a plurality of categories such as by field may be prepared, and the category may be designated when an unknown document is input. In this case, in the first embodiment, when inputting a learning document, it is necessary to specify a category of the learning document. Further, in the third and fourth embodiments, if there is a specialized dictionary to which the low-speed morphological analysis means is applied, it belongs to that category. Furthermore, in the fourth embodiment, the low-speed morpheme analysis result is reflected in the analysis execution time data of the category specified when the unknown document is input.
[0120]
In the description of the first embodiment, it is not clearly shown whether the storage in the learning data storage means 14 is an additional storage or a new storage (a storage after clearing the previous one). good. Alternatively, the storage method may be instructed from the outside to the conversion means 13 each time.
[0121]
Furthermore, in the first and fourth embodiments, the learning means 15 may be configured as follows. Only for the learning data added to the learning data storage means 14, the frequency of appearance of the character string is counted to generate analysis execution data. In this case, not only the chain probability but also the appearance frequency is stored in the analysis execution time data storage unit 16, and from the total result of this time and the appearance frequency already stored in the analysis execution time data storage unit 16, The learning means 15 may determine the character string of the existing analysis-time learning data or the chain probability of a newly generated character string.
[0122]
Furthermore, in the second to fourth embodiments, the low-precision character string is identified as not being present in the analysis execution time data storage means 16, but even if it exists, its value ( The recognition condition for the low-precision character string may be that the linkage probability is smaller than a predetermined threshold.
[0123]
In the second to fourth embodiments, the range of the low-precision character string is not a partial character string of one sentence as described above, but all one sentence including the determination character string is handled as the low-precision character string. You may do it. In the second embodiment, information on whether or not the accuracy is low is attached to each sentence. If it is 3rd Embodiment, the whole sentence will be analyzed by the low speed morpheme analysis means 11 at the time of low precision recognition. In the fourth embodiment Ah Then, the low-speed morpheme analysis result of the entire sentence is reflected in the stored contents of the analysis execution data storage unit 16. When estimating accuracy for the entire sentence in this way, normalize the chain probability in the optimal high-speed morpheme analysis result by the number of characters in the input sentence, etc., and compare the value with a threshold value. You may make it guess.
[0124]
In addition, a method of judging accuracy without using the stored contents of the analysis execution time data storage means 16 is adopted alone, or a method of using the storage contents of the analysis execution time data storage means 16 to judge accuracy. You may do it. For example, as a method for determining the accuracy without using the stored contents of the analysis execution data storage means 16, for example, a predetermined central portion of a portion where one kind of character type such as hiragana or kanji is continuously connected for a predetermined number of characters or more. A method of determining that the accuracy of the character number portion is low can be given. Further, it may be determined that the accuracy of a sentence including a second number of Chinese characters of a predetermined number or more is low.
[0125]
Furthermore, in the third and fourth embodiments, the low-speed morphological analysis corresponding to the low-precision character string is executed for each sentence. However, after the high-speed morphological analysis of the entire document, the accuracy is poor. A low-speed morphological analysis may be performed on the part.
[0126]
Furthermore, in the specific description of each of the above embodiments, the extended information of the extended characters constituting the data at the time of analysis execution indicates only the morpheme segmentation information. Information may be included. In this case, as a matter of course, the conversion means and the learning means also correspond to them. When analysis execution data is set in this way, it is possible to speed up morpheme analysis that performs word division and part-of-speech assignment and morpheme analysis that determines the pronunciation of a word.
[0127]
Further, in each of the above embodiments, the morpheme analyzer in which the target natural language is Japanese is shown, but the present invention can also be applied to morpheme analyzers of other languages.
[0128]
【The invention's effect】
As described above, according to the morpheme analyzer of the present invention, it is possible to expect improvement in accuracy of morpheme analysis results and reduction in analysis processing time without imposing a burden on the user.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a first embodiment.
FIG. 2 is a flowchart illustrating an overview of processing according to the first embodiment;
FIG. 3 is an explanatory diagram illustrating an example of a learning document according to the first embodiment.
FIG. 4 is an explanatory diagram showing a low-speed morpheme analysis result for the first sentence in FIG. 3;
FIG. 5 is an explanatory diagram showing learning data corresponding to the learning document in FIG. 3;
FIG. 6 is a flowchart illustrating an example of detailed processing by a conversion unit according to the first embodiment.
7 is an explanatory diagram of processing in step 206 in FIG. 6;
FIG. 8 is a block diagram showing a configuration of a second embodiment.
FIG. 9 is a flowchart showing processing of the second embodiment.
FIG. 10 is an explanatory diagram showing a morphological analysis target sentence.
FIG. 11 is an explanatory diagram showing an example of stored contents of an analysis execution data storage unit 16;
12 is an explanatory diagram showing an example of output contents obtained by analyzing the sentence of FIG. 10 in the second embodiment.
FIG. 13 is a block diagram illustrating a configuration of a third embodiment.
FIG. 14 is a flowchart showing processing of the third embodiment.
FIG. 15 is an explanatory diagram illustrating a low-speed morpheme analysis result example for a low-precision character string according to the third embodiment;
FIG. 16 is an explanatory diagram illustrating a final analysis result example obtained by analyzing the sentence of FIG. 10 in the third embodiment.
FIG. 17 is a block diagram showing a configuration of a fourth embodiment.
FIG. 18 is a flowchart showing processing of the fourth embodiment.
FIG. 19 is an explanatory diagram illustrating an example in which a low-speed morpheme analysis result for a low-precision character string in the fourth embodiment is converted into learning data.
FIG. 20 is an explanatory diagram illustrating an example of analysis execution data formed from learning data corresponding to a low-precision character string according to the fourth embodiment.
[Explanation of symbols]
10, 10A, 10B, 10C ... morphological analyzer,
11 ... Low-speed morphological analysis means,
13: Conversion means,
15. Learning means,
17 ... High-speed morphological analysis means,
18 ... accuracy judging means,
19: Accuracy / analysis result synthesis means,
20: Analysis result synthesis means.

Claims (7)

The partial character string consisting of the first predetermined number of characters appearing in the natural language sentence , the chain probability that is the appearance ratio of the partial character string in the existing document, and the character assigned to each character of the partial character string are morphemes. Analysis execution time data storage means for storing a large number of analysis execution time data, which is set data including at least the position information indicating whether the character is the last character, and the analysis execution time data storage means for unknown sentences A morpheme analyzer having first morpheme analysis means for executing morpheme analysis with reference to the stored content of
When the unknown text smaller than the first linkage probability in the analysis run time data is the threshold value in the morphological analysis means is stored in said analysis execution time data storage means applied against, or, the analysis execution When the unknown sentence has a portion where the analysis execution time data stored in the time data storage means cannot be applied , the whole unknown sentence or the chain probability from the first morphological analysis means is a threshold value. There is an accuracy determination means for estimating the accuracy of the morphological analysis result of a character string portion in an unknown sentence including a smaller partial character string of the analysis execution data or a portion to which the analysis execution data cannot be applied. A morphological analyzer characterized by the above. The partial character string consisting of the first predetermined number of characters appearing in the natural language sentence , the chain probability that is the appearance ratio of the partial character string in the existing document, and the character assigned to each character of the partial character string are morphemes. Analysis execution time data storage means for storing a large number of analysis execution time data, which is set data including at least the position information indicating whether the character is the last character, and the analysis execution time data storage means for unknown sentences A morpheme analyzer having first morpheme analysis means for executing morpheme analysis with reference to the stored content of
When the character of a specific character type is connected by a second predetermined number of characters or more in the whole or part of the unknown sentence, the whole unknown sentence or the character of the specific character type from the first morpheme analyzing means is 2. A morpheme analysis apparatus comprising: an accuracy determination unit that estimates that the accuracy of a morpheme analysis result for a character string portion in an unknown sentence including a portion connected to a predetermined number of 2 or more is low accuracy. The partial character string consisting of the first predetermined number of characters appearing in the natural language sentence , the chain probability that is the appearance ratio of the partial character string in the existing document, and the character assigned to each character of the partial character string are morphemes. Analysis execution time data storage means for storing a large number of analysis execution time data, which is set data including at least the position information indicating whether the character is the last character, and the analysis execution time data storage means for unknown sentences A morpheme analyzer having first morpheme analysis means for executing morpheme analysis with reference to the stored content of
When the character of the specific character type is included in the whole or part of the unknown sentence by the third predetermined number of characters , the whole unknown sentence or the character of the specific character type from the first morpheme analyzing means is A morpheme analyzer comprising: an accuracy determining unit that estimates that the accuracy of a morpheme analysis result for a character string portion in an unknown sentence including a portion including a third predetermined number of characters or more is low accuracy. Characterized in that the accuracy determination means includes an analysis result output means for clearly indicating an unknown sentence or a character string portion in the unknown sentence that is estimated to be low in accuracy and outputting a morpheme analysis result from the first morpheme analysis means. The morpheme analyzer according to any one of claims 1 to 3. The second morpheme analysis unit that performs morpheme analysis using a morpheme dictionary with respect to an unknown sentence or a character string portion in the unknown sentence that is estimated to be low in accuracy by the accuracy determination unit. 4. The morphological analyzer according to any one of 4 above. In the morpheme analysis result of the first morpheme analysis unit, the result corresponding to the unknown sentence or the character string portion in the unknown sentence that the accuracy determination unit estimated to be low in accuracy is used as the morpheme analysis of the second morpheme analysis unit. 6. The morpheme analyzer according to claim 5, further comprising an analysis result synthesizing unit that replaces the result. Learning data consisting of a set of each character and delimiter position information indicating whether or not the character is the final character of the morpheme is recognized from the morpheme analysis result of the second morpheme analysis means. Conversion means to create,
The training data was tabulated cut for each of the first predetermined number, to create a partial character string consisting of a first predetermined number, and a chain probability, multiple analysis runtime data that at least includes a delimiter position information learned The morpheme analyzer according to claim 5 or 6, further comprising: means.

Images