JP2004094434A - Language processing method, its program, and its device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To highly precisely achieve tagging in an unknown document by learning the relationship between the features of a target word and peripheral words including words before and after the target word and a correct tag. <P>SOLUTION: A learning processing step generates a plurality of learning results by using the group of a pair of: the features of a target word and peripheral words at the both sides included in tagged correct document data; and a correct tag. A decision processing step repeats the decision of the predicted tag of each target word from the features of the target word and the peripheral words at the both sides included in unknown document data to be tagged to decide the correct tag by using the plurality of learning results learned by using the features of the target word and the peripheral words. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a language processing method, a program, and a device for automatically tagging an unknown document by using the learning result of a correct document with a tag, and in particular, using features of a target word and peripheral words in the correct document. The present invention relates to a language processing method, a program and an apparatus for tagging an unknown document based on a learning result.
[0002]
[Prior art]
In recent years, research and development of linguistic processing to add a certain index tag to words included in documents has been extensively performed, such as machine translation using tagged documents, information retrieval, question answering, knowledge discovery, etc. It is expected that a system using language processing will be put to practical use.
[0003]
Conventionally, various types of methods have been proposed for learning rules (learning results) from tagged correct answer document data and automatically adding tags to unknown documents using the learned rules. For example, the following are mentioned.
[0004]
(1) D. M. Bikel, S.M. Miller, R.A. Schwartz and R.S. Weishedel. (1997) Nymble: A High-Performance Learning Name-Finder. In5th Conference on Applied Natural Lan-guage Processing. Here, proper nouns are treated as a tagging problem, and tagging is performed by learning a hidden Markov model.
[0005]
(2) A. Borthwick, J .; Sterling, E.A. Agic-htein and R.A. Gishman. (1998) Exploiti-ng Diversity Knowledge Sources Via Maxim-um Entropy in Named Entity Recognition. In Processeds of the 6th Works on Very Large Corporation. Here, the same problem as (1) is treated as learning by the Maximum Entropy method which features the outputs of a plurality of systems.
[0006]
(3) M.P. Collins and Y. Singer. (1999)
Unsupervised Models for Named Entity
Classification. In Processeds of the
Joint SIGDAT Conference on Empirical
Methods in Natural Language Processing
and Very Large Corporation. Here, a method of improving accuracy using raw data without a tag is adopted.
[0007]
These methods use features such as notation of a target word and surrounding words to determine a tag, but do not use how surrounding words are tagged.
[0008]
Tags attached to surrounding words are important clues for tags to be attached to the target word. For example, when tagging the word "Yamaguchi", "Yamaguchi" may take a tag for a person's name or a tag for a place name. If it is known that the word following “Yamaguchi” is “Taro” or the like and a tag of a personal name is given, “Yamaguchi” is considered to be a personal name. On the other hand, there is a word string such as "Yamaguchi" before "Yamaguchi", and if it is found that these are tagged with a place name, it is a clue that a place name tag is attached to "Yamaguchi".
[0009]
[Problems to be solved by the invention]
However, the tags attached to the surrounding words are important clues for determining the tag of the target word, but cannot be used for an unknown document because the surrounding tags cannot usually be known.
[0010]
The following is known as one of the methods for solving this problem.
(4) Hiroyasu Yamada, Taku Kudo, Yuji Matsumoto. (2001) "Extracting Japanese Named Expressions Using Support Vector Machines", Information Processing Society of Japan, Natural Language Processing Research Group 142-17.
[0011]
In this Japanese named entity extraction method, a method called a dynamic feature is used in order to use surrounding tags as features for determining a target word. In the method using the dynamic feature, at the time of learning, a tag of the correct answer data on the left side (or right side) of the target word is used as a surrounding tag. For an unknown document, the peripheral tag is not used for the leftmost (or rightmost) word, and for other words, the result determined by the learning device is used as the peripheral tag.
[0012]
Although the method using this dynamic feature is known to be effective, it is still insufficient in accuracy. There are two causes, both of which are related to how to handle surrounding tags.
[0013]
One is that only one tag can be used as a peripheral tag. The other is that, while learning, the correct data is used as a peripheral tag, but at the time of determination, prediction data that is unknown or not is used as a peripheral tag, and thus it is inconsistent. .
[0014]
An object of the present invention is to provide a language processing method, a program, and a device for learning the relationship between a feature of a target word, peripheral words including surrounding words, and correct tags and realizing tagging in an unknown document with high accuracy.
[0015]
[Means for Solving the Problems]
FIG. 1 is a diagram illustrating the principle of the present invention. In the language processing method according to the present invention, the learning processing unit 1 generates a plurality of learning results using a set of pairs of a target word included in a correct answer document data with a tag and features of surrounding words on both sides thereof and a correct answer tag. A learning processing step, and using the plurality of learning results, determine the tag of each target word from the characteristics of the target word included in the unknown document data to be tagged and the surrounding words on both sides of the target word. And a determination processing step.
[0016]
This learning processing step
An input step of inputting the correct answer document data with a tag from the outside by the input device;
A data converter for converting the correct document data into learning data in a format handled by the learning device,
A first learning step of generating a first learning result 24-1 for determining a prediction tag from a feature using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data by a learner; ,
A first determining step of determining a prediction tag of each target word from characteristics of the target word and peripheral words included in the learning data by using the first learning result 24-1 generated in the first learning step by the determiner; ,
A data reconstructing step of reconstructing the prediction tag determined in the first determining step into a part of the features of the learning data by the data reconstructing device;
A learning unit that determines a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of the target word and the surrounding word including the prediction tag included in the learning data reconstructed in the data reconstructing step; A second learning step of generating a second learning result 24-2;
Using the second learning result 24-2 generated in the second learning step, the determiner determines the tag of each target word from the characteristics of the target word including the prediction tag and the surrounding words included in the learning data. Two determination steps;
A learning repetition step of generating a plurality of learning results 24-3 to 24-N after the second learning result by repeating the data reconstructing step, the second learning step, and the second determination step a plurality of times;
It is characterized by having.
[0017]
The determination processing step using the learning result generated in such a learning processing step includes:
An input step of inputting unknown document data to be tagged from the outside by the input device;
A data conversion step of converting unknown document data into test data in a format handled by the learning device by a data converter;
A third determining step of determining a prediction tag of each target word from characteristics of the target word and the peripheral words included in the test data by using the first learning result 24-1 by the determiner;
A second data reconstructing step of reconstructing the prediction tag determined in the third determining step into a part of the features of the learning data by the data reconstructing device;
Using the second learning result 24-2, the determiner determines the tag of each target word from the characteristics of the target word including the predicted tag and the surrounding words included in the test data reconstructed in the second data reconstruction step. A fourth determination step of determining
A determination repetition step of determining the tag of the unknown document data by repeating the data reconstruction step and the fourth determination step a plurality of times while switching to the learning results 24-3 to 24-N after the second learning result;
An output step of inverting the unknown document data into document data with the determined tag by the output device and outputting the document data;
It is characterized by having.
[0018]
According to such a language processing method of the present invention, the peripheral prediction tag is not known at all for the first time, but it becomes a useful feature if the peripheral tag is correctly answered after the second time. For this reason, if the accuracy of the prediction tag is improved due to the correct answer of the peripheral tag, the determination can be performed with the peripheral tag having higher accuracy as a feature, and the tagging accuracy is improved.
[0019]
According to another aspect of the present invention, tagging of an unknown document is simply performed by generating two learning results from a correct document, that is, a learning result that does not use the characteristics of peripheral words, and a learning result that uses the characteristics of peripheral words. .
[0020]
In this case, the learning processing step
An input step of inputting the correct answer document data with a tag from the outside by the input device;
A data conversion step of converting the correct answer document data into learning data in a format not including a prediction tag handled by the learning device by a data converter;
A first learning step of generating, by a learning device, a first learning result for determining a prediction tag from a feature using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data;
A data conversion step of converting the correct answer document data into learning data in a format handled by a learning device including a prediction tag by a data converter;
The learning device generates a second learning result for determining a prediction tag from the feature by using a set of pairs of a feature and a correct answer tag of a peripheral word including a target word and a correct answer tag included in the learning data as a prediction tag. 2 learning steps,
It is characterized by having.
[0021]
For tagging an unknown document using the first learning result and the second learning result generated in the learning processing step, the determination processing step includes:
An input step of inputting unknown document data to be tagged from the outside by the input device;
A data conversion step of converting unknown document data into test data in a format handled by the learning device by a data converter;
A first determining step of determining a prediction tag of each target word from characteristics of the target word and surrounding words included in the test data by using the first learning result by the determiner;
A data reconstructing step of reconstructing the prediction tag determined in the first determining step into a part of the features of the learning data by the data reconstructing device;
A second determining step of determining a tag of each target word from characteristics of a target word including a prediction tag included in the test data reconstructed in the data reconstructing step and peripheral words using the second learning result by the determiner; When,
A determination repetition step of determining a tag of unknown document data by repeating the data reconstruction step and the second determination step a plurality of times using the second learning result;
An output step of inverting the unknown document data into document data with the determined tag by the output device and outputting the document data;
It is characterized by having.
[0022]
In this simple language processing method, the surrounding prediction tags are not known at all for the first time, so they are not used, and the learning result that does not use the prediction tags is used for determination. For the second and subsequent times, a learning result is used assuming that the peripheral tags determined last time are correct. As a result, if the accuracy of the prediction tag is improved by the correct answer of the peripheral tag, a determination with a higher accuracy of the peripheral tag can be performed, so that the tagging accuracy is improved.
[0023]
Here, the learning processing step and the determination processing step include a morphological analysis step of analyzing the correct document data or unknown document data received by the input device to determine a morpheme. In this case, the data conversion step converts the correct document data or the unknown document data including the analyzed morpheme into learning data in a format handled by the learning device.
[0024]
Such morphological analysis is suitable for tagging a language in which the processing language is not separated by a space for each word, such as Japanese. Therefore, a language in which words are separated by spaces, such as English, does not require morphological analysis.
[0025]
The learning processing step and the determination processing step include an external tagging step of passing the correct document data or unknown document data received by the input device to an external tagging system and acquiring a tagging result. In this case, in the data conversion step, the data converter converts the correct answer document data or the unknown document data including the tagging result obtained from the tagging system into learning data in a format handled by the learning device.
[0026]
The present invention provides a program for language processing. The program includes, in a computer, a learning processing step of generating a plurality of learning results by using a set of pairs of a feature word and a correct answer tag included in a target word included in the correct answer document data with tags and surrounding words on both sides thereof. A determination processing step of determining the tag of each target word from the characteristics of the target word included in the unknown document data to be tagged and the surrounding words on both sides thereof using the learning result of.
[0027]
The present invention provides a language processing apparatus, and includes a learning process for generating a plurality of learning results using a set of pairs of features of a target word and surrounding words included in a correct answer document data with a tag and correct answer tags. And a determination processing unit configured to determine a tag of each target word from characteristics of a target word and peripheral words included in unknown document data to be tagged by using a plurality of learning results by the determination processing device. It is characterized by the following.
[0028]
The details of the language processing program and the language processing device are basically the same as those of the language processing method.
[0029]
BEST MODE FOR CARRYING OUT THE INVENTION
FIG. 2 is a block diagram showing an embodiment of the language processing device according to the present invention. The language processing device in FIG. 2 has a function as a learning processing unit and a function as a determination processing unit. That is, the function as the learning processing unit generates a plurality of learning results by using a set of pairs of the feature of the target word and the surrounding words and the correct tag included in the correct document data with the tag. On the other hand, the function as the judgment processing unit uses the plurality of learning results obtained by the learning processing unit to determine the tag of each target word from the characteristics of the target word and surrounding words included in the unknown document data to be tagged. judge.
[0030]
In FIG. 2, the functional configuration as a learning processing unit in the language processing apparatus of the present invention for Japanese documents is shown by blocks. The language processing device functioning as the learning processing unit includes an input / output device 10, a morphological analysis device 11, a data converter 12, a learning device 14, a learning control device 16, a judgment device 18, a judgment control device 20, a learning result database 22, It is composed of a data reconstruction device 26.
[0031]
On the other hand, the language processing device functioning as the determination processing unit has a device configuration excluding the learning device 14 and the learning control device 16, as shown in FIG.
[0032]
The language processing apparatus according to the present invention as shown in FIG. 2 is realized by, for example, a hardware environment of a computer shown in FIG. In the computer shown in FIG. 3, a RAM 402, a hard disk controller (software) 404, a floppy disk driver (software) 410, a CD-ROM driver (software) 414, a mouse controller 418, a keyboard controller 422, and a display controller 426 are provided on a bus 401 of a CPU 400. , A communication board 430 is connected.
[0033]
The hard disk controller 404 connects the hard disk drive 406 and loads an application program for executing the language processing of the present invention. When the computer is started, the hard disk controller 404 calls a necessary program from the hard disk drive 406, develops the program on the RAM 402, and Execute.
[0034]
A floppy disk drive (hardware) 412 is connected to the floppy disk driver 410, and can perform reading and writing on the floppy disk (R). A CD drive (hardware) 416 is connected to the CD-ROM driver 414, and can read data and programs stored in the CD.
[0035]
The mouse controller 418 transmits an input operation of the mouse 420 to the CPU 400. The keyboard controller 422 transmits an input operation of the keyboard 424 to the CPU 400. The display controller 426 performs display on the display unit 428. The communication board 430 uses a communication line 432 including wireless communication to communicate with another computer or server via a network such as the Internet.
[0036]
Here, the principle of the learning processing unit in the language processing apparatus of the present invention having the configuration shown in FIG. 2 will be described. Consider a method of automatically extracting proper nouns in a Japanese document as an example. As an example of the target Japanese document data, a document "Mr. XX, chairman of the Japan Broadcasting Corporation at the ordinary parliament held the day before ..." is taken as an example.
[0037]
When this document data is decomposed into words by the morphological analysis device 11 in FIG. 2, the document data becomes, for example, document data 28 in FIG. The document data 28 includes, as proper nouns, “Normal Diet” as the proper name 28-1, “Japan Broadcasting Corporation” as the organization name 28-2, and “XX” as the personal name 28-3. I have.
[0038]
Extracting proper nouns in the text of the document data 28 in FIG. 4 can be defined as a tagging problem as in FIG. FIG. 5 shows the correct answer document data 30 in which the correct answer tag 34 is added to the word 32. In the present invention, in order to automatically add a tag to a word of unknown document data, as shown in FIG. A learning processing unit is performed by using this.
[0039]
The learning process of the present invention generates a learning result (learning rule) from the correct answer document data 30 in which the feature of the word 32 and the correct answer tag 34 are associated. The generated learning result can predict the tag from the characteristics of the word.
[0040]
For example, when looking at the word "broadcast", the correct answer document data 30 uses itself and the words before and after the word and the number of characters as features, and designates "I-ORG" as the correct answer tag. In the learning process, a learning result that can determine a tag from a feature is generated by using a set of pairs of a feature and a correct answer tag of the target word and surrounding words in the correct answer document data 30.
[0041]
Here, if the tag of the peripheral word with respect to the target word can be used as one of the features, it is expected that it is effective for predicting the tag of the target word as the correct answer data. This is due to the fact that tags representing proper nouns are likely to appear continuously. For example, taking the unknown document data 36 of FIG. 6 as an example, when the correct tag 42 is known for the word 38, the prediction can be effectively performed by using the characteristics of the surrounding tags for the target tag as the prediction tag 40.
[0042]
For example, if the tags of “Japan”, “broadcast”, and “association” in the word 38 are all “I-ORG”, but it is known that the tag of “association” is “I-ORG”, This is an important clue that the tag of “Japan” or “broadcast” located before that is “I-ORG”.
[0043]
However, the processing goal of the language processing apparatus of the present invention is to predict an unknown tag. Therefore, the correct tag 42 is unknown in the unknown document data 36 of FIG. I do not understand. Therefore, in the language processing method of the present invention, a tag assigned by a learning process based on the correct answer document data is used as a peripheral tag.
[0044]
7 and 8 are explanatory diagrams showing the procedure of the learning process executed by the configuration of the learning processing unit of the language processing apparatus of the present invention in FIG. The learning process in FIGS. 7 and 8 will be described below with reference to the device configuration in FIG.
[0045]
In FIG. 7, first, the correct answer document data 30 is input by the function of the input / output device 10 as an input unit. The correct answer document data 30 has, for example, the contents shown in FIG. Next, data conversion 44 is performed by the data converter 12. In this data conversion, the correct answer document data 30 is converted into learning data 46 in a format that can be handled by the learning device 14.
[0046]
The learning data 46 includes words 48, features 50, prediction tags 52, and correct answer tags 54. In this embodiment, the feature 50 includes four items: “notation”, “word length”, “character type”, and “part of speech”. The prediction tag 52 is "?" In the initial state.
[0047]
Next, the learning unit 14 controlled by the learning control device 16 uses the set of pairs of the feature and the correct answer tag of the target word and the surrounding words included in the learning data 46 to determine a prediction tag from the feature ( (First learning result: also referred to as classification rule) 24-1.
[0048]
For example, if the target word is “broadcast”, the learning process 60 includes, for the surrounding words “Japan” and “association” before and after the target word, the respective features 50 and the tag 58 of the target word “broadcast”, “I- A pair of “ORG” is set in the learning device 14, and a rule for obtaining the correct answer tag “I-ORG” based on the input of the feature data 56 is generated as a learning result.
[0049]
Such learning processing is performed on each of the second to fifth broadcasts, associations, chairpersons, and nos except for the first and last words "Japan" and "XX" in the learning data 46. The learning process is repeatedly performed using all pairs of the features of the surrounding words and the correct tags 54 as words, and a learning result 24-1 is generated. The generated learning result 24-1 is stored in the learning result database 22.
[0050]
Next, the determination unit 18 controlled by the determination control device 20 uses the learning result 24-1 generated in the learning process 60 to determine the characteristics of the target word and the surrounding words included in the learning data 46 and to determine the characteristics of each target word. The prediction tag 52 is determined.
[0051]
In the determination process 62 of FIG. 7, the second “broadcast” of the learning data 46 is set as a target word, and the characteristic data 56 including surrounding words “Japan” and “association” before and after the target word are input to the determiner 18. The determination data 64 is obtained by making a determination using 24-1. In this case, the prediction tag of the target word “broadcast” is “O”. The determination process 62 for obtaining the prediction tag of each target word in the determination data 64 is performed for all target words in the learning data 46 except for the first and last words. can get.
[0052]
Then, the data reconstruction 66 of FIG. This data reconstruction 66 generates the reconstruction learning data 68 by storing the prediction tag determined in the determination processing 62 of FIG. 7 as update information in the prediction tag 52 of the learning data.
[0053]
Next, a learning result (second learning) for determining a prediction tag from a feature using the set of pairs of the feature and the correct answer tag 54 of the target word and the surrounding words including the prediction tag 52 in the reconstructed learning data 68 by the learning device 14. Result) 24-2 is generated.
[0054]
For example, if the target word “broadcast” is taken as an example, the feature data 70 in the surrounding words “Japan” and “association” before and after the target word “broadcast” include the feature including the updated prediction tag obtained in the determination process 62 in FIG. The learning process 72 is performed using “I-ORG”, which is the correct answer tag 58 of the target word “broadcast”. This learning process 72 generates a learning result 24-2 as a second learning result by repeating similarly for each of the second to fifth target words except the first and last words in the reconstructed learning data 68. .
[0055]
Next, using the learning result 24-2 as the second learning result generated in the learning process 72 by the determiner 18, each of the features of the target word including the prediction tag and the surrounding words included in the reconstructed learning data 68 is used. The tag of the target word is determined. In this example, a pair of the feature data 70 of the target word “broadcast” in the reconstructed learning data 68 and its correct tag 58 “I-ORG” is input to the determiner 18 as the determination data 76, and the learning result 24 is obtained. 3 shows judgment data 76 of the prediction tag according to −2. This determination processing 74 is performed in the same manner for all of the second to fifth words except the first and last words in the reconfiguration learning data 68.
[0056]
When the determination process 74 is completed, it is determined in a determination step 78 whether or not the value of the number of learnings i is smaller than a predetermined number of learnings. If the value is smaller, the process returns to the data reconstruction 66 again, and the determination data 76 generated in the determination process 74 is determined. Of the reconstructed learning data 68 is included in the predicted tag 52, and a third learning process 72 is performed on the updated reconstructed learning data 68 to generate a learning result 24-3. I do. Then, in the determination process 74, the prediction tag of the determination data 76 is determined from the reconstructed learning data 68 using the learning result 24-3.
[0057]
The data reconstruction using the prediction tags, the learning process based on the reconstructed learning data, and the determination process based on the learning result obtained in the learning process are repeated until the number i of learning reaches N times. , The learning results 24-1 to 24-N are stored, and it is determined in the determination step 78 that the number of times of learning reaches N times, and the learning process ends.
[0058]
FIG. 9 is a flowchart of the learning processing by the language processing apparatus of FIG. 2 shown in FIGS. 7 and 8, and this flowchart shows the processing contents of the learning processing program according to the present invention.
[0059]
In FIG. 9, in the learning process, the correct document data is input by the input / output device 10 in step S1, and the data is converted into a set of features and tags by a data converter in step S2. Generate data.
[0060]
Next, in step S3, the learning device 14 generates a first learning result by using a set of pairs of the feature of the target word and the peripheral word and the correct answer tag in the learning data. Subsequently, in step S4, the tag of each word is predicted from the characteristics of the target word and the peripheral words in the correct answer data using the first learning result by the determiner 14. The processing of steps S1 to S4 corresponds to FIG.
[0061]
Subsequently, in step S5, the data reconstruction device 26 replaces the prediction tag acquired in step S4 with the prediction tag of the learning data to generate new learning data, that is, reconstructed learning data. Next, in step S6, the learning unit 14 generates an i-th learning result using a set of pairs of the feature of the target word including the prediction tag and the surrounding words and the correct answer tag in the i-th learning data. In this case, i = 2.
[0062]
Subsequently, in step S7, it is checked whether or not i is smaller than a predetermined number N. If it is smaller, i is increased by one in step S8, and the processing from step S5 is repeated until i becomes N in step S7. . The processing of steps S5 to S8 corresponds to FIG.
[0063]
FIG. 10 shows how the language processing apparatus of the present invention automatically tags unknown document data using the learning results 24-1 to 24-N of the learning result database 22 generated by the learning processing unit of FIG. FIG. 3 is a block diagram illustrating a configuration of a determination processing unit. The configuration as the determination processing unit is realized by a configuration excluding the functions of the learning control unit 16 and the learning device 14 in FIG.
[0064]
FIG. 11 and FIG. 12 are explanatory diagrams of the processing procedure by the determination processing unit of FIG. 10, and the following description will be made according to the configuration of FIG.
[0065]
In FIG. 11, first, unknown document data 80 is input using the function of the input unit of the input / output device 10. The unknown document data 80 is, for example, the document data 28 as shown in FIG. Next, data conversion by the data converter 12 is performed.
[0066]
The data conversion 82 converts the unknown document data 80 into test data 84 in a format handled by the learning device 14. The test data 84 includes a word 86, a feature 88, and a prediction tag 90, and the prediction tag 90 is "?" In an initial state.
[0067]
Next, using the first learning result 24-1 of the learning result database 22 by the determiner 18, the prediction tag of each target word is determined from the characteristics of the target word and the peripheral words included in the test data 84. Specifically, for the second to fifth target words excluding the first and last words in the test data 84, for example, if the second target word “broadcast” is taken as an example, the surrounding words “Japan”, “ The characteristic data 92 including "association" is input to the determiner 18, and the learning result 24-1 is applied to the characteristic data 92 to determine the prediction tag shown in the determination data 96. In this case, the prediction tag of the target word "broadcast" is It is "O".
[0068]
Next, proceeding to FIG. 12, the prediction tag 90 is updated by the data reconstructing device 26 so that the prediction tag determined in the determination process 94 of FIG. Generate.
[0069]
Subsequently, the second learning result 24-2 is extracted from the learning result database 22 by the determiner 18, and the tag of each target word is determined from the characteristics of the target word including the prediction tag included in the reconstructed test data 100 and the peripheral words. I do. In the determination processing 104 in this case, the target word “broadcast” in the reconstructed test data 100 is taken as an example, and the feature data 102 including surrounding words before and after the target word is input to the determiner 18 to determine the prediction tag. This shows a state in which the determination data 106 has been obtained.
[0070]
When the determination of the prediction tag based on the reconfiguration test data 100 is completed in the determination process 104, it is checked in a determination step 108 whether or not the number of determinations i is less than a predetermined number N. After performing the update including the prediction tag determined in the process 104 in the prediction tag 90 of the reconstructed test data 100, the determination process 104 is performed on the reconstructed test data 100 using the third learning result 24-3. The prediction tag is determined as the determination data 106.
[0071]
The determination process based on the learning result corresponding to the data reconstruction and the number of determinations is repeated N times, and a series of the determination processes ends. Since the prediction tag 90 in the reconstructed test data 100 is finally determined as a correct answer tag by this determination processing, the data converter 12 performs an inverse conversion of attaching the determined determination tag to each word of the input unknown document data. Thus, the tagged document data is output from the input / output device 10.
[0072]
FIG. 13 shows unknown document data 80 based on the determination data obtained for the target word “broadcast” in the determination process 94 of FIG. 11. In the first determination stage, the prediction tag 88 differs from the correct tag 112. However, the prediction tag 88 of the target word “broadcast” is updated to “I-ORG” of the correct answer tag 112 by repeating the determination process using the second and subsequent learning results 24-2 to 24-N shown in FIG. The tag that matches the correct answer tag can be determined.
[0073]
FIG. 14 is a flowchart of the determination process in FIGS. 11 and 12. This flowchart is a flowchart of the tag determination using the learning results 24-1 to 24-N obtained by the learning processing program according to the learning process flowchart of FIG. 9 shows a judgment processing program.
[0074]
In FIG. 14, first, unknown document data is input by the input / output device 10 in step S1, then converted into a set of features and tags by a data converter in step S2, the prediction tag is set to "?" Generate as test data. Next, in step S3, the tag of each word is predicted from the characteristics of the target word and the surrounding words in the test data by the determiner 18 using the first learning result. The processing of steps S1 to S3 corresponds to FIG.
[0075]
Next, in step S4, the data reconstruction device 26 replaces the prediction tag acquired by the determiner 18 with the prediction tag of the test data, and generates new test data, that is, reconstructed test data. Subsequently, in step S5, using the i-th learning result, in this case, the second learning result, by the determiner 18, the tag of each word is extracted from the features of the target word and the surrounding words including the prediction tag in the reconstructed test data. Predict.
[0076]
If the number of processes i is less than the predetermined value N in step S6, i is incremented by one in step S7, and then the processes from step S4 are repeated. When the determination of the tag based on the N-th learning result is completed, since i is N in step S6, a series of determination processing ends. Then, the data converter 12 performs an inverse conversion with the determined tag attached to the unknown document, and outputs the processed tagged document by the function of the input / output device 10 as an output unit. The processing of steps S4 to S7 corresponds to FIG.
[0077]
FIG. 15 shows another embodiment of the learning processing unit in the language processing apparatus according to the present invention. In this embodiment, English document data is targeted as the correct answer document data. In the case of an English document, since it is not necessary to perform morphological analysis to separate words as in Japanese, the morphological analyzer 11 provided in the embodiment of FIG. 2 is omitted, and the other configuration is the same. .
[0078]
FIG. 16 and FIG. 17 are explanatory diagrams of the learning processing performed by the learning processing unit of FIG. 15 on the correct English document data.
[0079]
In FIG. 16, the correct answer document data 114 in English is input by a function as an input unit of the input / output device 10, and converted into learning data 118 that can be handled by the learning device 14 by the data converter 12. The learning data 118 is composed of words 120, features 122, prediction tags 124, and correct answer tags 125. The features 122 are "notation", "word length", and "capital letter", and "part of speech" as in the Japanese correct answer document data in FIG. 4 is excluded.
[0080]
A learning process 130 is performed on the learning data 118 by the learning device 14 to generate a learning result 24-1. Subsequently, the judgment unit 18 uses the learning result 24-1 by the judgment processing 132 to input, for each target word of the learning data 118, feature data 128 including surrounding words such as the target word "Reserve" for each target word in the learning data 118, for example. Then, the prediction tag of the target word is determined based on the determination data 134.
[0081]
Next, data reconstruction 136 in FIG. That is, the prediction tag determined in the determination process 132 in FIG. 16 is taken into the learning data, and the reconstructed learning data 138 updated is generated.
[0082]
Next, a learning process 152 is performed by the learning device 14. The learning process 152 includes the predicted tag 144 determined in the feature data 148 of the target word in the reconstructed learning data 138, and performs learning using a set of pairs of the predicted tag 144 and the correct answer tag 146. Generate 2.
[0083]
Using the second learning result 24-2, the determination unit 18 inputs the feature 142 of the reconstructed learning data 138 and performs a determination process 154, and determines a prediction tag as the determination data 156. This process is repeated until the number of processes i reaches the predetermined number N in the determination step 158, whereby the learning results 24-1 to 24-N are stored in the learning result database 22.
[0084]
FIG. 18 is a block diagram of a tagging determination process for each word of unknown document data of English text using the learning results 24-1 to 24-N of the learning result database 22 generated by the learning processing unit of FIG. , The functions of the learning device 14 and the learning controller 16 are omitted, and the other configurations are the same.
[0085]
FIG. 19 and FIG. 20 are explanatory diagrams of the processing procedure of the determination processing for tagging the unknown English document data of FIG.
[0086]
In FIG. 19, the unknown document data 160 in English is input by the function of the input unit of the input / output device 10, and the data converter 12 performs data conversion 162 for converting the data into test data 164 that can be processed by the learning device 14. . Next, a predictor tag is determined from the test data 164 by using the first learning result 24-1 by the determiner 18, and the determination data 176 is generated.
[0087]
Next, proceeding to FIG. 20, data reconstruction 178 is performed by the data reconstruction device 26, and reconfiguration test data 180 in which the prediction tag 186 updated by incorporating the prediction tag determined in the determination process 174 of FIG. 19 is generated. .
[0088]
Next, the features of the target word and the surrounding words including the prediction tag 186 of the reconstructed test data 180 are input by the determiner 18, and the determination is performed using the second learning result 24-2. Thus, the prediction tag of each target word is determined.
[0089]
The data reconstruction 178 and the determination processing 190 are repeated for the remaining learning results 24-3 to 24-N until i becomes N in the determination step 194, and the prediction tag of each word is determined. Then, the unknown English document data is attached to the determined tag by the reverse conversion by the data converter 12 and output to the outside.
[0090]
FIG. 20 shows another embodiment of the learning processing unit in the language processing apparatus according to the present invention. In this embodiment, a learning result based on the correct answer document data is simply generated. I do.
[0091]
The learning processing unit of the embodiment in FIG. 20 has a configuration in which the determination unit 18, the determination control device 20, and the data reconfiguration device 26 in the configuration of the learning processing unit in FIG. The learning processing unit of this embodiment first generates a learning result 240-1 from the correct answer document data from the target word and the features of the peripheral word and the correct answer tag of the target word without using the peripheral word. Next, a learning result 240-2 is generated from a set of the features of the surrounding words using the target word and the correct word as predicted words and the correct tag of the target word.
[0092]
FIG. 22 and FIG. 23 are explanatory diagrams of the processing procedure in the simplified-type learning processing unit in FIG. 21, in which a Japanese document is to be processed.
[0093]
In FIG. 22, first, the correct answer document data 196 is input by the function of the input unit of the input / output device 10. The correct answer document data 196 is the same as the correct answer document data 30 in FIG. 5, for example. Next, the data converter 12 performs data conversion 198 to convert the data into learning data 200 that can be handled by the learning device 14. The learning data 200 does not include the item of the prediction tag.
[0094]
Next, when the learning device 14 takes the feature of the target word and the peripheral words in the learning data 200, for example, the target word “broadcast” as an example, the feature data 202 including the surrounding peripheral words “Japan” and “association” and the target word The correct answer tag 204 “I-ORG” is input to the learning device 14, and the learning process 206 generates a learning result 240-1 as a first learning result.
[0095]
Of course, the learning process 206 is performed repeatedly on the next set of features and correct tags including the surrounding words before and after, with the second to fifth words as target words, excluding the first and last words in the learning data 200. Generates the learning result 240-1.
[0096]
Next, the processing proceeds to FIG. 23, in which data reconstruction 208 is performed on the correct answer document data 196 to generate learning data 210. A new prediction tag item is added to the learning data 210.
[0097]
Next, a learning process 218 is performed by the learning device 218 using the next set of features of the surrounding words and the target word in the learning data 210 and the correct tag as the prediction tag, and a learning result (second learning result) 240 -2 is generated.
[0098]
For example, as for the second target word “broadcast”, in the characteristic data 212 in the surrounding words “Japan” and “association”, the correct answer tags are “I-ORG” as the prediction tag 214 and “I-ORG” as the prediction tag 216. The feature data 212 and the correct tag 204 of the target word “I-ORG” are set in the learning device 14, and the learning result 240 as a rule for obtaining a prediction tag from the feature data 212 is obtained. -2 is generated. Of course, in the learning process 218, the learning process is repeatedly performed using the second to fifth words in the learning data 210 excluding the first and last words as target words.
[0099]
FIG. 24 is a flowchart of the simplified learning process in FIG. 21, which shows the contents of a simple learning processing program. First, in step S1, a Japanese correct answer document is input from the input / output device 10, and in step S2, the data is converted into a set of features and tags by the data converter 12, thereby generating first learning data. No prediction tag is provided in the first learning data.
[0100]
Next, in step S3, a first learning result is generated by the learning device 14 using the set of pairs of the feature and the correct answer tag of the target word and surrounding words in the learning data. The processing of steps S1 to S3 corresponds to FIG.
[0101]
Next, in Step S4, the correct answer document data is converted into features and tags by the data converter 12, a prediction tag is newly provided, and this is set to "?" To generate second learning data. Finally, in step S5, the learning device 14 generates a second learning result by using a pair of the feature of the peripheral word including the target word and the correct tag in the prediction tag in the second learning data and the correct tag.
[0102]
FIG. 25 is a block diagram of a simple determination processing unit for tagging Japanese unknown document data using the learning result generated by the simple learning processing unit of FIG. This simple determination processing unit is basically the same as the embodiment of FIG. 10, except that the learning result database 22 has only two learning results 240-1 and 240-2.
[0103]
26 and 27 are explanatory diagrams of the processing procedure of the simple determination processing unit in FIG. In FIG. 26, first, unknown document data 220 in Japanese is input through the input / output unit of the input / output device 10, and is converted by the data converter 12 into test data 224 in a format that can be handled by the determiner.
[0104]
Next, the judgment unit 18 uses the learning result 240-1 obtained first to obtain a target word from the target data included in the test data 224, for example, “broadcast” and the characteristic data 226 of the peripheral words “Japan” and “association”. The prediction tag of the word is determined by the determination process 228, and the determination data 230 is generated.
[0105]
Next, proceeding to FIG. 27, the data reconstructing device 26 generates reconstructed test data 234 in which the prediction tag determined in the determination process 228 of FIG.
[0106]
Next, using the second learning result 240-2, the determiner 18 uses the features of the target word and the peripheral words including the prediction tag included in the reconstructed test data 234, for example, in the case of the target word “broadcast”, The determination processing 238 for determining the tag of the target word from the characteristic data 236 is performed, and the determination data 242 is generated.
[0107]
Of course, the determination step 244 determines a prediction tag using the learning result 240-2 for each of the second to fifth target words excluding the first and last words in the reconstructed test data 234.
[0108]
In the next determination setting 224, if the number of processes i is less than the predetermined value N, the process returns to the data reconstruction 232, updates the prediction tag obtained in the determination process 238, creates the reconstruction test data 234 again, and performs the same learning. The determination process using the result 240-2 is repeated, and when the number of processes reaches N, the prediction tag of the reconstructed test data 234 obtained at that time is used as the tag determination result, and the data converter 12 After performing the inverse conversion of attaching the determined tag to each word of the input unknown document data, the function of the input / output device 10 as an output unit outputs the tagged document data to the outside.
[0109]
FIG. 28 is a flowchart of the simple determination processing unit of FIG. 25, and this flowchart corresponds to the content of the simple determination processing program. In FIG. 28, in step S1, Japanese unknown document data is input by the input / output device 10, and in step S2, the data is converted into a set of features and tags by the data converter 12, and the prediction tag is set to "?". Generate as the second test data.
[0110]
Next, in step S3, the tag of each word is predicted from the characteristics of the target word and the peripheral words in the test data by the determiner 14 using the first learning result. The processing of steps S1 to S3 corresponds to FIG. Next, in step S4, the prediction tag acquired in step S3 by the data reconstructing device 26 is replaced with the prediction tag of the test data, and new test data is generated as reconstructed test data. (2) The tag of each word is predicted from the characteristics of the target word including the prediction tag and the surrounding words in the test data using the learning result.
[0111]
In step S4, S5 or S6, the process is repeated while increasing i one by one in step S7 until the number of processes reaches the predetermined number N. Then, when the determination processing for N times is completed, inverse conversion is performed to convert the determination tag of each word determined to each word of the unknown document data, and output to the outside.
[0112]
FIGS. 29 and 30 show the same simple learning processing procedure as that of the embodiment of FIG. 21 for English correct answer document data. Since the simple learning process for the correct answer document data of the English sentence can be realized by a configuration as a learning processing unit excluding the morphological analysis device 11 in FIG. 21, the processing procedures in FIG. 29 and FIG. Description will be made with reference to FIG. 25 as in the case of Japanese.
[0113]
In FIG. 29, the correct answer document data 246 in English is input by the function of the input unit of the input / output device 10, and is converted by the data converter 12 into learning data 250 that can be handled by the learning device 14.
[0114]
Subsequently, the learning device 14 outputs another target word including the pair of the feature of the target word and the peripheral words in the learning data 250, for example, the feature data 252 of the target word “Reserve” and its correct tag 254, “I-ORG”. The learning process is performed by the learning device 14 including the feature in the above, and a learning result 240-1 for determining a prediction tag from the feature data 252 is generated.
[0115]
Subsequently, the process proceeds to FIG. 30, where the data converter 12 generates second learning data 260 including a prediction tag from the correct answer document data 246 in English. A learning process 268 is performed on the learning data 260 by the learning device 14 to generate a learning result 240-2.
[0116]
The learning process 268 uses the pair of the feature 262 of the peripheral word including the target word and the correct tag as the prediction tags 264 and 265 and the correct tag 266 of the target word, for example, with the target word “Reserve” as an example. , A learning result 240-2 for determining the prediction tag is generated. Such learning is repeatedly performed for each of the second to fifth words excluding the first and last words of the learning data 260 to obtain a learning result 240-2.
[0117]
FIGS. 31 and 32 show a case where tags are determined from unknown English document data using two learning results 240-1 and 240-2 for English sentences obtained by the simple learning process of FIGS. 29 and 30. It is a processing procedure of a simple determination process.
[0118]
Since this simple judgment processing procedure can be realized with the same configuration as the simple judgment processing unit for Japanese documents shown in FIG. 25, a simple judgment processing for English unknown document data will be described with reference to FIG. The following describes the determination processing.
[0119]
In FIG. 31, unknown document data 270 in English is input by the function of the input unit of the input / output device 10, and is converted by the data converter 12 into test data 274 in a format that can be handled by the determiner 18.
[0120]
Next, the determination unit 18 performs a determination process 278 of determining a prediction tag of each target word from the characteristics of the target word and the peripheral words of the test data 274 using the first learning result 240-1, and generates determination data 280. .
[0121]
Next, the processing proceeds to FIG. 32, in which the data reconstruction device 26 inserts the prediction tag obtained in the determination processing 278 in FIG. 31 into a part of the prediction tags of various data to generate reconstructed learning data 284.
[0122]
Subsequently, the determination processing 288 is performed by the determiner 18 on the feature data 286 of the reconstructed learning data 284 using the second learning result 240-2, and the determination data 290 for determining the prediction tag of each word is generated. .
[0123]
The data reconstruction 282 and the determination process 288 are repeatedly performed in the determination step 292 using the same learning result 240-2 until the number of processes i reaches the predetermined number N, and the prediction tag of the reconstruction test data 284 is used as the correct answer tag. Determine.
[0124]
Then, after performing the inverse conversion of attaching the determined tag to the unknown English document data input by the data converter 12, the tagged English document data processed by the function of the output unit of the input / output device 10 is output to the outside. Output.
[0125]
FIG. 33 shows another embodiment of the document processing apparatus of the present invention. In this embodiment, an external tagging system interface 296 is further provided in the embodiment of FIG.
[0126]
The external tagging system interface 296 uses the function of the input unit of the input / output device 10 to pass the received Japanese correct document data or unknown document data to the external tagging system 298 via a network such as the Internet, and to output the tagging result. The learning process and the judgment process are performed by using the acquired external tag obtained from the external tagging system 298 as a prediction tag.
[0127]
FIG. 33 shows an example of application of the document processing apparatus for Japanese document data shown in FIG. 2, but English document data without the morphological analyzer 11 can be applied to the language processing apparatus as it is. In the embodiment of FIG. 33, an example of generating and using the learning results 24-1 to 24-N is taken as an example, but two learning data 240-1 and 240-2 are generated and used. The same can be applied to an embodiment of a simple language processing device.
[0128]
Next, specific processing contents in the learning device and the determination device will be described by taking the learning process in the embodiment in FIG. 2 and the determination process in the embodiment in FIG. 10 using the learning result in FIG. 2 as an example.
[0129]
FIG. 34 is an explanatory diagram of performing the first learning from the correct answer data in the learning processing of the present invention and generating a first classification rule as a first learning result. In the following specific example of the learning process, a case where a decision tree generation algorithm, which will be described later, is used as a learning device is taken as an example.
[0130]
In FIG. 34, the correct answer tag 304 is added to the correct answer document data 300 for the word 302. The correct answer document data 300 exemplifies a document that states, "Wimbledon held in England in 2002 marked the beginning of a turbulence. Winner Selena Williams lost to Maria Goose of Australia in the first round." The document of the correct answer document data 300 is divided into parts of speech by morphological analysis, arranged as shown in the figure, and a correct answer tag 304 is attached to each.
[0131]
The decision tree generator 306, which functions as a learning unit, learns the correct answer document data 300 in accordance with the decision tree algorithm, thereby generating a first classification rule 308 serving as a first decision tree. The first classification rule 308 generated in this manner is stored in the learning result database 22-2 by storing the first classification rule 310 in the empty learning result database 22-1 in the initial state as shown in FIG. You.
[0132]
Next, as shown in FIG. 36, the prediction tag 312-1 of the correct document data 300 shown on the right side is determined by a determination process in which the generated first classification rule 308 is applied to the word 302 of the correct document data 300.
[0133]
Next, as in the case of the correct answer document data 300 in FIG. 37, data reconstruction is performed by capturing the prediction tag 312-1 determined in FIG. 36, and the reconstructed correct answer document data 300 is sent to the decision tree generator 314. A second learning is performed by inputting, and a second classification rule 316 as a second decision tree is generated.
[0134]
Next, as shown in FIG. 38, in addition to the first classification rule 308 already stored in the learning result database 22-2, the second classification rule 316 generated in FIG.
[0135]
Next, as shown in FIG. 39, the prediction tag of the correct document data 300 shown on the right side is determined by applying the second classification rule 316 to the correct document data 300 having the previous prediction tag 312-1 with respect to the word 302. 312-2 is determined and assigned.
[0136]
The above is the basic processing of the learning processing in the present invention shown in the embodiment of FIG. 2, but in practice, the third learning, the fourth learning,... Further, the third to Nth classification rules are generated. Needless to say, in the present invention, up to the second classification rule may be created.
[0137]
Here, an algorithm for creating a classification rule, that is, a decision tree in the learning process of the present invention will be described.
[0138]
FIG. 40 shows a function MakeTree 320 that generates a decision tree in the learning processing of the present invention. There are a plurality of algorithms for the function MakeTree 320 for generating this decision tree. For example, the following decision tree generation method is used.
[0139]
The decision tree generation method used in the present invention is proposed in Quinlan “C4.5: Pro-grams for Machine Learning” (1993) (Japanese translation “Data analysis by AI”) and the like. The outline of the algorithm is shown below. Here, the correlation between the category and the feature is based on information amount gain, mutual information amount, ² There are various methods, such as a test amount.
[0140]
[Decision Tree Generation Algorithm Generatetree (Learning Set, Tree)]
Step S1 :. If the training set is empty, assign the default category and exit the function.
Step S2: If the categories of the learning set are all the same, add a node to be judged to be the category to the tree and exit the function.
Step S3: From the features, the feature having the strongest correlation with the category is selected.
Step S4: Add a division with the selected feature to the tree.
Step S5: Divide the learning set by the selected features.
Step S6: Call Generatetree () with each of the divided learning sets as an argument.
[0141]
Regarding this decision tree generation algorithm, the main contents of the decision tree generation processing when the correct answer document data 300 is input are as follows. First, the character type of the correct answer document data 300 is selected through the processing of steps S1 and S2. In this case, four types of characters are included: kanji, hiragana, katakana, and alphanumeric characters.
[0142]
Next, in step S4, the learning set that is the correct document data 300 is decomposed according to the character type. That is, as shown in FIG. 41, the kanji set 300-1, the hiragana set 300-2, the katakana set 300-3, and the alphanumeric set 300-4 are decomposed.
[0143]
Subsequently, each set decomposed by the process of step S6 is input, and a function GenerateTree is called. With this function, for example, in the case of the kanji set 300-1 as an example, the category of “O” is attached as the kanji correct tag in the correct document data 300 of all nodes, so the “O” node is added as the correct tag, By repeating this for another set, for example, the first classification rule 308 shown in FIG. 34 is generated.
[0144]
Here, in the first classification rule 308 in FIG. 34, the features of the peripheral words are not incorporated, but in the second classification rule 316 after the addition of the prediction tags in FIG. Has been incorporated into.
[0145]
FIG. 42 is a specific example of a determination process for applying a prediction tag by applying the first classification rule 308 and the second classification rule 316 generated by the learning processes of FIGS. 34 to 39 to unknown document data.
[0146]
In FIG. 42, as the unknown document data 322, "Mika Hakkinen unfortunately missed the victory in the match at San Marino in 1998" is input, and the data is divided into words 324 in parts of speech by morphological analysis.
[0147]
A prediction tag 326-1 is assigned to each of the words 324 by a determination process that applies the first classification rule 308 stored in the learning result database 22 to the unknown document data 322.
[0148]
Next, the second classification rule 316 of the learning result database 22 is applied to the unknown document data 322 to which the prediction tag 326-1 is added, and the prediction tag 326-2 is added to each of the words 324.
[0149]
FIG. 43 shows the addition of a prediction tag by applying the first classification rule 308 in the first determination process of FIG. 42, for a part of “Mika Hakkinen is a match” which is a part of unknown document data.
[0150]
Here, the first classification rule 308 has a tree structure. For example, as shown by an arrow 308-1, the target word "game" includes nodes of "character type = kanji", "word length <= 3", and "part of speech = others". Thus, the prediction tag “O” is determined, and in this case, the answer is correct.
[0151]
On the other hand, “Mika” determines the prediction tag “I-LOC” through the nodes of “character type = Katakana” and “word length <4” as indicated by an arrow 308-2. In this case, the determined prediction tag is It is incorrect.
[0152]
FIG. 44 is a specific example of the determination processing according to the second classification rule 316 in which the prediction tag determined by the first classification rule 308 in FIG. 43 is added to the feature. First, although the target word “match” is correct in the first classification rule 308, even in the second classification rule 316, the prediction tag is “O” as indicated by an arrow 316-2, which is also correct.
[0153]
On the other hand, as for the target word “Mika”, as shown by an arrow 316-3, a “back tag = I-PER” is determined following a node of “character type = Katakana” and “word length <4”. It is determined to be "I-PER", and the answer is correct according to the second classification rule 316 in which the prediction tag of the peripheral word is incorporated in the feature.
[0154]
As described above, according to the present invention, the tag of the peripheral word is predicted for the target word of the unknown document by the first classification rule (first learning result) 308, and the second classification rule (second learning result) 316 By using a classification rule characterized by a prediction tag of a peripheral word, a tag of a target word can be determined with high accuracy from a correlation with the prediction tag of a peripheral word.
[0155]
FIG. 45 shows a comparison between the conventional method and the method of the present invention on the characteristic information when tagging the target word “Hakkinen” for “Mika Hakkinen is in the game” in the same unknown document data as in FIGS. 43 and 44. ing.
[0156]
FIG. 45A shows a feature of the conventional method 328 in which the tag of the peripheral word is not included in the feature, and no consideration is given to what kind of tag the preceding and following words are determined.
[0157]
FIG. 45 (B) shows the method 330 of Yamada shown in the prior art (A), in which a prediction tag of one side, for example, a preceding peripheral word is used for a target word. However, in this case, it works effectively only when there is a clue that the tag of the target word “Hakkinen” becomes “I-PER” in the word before the target word. For example, if the target word is “Mika”, “I-PER” of the surrounding word “Hakkinen” as a clue cannot be used effectively because it becomes a prediction tag of the subsequent word.
[0158]
FIG. 45 (C) is a feature of the technique 332 of the present invention, in which the target word “Hakkinen” uses, as its feature, prediction tags of a plurality of peripheral portions that match before and after. Therefore, if there is a clue somewhere around the target word, it can be used to accurately predict the tag.
[0159]
It should be noted that the present invention is not limited to the above embodiment, but includes appropriate modifications without impairing the objects and advantages thereof. Further, the present invention is not limited by the numerical values shown in the above embodiments.
[0160]
Here, the features of the present invention are listed as follows.
(Note)
(Appendix 1)
A learning processing step of generating a plurality of learning results by using a set of pairs of a target word included in a correct answer document data with a tag and features of surrounding words on both sides thereof and a correct answer tag,
A determination processing device, using the plurality of learning results, a determination processing step of determining a tag of each target word from the characteristics of the target word included in the unknown document data to be tagged and the surrounding words on both sides thereof;
A language processing method comprising: (1)
[0161]
(Appendix 2)
In the language processing method according to Supplementary Note 1, the learning processing step includes:
An input step of inputting the correct answer document data with a tag from the outside by the input device;
A data converter for converting the correct document data into learning data in a format handled by the learning device;
A first learning step of generating a first learning result of determining a prediction tag from the feature by using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data,
A first determining step of determining, by using a first learning result generated in the first learning step, a prediction tag of each target word from characteristics of the target word and peripheral words included in the learning data,
A data reconstructing step of reconstructing the prediction tag determined in the first determining step into a part of the features of the learning data by a data reconstructing device;
The learning device determines a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of the target word and the surrounding word including the prediction tag included in the learning data reconstructed in the data reconstruction step. A second learning step of generating a second learning result;
Using the second learning result generated in the second learning step, the determiner determines a tag of each target word from characteristics of the target word including the prediction tag and surrounding words included in the learning data. Two determination steps;
A learning repetition step of generating the plurality of learning results after the second learning result by repeating the data reconstructing step, the second learning step, and the second determination step a plurality of times;
A language processing method comprising: (2)
[0162]
(Appendix 3)
In the language processing method according to Supplementary Note 2, the determination processing step includes:
By the input device, an input step of inputting unknown document data to be tagged externally,
A data conversion step of converting the unknown document data into test data in a format handled by the learning device by the data converter;
A third determining step of determining a prediction tag of each target word from the characteristics of the target word and the peripheral words included in the test data by using the first learning result by the determiner; A second data reconstructing step of reconstructing the prediction tag determined in the third determining step by incorporating the prediction tag into a part of the feature of the learning data;
Using the second learning result, the determiner determines a tag of each target word from the characteristics of the target word and the surrounding words including the prediction tag included in the test data reconstructed in the second data reconstruction step. A fourth determining step of determining;
A determination repetition step of determining the tag of the unknown document data by repeating the data reconstruction step and the fourth determination step a plurality of times while switching to a learning result after a second learning result;
An output step of inverting the unknown document data into document data with the determined tag and outputting the unknown document data,
A language processing method comprising: (3)
[0163]
(Appendix 4)
In the language processing method according to Supplementary Note 1, the learning processing step includes:
An input step of inputting the correct answer document data with a tag from the outside by the input device;
A data conversion step of converting the correct answer document data into learning data in a format not including a prediction tag handled by the learning device by a data converter,
A first learning step of generating a first learning result of determining a prediction tag from the feature by using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data,
A data conversion step of converting the correct document data into learning data of a format handled by a learning device including a prediction tag by a data converter;
Using a set of pairs of a feature and a correct answer tag of a peripheral word including the target word and the correct answer tag included in the learning data as a predictive tag, the learning device generates a second learning result for determining the predictive tag from the feature A second learning step;
A language processing method comprising: (4)
[0164]
(Appendix 5)
In the language processing method according to Supplementary Note 4, the determination processing step includes:
By the input device, an input step of inputting unknown document data to be tagged externally,
A data conversion step of converting the unknown document data into test data in a format handled by the learning device by the data converter;
A first determining step of determining, by the determiner, a prediction tag of each target word from characteristics of the target word and the peripheral words included in the test data, using the first learning result; and A data reconstructing step of reconstructing the prediction tag determined in the first determining step by incorporating the prediction tag into a part of the feature of the learning data;
Using the second learning result, the determiner determines a tag of each target word from features of the target word including the prediction tag and surrounding words included in the test data reconstructed in the data reconstructing step. A second determination step;
A determination repetition step of determining the tag of the unknown document data by repeating the data reconstruction step and the second determination step a plurality of times using a second learning result;
An output step of inverting the unknown document data into document data with the determined tag and outputting the unknown document data,
A language processing method comprising: (5)
[0165]
(Appendix 6)
In the language processing method according to any one of supplementary notes 2 to 5,
The notation learning processing step and the determination processing step include a morphological analysis step of analyzing the correct document data or unknown document data received by the input device to determine a morpheme,
The language processing method is characterized in that, in the data conversion step, the correct answer document data or the unknown document data including the analyzed morpheme is converted into learning data of a format handled by a learning device by a data converter.
[0166]
(Appendix 7)
In the language processing method according to any one of supplementary notes 2 to 5,
The learning processing step and the determination processing step include an external tagging step of passing the correct document data or unknown document data received by the input device to an external tagging system to obtain a tagging result,
The data conversion step includes a step of converting the correct document data or the unknown document data, including the tagging result obtained from the tagging system, into learning data in a format handled by a learning device by a data converter. Processing method.
[0167]
(Appendix 8)
On the computer,
A learning processing step of generating a plurality of learning results by using a set of pairs of a target word included in a correct answer document data with a tag and features of surrounding words on both sides thereof and a correct answer tag,
A determination processing step of determining the tag of each target word from the characteristics of the target word included in the unknown document data to be tagged and the surrounding words on both sides using the plurality of learning results;
A program characterized by executing (6)
[0168]
(Appendix 9)
In the program according to attachment 8, the learning processing step includes:
An input step of inputting the correct answer document data with a tag from the outside by the input device;
A data converter for converting the correct document data into learning data in a format handled by the learning device;
A first learning step of generating a first learning result of determining a prediction tag from the feature by using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data,
A first determining step of determining, by using a first learning result generated in the first learning step, a prediction tag of each target word from characteristics of the target word and peripheral words included in the learning data,
A data reconstructing step of reconstructing the prediction tag determined in the first determining step into a part of the features of the learning data by a data reconstructing device;
The learning device determines a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of the target word and the surrounding word including the prediction tag included in the learning data reconstructed in the data reconstruction step. A second learning step of generating a second learning result;
Using the second learning result generated in the second learning step, the determiner determines a tag of each target word from characteristics of the target word including the prediction tag and surrounding words included in the learning data. Two determination steps;
A learning repetition step of generating the plurality of learning results after the second learning result by repeating the data reconstructing step, the second learning step, and the second determination step a plurality of times;
A program characterized by comprising: (7)
[0169]
(Appendix 10)
In the program according to supplementary note 9, the determination processing step includes:
An input step of inputting unknown document data to be tagged from outside,
A data conversion step of converting the unknown document data into test data in a format handled by the learning device;
A third determining step of using the first learning result to determine a prediction tag of each target word from characteristics of the target word and surrounding words included in the test data;
A second data reconstructing step of reconstructing the prediction tag determined in the third determining step by incorporating the prediction tag into a part of the feature of the learning data;
Using the second learning result, determine a tag of each target word from features of the target word including the prediction tag and surrounding words included in the test data reconstructed in the second data reconstruction step Steps and
A determination repetition step of determining the tag of the unknown document data by repeating the data reconstruction step and the fourth determination step a plurality of times while switching to a learning result after a second learning result;
An output step of inversely converting the unknown document data into document data with the determined tag and outputting the document data;
A program characterized by comprising: (8)
[0170]
(Appendix 11)
In the program according to attachment 8, the learning processing step includes:
An input step of inputting correct answer document data with a tag from outside,
A data conversion step of converting the correct answer document data into learning data not including a prediction tag handled by a learning device,
A first learning step of generating a first learning result for determining a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of the target word and surrounding words included in the learning data;
A data conversion step of converting the correct document data into learning data in a format handled by a learning device including a prediction tag;
A second learning step of generating a second learning result of determining a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of a peripheral word including the target word and the correct answer tag included in the learning data as a prediction tag When,
A program characterized by comprising:
[0171]
(Appendix 12)
In the program according to supplementary note 11, the determination processing step includes:
An input step of inputting unknown document data to be tagged from outside,
A data conversion step of converting the unknown document data into test data in a format handled by the learning device;
A first determination step of determining a prediction tag of each target word from characteristics of the target word and surrounding words included in the test data using the first learning result;
A data reconstructing step of reconstructing the prediction tag determined in the first determining step into a part of the features of the learning data by the data reconstructing device;
A second determining step of using the second learning result to determine a tag of each target word from characteristics of the target word and the surrounding words including the prediction tag included in the test data reconstructed in the data reconstructing step; ,
A determination repetition step of determining the tag of the unknown document data by repeating the data reconstruction step and the second determination step a plurality of times using a second learning result;
An output step of inversely converting the unknown document data into document data with the determined tag and outputting the document data;
A program characterized by comprising:
[0172]
(Appendix 13)
In the program according to any one of supplementary notes 9 to 12,
The learning processing step and the determination processing step include a morphological analysis step of determining a morpheme by analyzing correct document data or unknown document data received by the input device,
The data conversion step is a program for converting, by a data converter, the correct document data or unknown document data including the analyzed morpheme into learning data in a format handled by a learning device.
[0173]
(Appendix 14)
In the program according to any one of supplementary notes 9 to 12,
The learning processing step and the determination processing step include an external tagging step of passing the correct document data or unknown document data received by the input device to an external tagging system to obtain a tagging result,
The data conversion step includes a step of converting the correct document data or unknown document data, including a tagging result obtained from the tagging system, into learning data in a format handled by a learning device by a data converter. .
[0174]
(Appendix 15)
A learning processing unit that generates a plurality of learning results using a set of pairs of a target word included in a target word included in a correct answer document data and a surrounding word on both sides thereof and a correct answer tag,
A determination processing unit configured to determine a tag of each target word from characteristics of the plurality of peripheral words on both sides thereof,
A language processing device comprising: (9)
[0175]
(Appendix 16)
In the language processing device according to attachment 15, the learning processing unit includes:
An input device for inputting tagged document data from outside,
A data conversion device that converts the correct answer document data into learning data of a format handled by a learning device,
A first learning device that generates a first learning result for determining a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data;
A first determiner configured to determine a prediction tag of each target word from characteristics of the target word and surrounding words included in the learning data, using a learning result generated in the first learning step;
A data reconstructing device for reconstructing the prediction tag determined by the first determiner into a part of the feature of the learning data by a data reconstructing device,
A second learning result for determining a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of a target word and a surrounding word including the prediction tag included in the learning data reconstructed by the data reconstruction device. A second learning device that generates
A second determiner that determines a tag of each target word from characteristics of the target word and the peripheral words including the prediction tag included in the learning data, using a second learning result generated by the second learning device;
A learning repetition unit that generates the plurality of learning results after the second learning result by repeating the data reconstructing device, the second learning device, and the second determination device a plurality of times;
A language processing device comprising: (10)
[0176]
(Appendix 17)
In the language processing device according to Supplementary Note 16, the determination processing unit includes:
An input device for inputting unknown document data to be tagged from outside,
A data conversion device that converts the unknown document data into test data in a format handled by the learning device;
A third determiner that determines a prediction tag of each target word from the characteristics of the target word and surrounding words included in the test data using the first learning result;
A second data reconstructing device configured to reconstruct the prediction tag determined by the third determiner into a part of the feature of the learning data by the data reconstructing device;
Using the second learning result, determine a tag of each target word from characteristics of the target word including the prediction tag and surrounding words included in the test data reconstructed by the second data reconstructing device. Vessels,
A determination repetition unit that determines the tag of the unknown document data by repeating the second data reconstruction device and the fourth determination unit a plurality of times while switching to the learning result after the second learning result;
An output device that inversely converts the unknown document data into document data with the determined tag and outputs the document data;
A language processing device comprising: (11)
[0177]
(Appendix 18)
In the language processing device according to attachment 15, the learning processing unit includes:
An input device for inputting tagged document data from outside,
A data conversion unit that converts the correct answer document data into learning data in a format not including a prediction tag handled by a learning device,
A first learning device that generates a first learning result for determining a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data;
A data conversion device that converts the correct answer document data into learning data of a format handled by a learning device including a prediction tag,
A second learning unit that generates a second learning result for determining a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of a peripheral word including the target word and the correct answer tag included in the learning data as a prediction tag. When,
A language processing device comprising:
[0178]
(Appendix 19)
In the language processing device according to attachment 18, the determination processing unit includes:
An input device for inputting unknown document data to be tagged from outside,
A data conversion device that converts the unknown document data into test data in a format handled by the learning device;
A first determiner that determines a prediction tag of each target word from characteristics of the target word and surrounding words included in the test data using the first learning result;
A data reconstructing device for reconstructing the prediction tag determined by the first determiner into a part of the feature of the learning data,
A second determiner that determines a tag of each target word from characteristics of the target word and the surrounding words including the prediction tag included in the test data reconstructed by the data reconstructing device using the second learning result; and ,
A determination repetition unit that determines the tag of the unknown document data by repeating the data reconstructing device and the second determination unit a plurality of times using the second learning result;
An output device that inversely converts the unknown document data into document data with the determined tag and outputs the document data;
A language processing device comprising:
[0179]
(Appendix 20)
In the language processing device according to any one of supplementary notes 16 to 19,
The learning processing unit and the determination processing unit include a morphological analysis device that determines the morpheme by analyzing the correct document data or unknown document data received by the input device,
A language processing apparatus, wherein the data conversion device converts the correct document data or unknown document data including the analyzed morpheme into learning data in a format handled by a learning device.
[0180]
(Appendix 21)
In the language processing device according to any one of supplementary notes 16 to 19,
The learning processing unit and the determination processing unit include an external tagging interface that obtains a tagging result by passing correct document data or unknown document data received by the input device to an external tagging system,
A language processing apparatus, wherein the data conversion device converts the correct document data or unknown document data into learning data in a format handled by a learning device, including a tagging result obtained from the tagging system.
[0181]
【The invention's effect】
As described above, according to the present invention, a plurality of learning results are generated using a set of pairs of a correct word and a feature of a target word and surrounding words included in the correct word data with a tag. In order to determine the tag of each target word from the characteristics of the target word and surrounding words included in the unknown document data using the result, the unknown document is determined using the tags of the surrounding words that have a correlation with the tag of the target word. Tag can be determined, and furthermore, a plurality of learning results are generated, and this is used for the tag determination of the unknown document, and the tag determination of the unknown document is repeated. As a result, the accuracy of the target word determination tag is improved, and the tagging of the unknown document can be performed with high accuracy.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating the principle of the present invention.
FIG. 2 is a block diagram of a language processing device functioning as a learning processing unit for Japanese correct answer documents;
FIG. 3 is an explanatory diagram of a hardware environment of a computer to which the language processing device of the present invention is applied;
FIG. 4 is an explanatory diagram of Japanese document data subjected to morphological analysis.
FIG. 5 is an explanatory diagram of the correct answer document data.
FIG. 6 is an explanatory diagram of unknown document data.
FIG. 7 is an explanatory diagram of a processing function in a learning processing unit in FIG. 2;
FIG. 8 is an explanatory diagram of a processing function following FIG. 7;
FIG. 9 is a flowchart of a learning process in FIG. 2;
10 is a block diagram of a language processing device that functions as a determination processing unit using the learning result of FIG. 2;
FIG. 11 is an explanatory diagram of a processing function in a determination processing unit in FIG. 10;
FIG. 12 is an explanatory diagram of a processing function following FIG. 11;
FIG. 13 is an explanatory diagram of a determination result for an unknown document.
FIG. 14 is a flowchart of a determination process in FIG. 11;
FIG. 15 is a block diagram of a language processing apparatus functioning as a learning processing unit for an English correct answer document;
16 is an explanatory diagram of a processing function in the learning processing unit in FIG.
FIG. 17 is an explanatory diagram of a processing function following FIG. 16;
FIG. 18 is a block diagram of a language processing device functioning as a determination processing unit using the learning result of FIG. 16;
FIG. 19 is an explanatory diagram of a processing function in a determination processing unit in FIG. 18;
FIG. 20 is an explanatory diagram of a processing function following FIG. 19;
FIG. 21 is a block diagram of a language processing apparatus functioning as a simple learning processing unit for a Japanese correct answer document;
FIG. 22 is an explanatory diagram of a processing function in the learning processing unit in FIG. 21;
FIG. 23 is an explanatory diagram of a processing function following FIG. 22;
FIG. 24 is a flowchart of a learning process in FIG. 21;
FIG. 25 is a block diagram of a language processing device functioning as a determination processing unit using the learning result of FIG. 21;
FIG. 26 is an explanatory diagram of a processing function in the determination processing unit of FIG. 25;
FIG. 27 is an explanatory diagram of a processing function following FIG. 26;
FIG. 28 is a flowchart of a determination process in FIG. 25;
FIG. 29 is an explanatory diagram of a processing function in a simple learning processing unit for an English correct answer document;
FIG. 30 is an explanatory diagram of a processing function following FIG. 29;
FIG. 31 is an explanatory diagram of a processing function in a determination processing unit using the learning results of FIGS. 20 and 30;
FIG. 32 is an explanatory diagram of a processing function following FIG. 31;
FIG. 33 is a block diagram of the language processing apparatus of the present invention having an interface for accessing an external tagging system;
FIG. 34 is an explanatory diagram of a learning process of generating a first classification rule by performing a first learning from correct answer document data.
FIG. 35 is an explanatory diagram of a learning result database storing the first classification rules generated in FIG. 34;
FIG. 36 is an explanatory diagram of a determination process of applying a first classification rule to a correct answer document data and adding a prediction tag;
FIG. 37 is an explanatory diagram of a learning process of generating a second classification rule by performing a second learning from the correct answer document data to which the prediction tag is added in FIG. 36;
FIG. 38 is an explanatory diagram of the learning result database storing the second classification rules generated in FIG. 37 in addition to the first classification rules.
39 is an explanatory diagram of a judgment learning process of applying a classification rule to the correct answer document data to which a prediction tag has been added and determining a prediction tag; FIG.
FIG. 40 is an explanatory diagram of a function MakeTree for generating a decision tree serving as a classification rule.
FIG. 41 is an explanatory diagram of a decision tree generation process for dividing a learning set by character types.
FIG. 42 is an explanatory diagram of a determination process of applying a first classification rule and a second classification rule to unknown document data successively to add a prediction tag;
FIG. 43 is an explanatory diagram of a determination process for assigning a prediction tag by applying a first classification rule by using a part of unknown document data as an example;
FIG. 44 is an explanatory diagram of a determination process of adding a prediction tag to unknown document data including the prediction tag added in FIG. 43 by applying a second classification rule.
FIG. 45 is a diagram illustrating a comparison between the conventional method and the method of the present invention in which a prediction tag is added.
[Explanation of symbols]
10: I / O device
11: Morphological analyzer
12: Data converter
14: Learning device
16: Learning controller
18: Judge
20: Judgment controller
22: Learning database
24-1, 240-1: learning result (first learning result)
24-2, 240-2: Learning result (second learning result)
24-3 to 24-N: Learning result
26: Data reconstruction device
28: Japanese document data
30: Correct answer document data
32, 80, 200: Learning data
48: Word
50: Features
52: prediction tag
54: Correct answer tag
56, 70: feature data
64, 76, 96, 106, 134, 156, 176, 192, 230, 242, 280, 290: judgment data
68, 138, 210: Reconstruction learning data
84, 164, 224, 250, 274: test data
100, 180, 234, 260, 284: Reconstruction test data
300: External tagging system interface

Claims (11)

A learning processing step of generating a plurality of learning results by using a set of pairs of a target word included in a correct answer document data with a tag and features of surrounding words on both sides thereof and a correct answer tag,
A determination processing device, using the plurality of learning results, a determination processing step of determining a tag of each target word from the characteristics of the target word included in the unknown document data to be tagged and the surrounding words on both sides thereof;
A language processing method comprising: In the language processing method according to claim 1, the learning processing step includes:
An input step of inputting the correct answer document data with a tag from the outside by the input device;
A data converter for converting the correct document data into learning data in a format handled by the learning device;
A first learning step of generating a first learning result of determining a prediction tag from the feature by using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data,
A first determining step of determining, by using a first learning result generated in the first learning step, a prediction tag of each target word from characteristics of the target word and peripheral words included in the learning data,
A data reconstructing step of reconstructing the prediction tag determined in the first determining step into a part of the features of the learning data by a data reconstructing device;
The learning device determines a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of the target word and the surrounding word including the prediction tag included in the learning data reconstructed in the data reconstruction step. A second learning step of generating a second learning result; and the target word including the prediction tag included in the learning data and a surrounding area, using the second learning result generated by the second learning step by the determiner. A second determining step of determining a tag of each target word from the characteristics of the word;
A learning repetition step of generating the plurality of learning results after the second learning result by repeating the data reconstructing step, the second learning step, and the second determination step a plurality of times;
A language processing method comprising: In the language processing method according to claim 1, the learning processing step includes:
An input step of inputting the correct answer document data with a tag from the outside by the input device;
A data conversion step of converting the correct answer document data into learning data in a format not including a prediction tag handled by the learning device by a data converter,
A first learning step of generating a first learning result of determining a prediction tag from the feature by using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data,
A data conversion step of converting the correct document data into learning data of a format handled by a learning device including a prediction tag by a data converter;
Using a set of pairs of a feature and a correct answer tag of a peripheral word including the target word and the correct answer tag included in the learning data as a predictive tag, the learning device generates a second learning result for determining the predictive tag from the feature A second learning step;
A language processing method comprising: In the language processing method according to claim 3, the determination processing step includes:
By the input device, an input step of inputting unknown document data to be tagged externally,
A data conversion step of converting the unknown document data into test data in a format handled by the learning device by the data converter;
A first determining step of determining, by the determiner, a prediction tag of each target word from characteristics of the target word and surrounding words included in the test data, using the first learning result;
A data reconstructing step of reconstructing the prediction tag determined in the first determining step into a part of the features of the learning data by the data reconstructing device;
Using the second learning result, the determiner determines a tag of each target word from features of the target word including the prediction tag and surrounding words included in the test data reconstructed in the data reconstructing step. A second determination step;
A determination repetition step of determining the tag of the unknown document data by repeating the data reconstruction step and the second determination step a plurality of times using a second learning result;
An output step of inverting the unknown document data into document data with the determined tag and outputting the unknown document data,
A language processing method comprising: In the language processing method according to any one of claims 2 to 4,
The learning processing step and the determination processing step include a morphological analysis step of determining a morpheme by analyzing correct document data or unknown document data received by the input device,
The language conversion method, wherein the data conversion step converts the correct document data or the unknown document data including the analyzed morpheme into learning data in a format handled by a learning device by a data converter. On the computer,
A learning processing step of generating a plurality of learning results by using a set of pairs of a target word included in a correct answer document data with a tag and features of surrounding words on both sides thereof and a correct answer tag,
A determination processing step of determining the tag of each target word from the characteristics of the target word included in the unknown document data to be tagged and the surrounding words on both sides using the plurality of learning results;
A program characterized by executing In the program according to claim 6, the learning processing step includes:
An input step of inputting the correct answer document data with a tag from the outside by the input device;
A data converter for converting the correct document data into learning data in a format handled by the learning device;
A first learning step of generating a first learning result of determining a prediction tag from the feature by using a set of pairs of a feature and a correct answer tag of a target word and surrounding words included in the learning data,
A first determining step of determining, by using a first learning result generated in the first learning step, a prediction tag of each target word from characteristics of the target word and peripheral words included in the learning data,
A data reconstructing step of reconstructing the prediction tag determined in the first determining step into a part of the features of the learning data by a data reconstructing device;
The learning device determines a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of the target word and the surrounding word including the prediction tag included in the learning data reconstructed in the data reconstruction step. A second learning step of generating a second learning result; and the target word including the prediction tag included in the learning data and a surrounding area, using the second learning result generated by the second learning step by the determiner. A second determining step of determining a tag of each target word from the characteristics of the word;
A learning repetition step of generating the plurality of learning results after the second learning result by repeating the data reconstructing step, the second learning step, and the second determination step a plurality of times;
A program characterized by comprising: In the program according to claim 7, the determination processing step includes:
An input step of inputting unknown document data to be tagged from outside,
A data conversion step of converting the unknown document data into test data in a format handled by the learning device;
A third determining step of using the first learning result to determine a prediction tag of each target word from characteristics of the target word and surrounding words included in the test data;
A second data reconstructing step of reconstructing the prediction tag determined in the third determining step by incorporating the prediction tag into a part of the feature of the learning data;
Using the second learning result, determine a tag of each target word from features of the target word including the prediction tag and surrounding words included in the test data reconstructed in the second data reconstruction step Steps and
A determination repetition step of determining the tag of the unknown document data by repeating the data reconstruction step and the fourth determination step a plurality of times while switching to a learning result after a second learning result;
An output step of inversely converting the unknown document data into document data with the determined tag and outputting the document data;
A program characterized by comprising: A learning processing unit that generates a plurality of learning results using a set of pairs of a target word included in a target word included in a correct answer document data and a surrounding word on both sides thereof and a correct answer tag,
A determination processing unit configured to determine a tag of each target word from characteristics of the plurality of peripheral words on both sides thereof,
A language processing device comprising: The language processing apparatus according to claim 9, wherein the learning processing unit comprises:
An input device for inputting tagged document data from outside,
A data conversion device that converts the correct answer document data into learning data of a format handled by a learning device,
A first learning unit that generates a first learning result for determining a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of the target word and the surrounding words included in the learning data; A first determiner for determining a prediction tag of each target word from characteristics of the target word and surrounding words included in the learning data using a learning result generated in the first learning step;
A data reconstructing device for reconstructing the prediction tag determined by the first determiner into a part of the feature of the learning data by a data reconstructing device,
A second learning result for determining a prediction tag from the feature using a set of pairs of a feature and a correct answer tag of a target word and a surrounding word including the prediction tag included in the learning data reconstructed by the data reconstruction device. A second learning device that generates
A second determiner that determines a tag of each target word from characteristics of the target word and the peripheral words including the prediction tag included in the learning data, using a second learning result generated by the second learning device;
A learning repetition unit that generates the plurality of learning results after the second learning result by repeating the data reconstructing device, the second learning device, and the second determination device a plurality of times;
A language processing device comprising: In the language processing device according to claim 10, the determination processing unit includes:
An input device for inputting unknown document data to be tagged from outside,
A data conversion device that converts the unknown document data into test data in a format handled by the learning device;
A third determiner that determines a prediction tag of each target word from the characteristics of the target word and surrounding words included in the test data using the first learning result;
A second data reconstructing device configured to reconstruct the prediction tag determined by the third determiner into a part of the feature of the learning data by the data reconstructing device;
Using the second learning result, determine a tag of each target word from characteristics of the target word including the prediction tag and surrounding words included in the test data reconstructed by the second data reconstructing device. Vessels,
A determination repetition unit that determines the tag of the unknown document data by repeating the second data reconstruction device and the fourth determination unit a plurality of times while switching to the learning result after the second learning result;
An output device that inversely converts the unknown document data into document data with the determined tag and outputs the document data;
A language processing device comprising:

Images