KR20000073477A - Method and device of korean-to-english query translation using mutual information - Google Patents

Abstract

PURPOSE: A Korean-English query conversion method using inter-information and an apparatus thereof are provided for generating a high quality inquiry language by adapting an inter-information to multiple processes of a conversion operation. CONSTITUTION: An inquiry language conversion apparatus using an inter-information includes a keyword extractor(11), a dictionary based inquiry language converter(12), a band language ambiguity overcoming unit(13), an inquiry language word weight providing unit(14), and a band dictionary(15). The keyword extractor(11) extracts a keyword for a search based on a type analysis and a part of a speech tagging with respect to an inquiry language of an original language.

Description

Method and device of korean-to-english query translation using mutual information}

The present invention relates to a method and an apparatus for converting Korean-English queries using mutual information, and more particularly, to querying mutual information in a cross-language document search that enables a user to search for a document written in a language different from that of a query language. The present invention relates to a query conversion method and apparatus for improving the accuracy of cross-language document retrieval by enabling high quality query to be applied to various stages of the conversion process.

In the present invention, in the case of cross-language document retrieval that makes it possible to search different language documents easily through query conversion, especially when the language of the query language and the search target language are different in the text information retrieval, the bandword ambiguity in the conversion of the query word using mutual information The present invention relates to a query conversion method and an apparatus capable of generating a high quality band query by resolving and assigning a query word weight.

Conventionally, in cross-language document search by query conversion, only a band dictionary is used, so when a document is searched using a converted query including band words with many word ambiguities, it is about 40% to one-word search. It is known that the search accuracy is only 60%. Later, as a result of research on effective query conversion method, the method of using word air frequency information to solve band word ambiguity or using word concept information such as the thesaurus was studied. However, in the case of simply using word frequency information, since the simple air frequency absolute value of two words is used, it is difficult to determine the relative importance of band words in a large amount of text corpus. This results in inaccurate band word generation, resulting in poor performance of cross-language document retrieval using these band queries.

Accordingly, the present invention is to solve the above problems of the prior art, an object of the present invention is to replace the relative air of the word pair from the large text corpus of the search target instead of using the absolute frequency of the word as the air information of the band words By extracting the mutual information from the information, applying it to the word ambiguity resolution of the band word, and applying the mutual information of the band word to the query word weighting on the remaining band words without the band word resolution, The present invention provides a method and apparatus for converting a query word, which can generate a, thereby greatly improving the search accuracy of cross-language document search.

1 is a block diagram of a query conversion device using mutual information according to the present invention;

Fig. 2 is a diagram showing an example of mutual information extraction of search query words extracted from an AP communication corpus in a cross-language document search environment of TREC-6.

FIG. 3 is a diagram illustrating mutual information values extracted from a band dictionary based query conversion result for Korean query "car air pollution"; FIG.

Fig. 4 is a diagram showing the experimental results of the cross-language document search of the query conversion method of the present invention in the TREC-6 cross-language document search environment.

* Explanation of symbols for main parts of the drawings

11: keyword extraction unit

12: dictionary-based query conversion unit

13: Bandword ambiguity resolution part

14: query word weighting unit

15: Band Dictionary

16: English Corpus

In accordance with an aspect of the present invention, there is provided an apparatus for query query conversion. The apparatus for query query converting a cross-language document includes: keyword extraction means for extracting a keyword for searching by morpheme analysis and part-of-speech tagging of a query language of a source language; ; Query word converting means for converting a source language keyword extracted by said keyword extracting means into candidate keywords of another language by searching a band dictionary; Computing mutual information from the candidate text corpus of the bandword language for the candidate bandwords generated by the query word converting means, and determining a first band query word for selecting a candidate bandword whose final information is equal to or greater than an arbitrary first threshold as a final query. Way; And second band query determining means for determining a band query by giving different weights to words for which the band query is not determined by the first band query determining means according to the size of the mutual information. .

In addition, the query conversion method according to the present invention for achieving the above object, in the query conversion method for cross-language document search, extracting a keyword for the search by morphological analysis and part-of-speech tagging for the query language of the source language Stage 1; A second step of converting a source language keyword extracted in the first step into candidate keywords of another language by searching a band dictionary; Calculating a mutual information from the search target text corpus of the bandword language with respect to the candidate bandwords converted in the second step, and selecting a candidate bandword whose final information is equal to or greater than an arbitrary first threshold as a final query; And a fourth step of determining a final band query by assigning different weights to words not determined in the third step according to the size of the mutual information.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a query conversion apparatus using mutual information according to the present invention, in which 11 is a keyword extraction unit, 12 is a dictionary-based query conversion unit, 13 is a band word ambiguity resolving unit, and 14 is a query word weighting unit. , 15 are band dictionaries, and 16 are English corpus.

The present invention constructs a multi-step query conversion process using mutual information, which is word air information extracted from a text corpus of search documents, as well as a simple band dictionary, to generate a high quality target query from a source query. This multi-stage query conversion process includes a keyword extraction unit 11, a band dictionary-based query conversion unit 12, a band word ambiguity elimination unit 13, and a query word weighting unit 14 as shown in FIG. Create

The keyword extracting unit 11 is a first step of the query conversion step. The keyword extracting unit 11 extracts a keyword for searching by morphological analysis and part-of-speech tagging of the query language of the source language. Here, the part of the part-of-speech tagger is to obtain an accurate morphological sequence from a plurality of candidate morphological columns generated by morphological analysis. Without this tagging, all unnecessary keywords are used in a search process to reduce the effectiveness of the search.

The band dictionary-based query term converter 12 converts the raw query keyword extracted by the keyword extractor 11 into band words of another language using a band dictionary to generate a plurality of candidate band words. In the query conversion unit 12 of the present invention, the band dictionary reference is composed of word-by-word conversion and phrase-level conversion. In the original query, the exact phrase recognition is found by the phrase conversion of the band dictionary by the phrase reference, but the phrase band dictionary is usually composed of the band dictionary with the unknown word problem such as new words and terminology. There is a difficulty. In actual query conversion, query conversion failure due to unregistered words or cross-language document retrieval performance are a large part.

The bandword ambiguity solving unit 13 cancels bandword ambiguity for a plurality of candidate bandwords generated by the dictionary-based query word converting unit 12. In this case, the mutual information, which is word air information extracted from the search target text corpus of the bandword language, is removed. Take advantage. That is, the bandword ambiguity canceling unit 13 uses the correlation information between the bandword words obtained from the frequency of occurrence of the word through the text corpus of the language, so that the mutual information is equal to or greater than an arbitrary threshold value (2.0 in the embodiment of the present invention). If it is, the mutual name is recognized as high and the final band language is selected. The mutual information calculation obtained from the text corpus is shown in Equation 1.

That is, the mutual information exists in which the word x and the word y coexist together in an arbitrary context range (in the present invention, obtained by an experiment with a range w of 6 in which the two words appear together) on an N size text corpus. Relative statistics. In other words, mutual information refers to the frequency with which two words exist simultaneously on the text corpus. If the mutual information MI (x, y) is greater than or equal to a certain threshold, the word is highly correlated and can be selected as the final band word.However, if MI (x, y) has a value less than 0, the words x and y are complementary to each other. Since it has a distribution, it is not directly applicable to solving the bandword ambiguity. Therefore, in order to apply the mutual information value to the bandword ambiguity resolution, an appropriate threshold value is used as the bandword ambiguity resolution standard.

2 shows examples of mutual information on an English text corpus of English query words used in an embodiment of the present invention.

The query word weighting unit 14 is a final step of query conversion, and is a function unit that selects a final query by assigning a query weight to words whose ambiguity has not been resolved by the bandword ambiguity canceling unit 13. Use informational values. At this time, the mutual information is assigned one or more values smaller than the mutual information threshold applied to the ambiguity resolution, and given a relative query weight for the interval mutual information value in any interval in consideration of the distribution of the mutual information values of the query word pairs. Apply the method. That is, the query word weighting unit 14 determines the final band query by weighting the mutual information of the two words in the band word ambiguity eliminating unit 13 according to the mutual information when the mutual information of the two words is equal to or less than a certain threshold (2.0). do. In the exemplary embodiment of the present invention, when the mutual information is larger than 1 and smaller than 2.0, a weight of 0.75 is assigned, and when the mutual information is smaller than 1 and larger than 0, a weight of 0.6 is assigned.

The query transformation using mutual information is performed with the apparatuses of the query transformation block diagram of FIG. 1, and the actual query transformation requires a raw query, a searched text document corpus 16, and a band dictionary 15. In addition, the quality of the query words generated by the query conversion can be known by comparing the search performance by the cross-language document search system.

In query conversion using a general band dictionary, band ambiguity occurs in which a word of a target query word is translated into a plurality of band words. For example, if the Korean query is "Car Air Pollution", converting it to English query language through the band dictionary converts "car" to "motocar", "automobile", "car", and "air" Are converted to "air", "atmosphere", "empty vessel", "bowl", and "contamination" to "pollution" and "contamination", respectively. Searching using all of these band words will result in poor search performance. The operation of the multi-level query conversion method using the mutual information of the present invention can be described by the above-described Korean-English query conversion and cross-language document search experiment.

The mutual information threshold value for band word ambiguity resolution used in the embodiment of the present invention was 2.0, and the weight of each information interval for query word weighting is 0.75 when the mutual information MI is greater than 1.0, as shown in FIG. If mutual information MI was in the range of 0 to 1, it was 0.6.

In the above example of Korean-English query conversion, an embodiment of resolving bandword ambiguity using mutual information and assigning query word weights will be described.

In the case of the Korean query of "car air pollution", in the simple band dictionary-based query word conversion, several English words appear as shown in Fig. 3 due to translation ambiguity, and the English conversion words of the Korean word w1 (car) and w2 (air) In the line between and w3 (pollution), the mutual information MI value of each word pair is calculated. In the case of the words w1 (car) and w2 (air), the band words automobile and air have the largest mutual information value, and in the case of the words w2 (air) and w3 (pollution), air and pollution are the most It has a large mutual information value. Accordingly, in this example, 4b of FIG. 3, which is the case of the band words air and pollution, is selected as the final band words by applying bandword ambiguity resolution of which the mutual information value exceeds the threshold 2.

Next, an embodiment of query word weighting using mutual information will be described.

Since the mutual information threshold for eliminating the wordword ambiguity is set to 2.0, the mutual information value for assigning the mutual information interval weight is 0.75 based on 1.0. Therefore, word pairs having a mutual information MI value between 1.0 and 2.0 have a query weight value of 0.75 and the remaining word pairs n have a value of 1-0.75 divided by n. That is, a value of 0.25 / n is given. In the example of FIG. 3, in the case of the band words automobile and air, 4a of FIG. 3 indicates that the band words of the query word w1 (auto) are (motocar, 0.125), (automobile, 0.75), has a word weight of (car, 0.125).

The operation of the Korean-English query translation method for cross-language document retrieval of the present invention is experimentally performed in the cross-language document retrieval environment of TREC-6. First, the English language query set for cross-language is translated into Korean by hand, and the English language is converted by the Korean-English query conversion method and apparatus of the present invention. In addition, the search was performed using the Smart 11.0 system developed by Cornell University in the United States as an English document search system for evaluating cross-language document search performance of the query method of the present invention. The search results were calculated with 11-point average accuracy by using the original English query and the conversion of Korean query. At this time, the cross-language document search performance effect on the single-language document search reached an average of about 78.6% as shown in the experimental results of FIG. This shows that the cross-language document retrieval performance is improved by about 20% compared to the simple band dictionary method.

As described above, according to the present invention, a high-quality English query can be generated from a Korean-English cross-language document search for searching an English document using a Korean query, and thus a high performance search accuracy can be obtained. In addition, the present invention can be applied to cross-language document search by applying not only Korean-English query language conversion but also query language conversion between various languages, so that richer and more accurate information can be obtained by cross-language document search.

The technical spirit of the present invention has been described above with reference to the accompanying drawings, but this is by way of example only for describing the best embodiment of the present invention and not for limiting the present invention. In addition, it is obvious that any person skilled in the art can make various modifications and imitations without departing from the scope of the technical idea of the present invention.

Claims (7)

In the query converter for cross-language document search, Keyword extracting means for extracting a keyword for searching by querying morphemes and part-of-speech tagging of the source language; Query word converting means for converting a source language keyword extracted by said keyword extracting means into candidate keywords of another language by searching a band dictionary; Computing mutual information from the candidate text corpus of the bandword language for the candidate bandwords generated by the query word converting means, and determining a first band query word for selecting a candidate bandword whose final information is equal to or greater than an arbitrary first threshold as a final query. Way; And Second band query determining means for determining a band query by assigning different weights to words for which the band query is not determined by the first band query determining means according to the size of the mutual information; Query converter comprising a. The method of claim 1, And the first band query word determining means sets the word as the final band query word when the calculated mutual information between the words is about 2.0 or more. The method of claim 1, The second band query word determining means assigns an arbitrary first weight when the mutual information between the words is less than the first threshold and greater than the second threshold, and the mutual information between the words is greater than the second threshold. And a second weighted value if it is smaller than the third threshold. In the query conversion method for cross-language document search, A first step of extracting a keyword for searching by using morpheme analysis and part-of-speech tagging for a query language of a source language; A second step of converting a source language keyword extracted in the first step into candidate keywords of another language by searching a band dictionary; Calculating a mutual information from the search target text corpus of the bandword language with respect to the candidate bandwords converted in the second step, and selecting a candidate bandword whose final information is equal to or greater than an arbitrary first threshold as a final query; And A fourth step of determining a final band query by giving different weights to words not determined in the third step according to the size of the mutual information; Query word conversion method comprising the. The method of claim 4, wherein The fourth step assigns an arbitrary first weight if the mutual information between the words is less than the first threshold and greater than the second threshold, otherwise the mutual information between the words is less than the second threshold and any And if greater than the third threshold, give a final second weight to determine the final band query. The method of claim 5, And wherein the first threshold is about 2.0, the second threshold is about 1.0, the third threshold is about 0, and the first weight is about 0.75 and the second weight is about 0.6. A recording medium having recorded thereon a program for executing a band query conversion function for cross-language document retrieval, A first step of extracting a keyword for searching by using morpheme analysis and part-of-speech tagging for a query language of a source language; A second step of converting the extracted source language keyword into candidate keywords of another language by searching a band dictionary; Calculating a mutual information from candidate text corpus of a bandword language with respect to candidate keywords of the converted other languages, and selecting a candidate bandword whose final information is equal to or greater than an arbitrary first threshold as a final query; And A fourth step of determining a final band query by giving different weights to words not determined in the third step according to the size of the mutual information; A computer-readable recording medium having recorded thereon a program for executing the program.