JP2002259445A - Corresponding category retrieval system and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to easily determine the corresponding relation between categories. SOLUTION: Depending on learning data, a word vector set is produced with a word vector producing means 13. With a document vector producing means 14, the word vector set is referred to and document vector sets in both categories are produced. With a multilingual retrieval means 15, a document pair having close semantic contents is extracted on the basis of the text vector. With a retrieval result holding means 16, the obtained document pair is added to a learning data holding part 12 as a piece of the learning data. The above processing is repeatedly executed. A category corresponding relation determining means 17, for instance, if the ratio of the sum total of the document pairs stored in the retrieval result holding means 16 to the number of total documents included in both categories is greater than a predetermined threshold, determines that both categories have a similarity.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is directed to a category structure in which a document set is classified into a plurality of categories, and determines a category correspondence between the plurality of category structures constructed for different languages. About technology.

[0002]

2. Description of the Related Art As one of methods for facilitating access to a large number of document sets, there is a method of classifying the document sets into a plurality of categories. When the document set is classified into categories, it is possible to efficiently obtain a desired document by performing a search with only the category to which the document desired by the user belongs as a search target.
In some cases, a set of documents is manually categorized into categories.
avid Ellis, Maruzen Co., Ltd., (199
Many techniques for automating categorization as described in "4)" have been proposed.

When such a categorized document set (hereinafter also referred to as a category structure) is constructed for a plurality of languages, the correspondence of the categories between the plurality of category structures (a document having similar semantic contents) Determining the correspondence between categories including a set) is important in performing document search across languages (multilingual document search). That is,
Improve search accuracy by limiting the set of documents in the target language (language used for secondary search) to be searched to categories close to search requests in the source language (language used directly for search) It is possible to do.

As a method for automatically determining the correspondence between such categories, a method of diverting a multilingual document search technique is mainly used. For example, the document "Hirosh
iMasuichi, Raymond Flourno
y, Stefan Kaufmann and Sta
nley Peters, "Query Transl
ation Method for Cross La
ngeage Information Retrie
val ", The Proceedingsof Ma
chine Translation Summit
VII '99 Works on Machi
ne Translation for Cross
Language Information Retr
ieval, (1999) ", a set of translation pairs (parallel corpus) is used as learning data, each document written in a different language is expressed as a document vector in the same vector space, and the cosine value between the vectors is expressed as a document. A method of performing a multilingual document search on the basis of the similarity between them has been proposed. With this method, the sum of the document vectors corresponding to all the documents belonging to the category is defined as a category vector, and the cosine between the category vectors is defined as a similarity. It is possible to determine the correspondence of categories between category structures.

[0005]

However, at present, it is difficult to say that practically sufficient accuracy has been obtained in determining the corresponding category by the above method.
In general, the biggest factor that reduces the search accuracy of multilingual information search is the problem of semantic ambiguity of words or phrases. When translating a word (phrase) in the first language into a word (phrase) in the second language, there are many translation candidates. For example, the word "base" in English is "base" as a military term, "base" as a baseball term, "support matrix" as a political term, "radix" as a mathematical term, and "base" as a chemical term. There are various translation candidates depending on the field, such as "", a grammatical term "period", and an architectural term "a main component of (paint)". Since these translation candidates are often field-dependent, it is said that in multilingual information search, high accuracy can be obtained if the search target is limited to a set of documents in a specific field. That is, there is a translation in the category according to the field of the category, and it is necessary to perform a multilingual document search using an appropriate translation for each field. In the method using the above document vector,
Since one parallel corpus is used as learning data, it is not possible to generate a document vector appropriate for the field, and thus it is not possible to solve the problem of semantic ambiguity. If a parallel corpus can be prepared for each category, it is possible to generate an appropriate document vector according to the field, but in general, it is difficult to obtain a parallel corpus, and in practice such an approach is not possible. is there. Many multilingual document search methods using bilingual dictionaries have been proposed in addition to multilingual document search methods using a parallel corpus as learning data. However, when a general bilingual dictionary is used, semantic ambiguity is obtained. Problem can not be solved,
It is exactly the same that it is practically impossible to prepare a bilingual dictionary for each field (category).

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and has as its object to provide a system capable of determining the correspondence between categories with high accuracy without preparing a parallel corpus for each category. Aim.

[0007]

Means for Solving the Problems Reference "Hiroshi"
Masuichi, Raymond Flourno
y, Stefan Kaufmann and Sta
nley Peters, "A Bootstrapp
ing method for Extracting
Bilingual Text Pairs ", Th
e Proceedings of The 18th
International Conference
on Computational Linguis
tics, pp. 1066-1070 (2000) ", the following bilingual similar document pair determination method is proposed.

[0008] A multilingual document search is performed using a certain parallel corpus as initial learning data, a similar bilingual document pair is determined from a document set in which bilingual documents are mixed, and the obtained document pair is initialized. The multilingual document search is performed again based on the obtained learning data, and the multilingual document search process and the process of adding the obtained document pair to the learning data are alternately repeated.
The learning pairs are grown, and finally a highly accurate parallel corpus (bilingual document pair) is obtained. "

As described in the above document, this method works effectively only when the meanings of the documents in the set of documents to be searched for multilingual documents are similar (in the same field). . The present invention exploits this property of the above approach in reverse. That is, a combination of a first category including a set of documents written in a first language and a second category including a set of documents written in a second language is used as a target of multilingual document search. Is applied, and if the learning pair grows, it is determined that the fields of the first category and the second category are similar.

One configuration of the present invention is, as shown in FIG. 1, composed of a first category structure generated for a first language and a second category structure generated for a second language. A category structure holding unit (1) for holding, a learning data holding unit (2) for holding learning data for performing a multilingual document search, and a category structure using the learning data held in the learning data holding unit. A multilingual document search is performed for the category in the first category and the category in the second category held in the holding unit, and a similar bilingual document pair of the first language and the second language is searched. Multilingual document search means (3) to be determined, and search result holding means (4) for holding a document pair obtained by multilingual document search and adding the document pair to the learning data holding means
And a category correspondence determining means (5) for determining the correspondence between the categories with reference to the document pair held in the search result holding means. Multilingual search processing by
The process of adding the document pair to the learning data by the search result holding means is alternately repeated.

The present invention can be realized not only as an apparatus or a system but also as a method.
Of course, at least a part thereof can be configured as a computer program.

[0012] The above-mentioned configuration of the present invention and other configurations of the present invention are clearly described in the claims, and are described in detail below by using embodiments.

[0013]

Embodiments of the present invention will be described below.

FIG. 2 shows the configuration of the corresponding category search system according to the embodiment of the present invention. In this embodiment, the description is given for Japanese and English, but the same effect can be obtained in any language to which morphological analysis processing (processing of dividing a sentence into words) can be applied. Can be obtained.

In FIG. 2, the category structure holding means 11
Is a means for storing in a computer a category structure (a first category structure and a second category structure) in which a plurality of Japanese documents and a plurality of English documents are classified into categories and stored.

The learning data holding means 12 is a means for holding a set of Japanese-English translation document pairs (Japanese-English parallel corpus) as initial learning data. The parallel corpus is
The field is not particularly limited, and is a general-purpose parallel corpus that is easily available. Search result holding means 1
When a Japanese-English document pair is received from No. 6, it is added to the parallel corpus, which is the initial learning data, and held as a new Japanese-English parallel corpus.

The word vector generating means 13 uses the Japanese-English parallel corpus held in the learning data holding means 12 as learning data, and for all Japanese words and English words contained therein, a corresponding multidimensional vector. (Word vector). Hereinafter, an algorithm for calculating a word vector will be described.

[Step 1]: All Japanese and English documents included in the learning data are subjected to morphological analysis. [Step 2]: Of all the words obtained in Step 1,
In the learning data, n words are selected in descending order of appearance frequency. The n words obtained here are called feature expression words. The value of n is on the order of thousands. [Step 3]: Create matrices whose rows and columns correspond to all the Japanese / English words and feature expression words obtained in step 1, respectively. If the total number of different words of all the Japanese / English words obtained in step 1 is 100,000 and the value of n is 3,000, a matrix of 100,000 rows × 3,000 columns can be formed. For each element of this matrix, how many times the word corresponding to the row of the element and the characteristic expression word corresponding to the column co-occur in all the Japanese-English document translation pairs included in the training data (at the same time) Record whether it appears). That is, the translation pair in Japanese and English is regarded as one document, and the number of co-occurrences in the document is counted. The matrix thus obtained is called a co-occurrence matrix. In this way,
A co-occurrence matrix that represents all Japanese words and all English words by n-dimensional vectors can be created. This vector is
This can be said to be a vector indicating in what context each word tends to appear.

[Step 4]: n obtained in step 3
Since a dimensional vector has a large number of dimensions, the processing required later requires an enormous amount of calculation time. Therefore,
In order to suppress the calculation process in the real-time range, the original n-dimensional vector is compressed into an n′-dimensional (hundreds of dimensions) vector by a matrix dimension compression method. There are various dimensional compression methods, but “Berry, M., Do,
T. O'Brien, G .; Krishna, V .;
and Varadhan, S.M. , "SVDPACKC
USER'S GUIDE ".Tech.Rep.C
S-93-194. University of Te
nnessee, Knoxville, TN (199
Singular, which is described in detail in 3)
Value Decomposition is a typical example. The n′-dimensional vector obtained for all Japanese words and English words in this manner is called a word vector.

The document vector generation means 14 uses the word vectors obtained by the word vector generation means 13 to convert all Japanese documents in category A and all English documents in category B held in the category structure holding means 11. Is a means for calculating a document vector corresponding to. First, all Japanese documents in category A and all English documents in category B held in the category structure holding means 11 are subjected to morphological analysis to divide them into words. Next, a vector (where the length of the vector is 1) obtained by normalizing the sum of the word vectors corresponding to all the words included in each document is calculated,
Let the obtained vector be a document vector. However, a word whose corresponding word vector has not been generated by the word vector generating means 13 is ignored.

The multilingual search means 15 includes an arbitrary Japanese category (category A) in the first category and an arbitrary English category (category B) in the second category held in the category structure holding means 11. Is a means of searching for a similar Japanese-English document pair from the category pairs. Therefore, it is assumed that the following processing is performed for all the category pairs of the Japanese category and the English category (any combination of category A and category B).

First, a pair of a Japanese document and an English document satisfying the following conditions is extracted from all the document sets belonging to category A and category B by referring to the document vector obtained from the document vector generating means 14. .

"The English document vector having the highest degree of relevance (having a large inner product value) to the document vector corresponding to the Japanese document in the document pair is the English document vector in the pair, and conversely the English document vector in the pair. The most relevant Japanese document vector is the Japanese document vector in the pair. "

Next, of the Japanese-English document pairs satisfying the above conditions, a pair in which the value of the inner product between the Japanese-English document vectors corresponding to the Japanese-English documents in the pair is larger than a preset threshold value is extracted. The Japanese-English document pair obtained in this way has very close semantic contents, and can be used as learning data.

The search result holding means 16 is a means for holding a set of Japanese-English document pairs obtained from the multilingual search means 15 for the categories A and B in the computer. The obtained document pair set is passed to the learning data holding unit 12 as a part of new learning data.

As described above, (1) a word vector set is generated by the word vector generating means 13 based on the learning data held in the learning data holding means 12,
(2) A document vector set is generated by the document vector generation means 14, (3) Japanese-English document pairs having similar meanings are extracted by the multilingual search means 15, and (4) Search result holding means 16 obtains the document pairs. The added document pair is added to the learning data holding unit 12 as a part of the learning data (if it has already been added, replace it with the previous one). By repeatedly performing the processing described above, only when the semantic contents of the Japanese document set in the category A and the English document set in the category B are close (the category A and the category B belong to the same field), the search result holding unit 16 The number of document pairs held therein will gradually increase.

The category correspondence determination unit 17 refers to the ratio of the “total number of documents included in the category A and the category B” of “the total number of document pairs held in the search result holding unit 16”, and If it is larger than the predetermined threshold T, it is determined that the category A and the category B are similar (in the same field) category pairs. If the ratio does not exceed the threshold value T even though the above-described repetition processing has been performed on the category A and the category B for a certain number of times or more, the category A and the category B are similar (of the same field). Decide not a category pair.

The determination of the category correspondence may be completed only by one document search. Further, the threshold value may be set in multiple stages. For example, a threshold a (a
If it is less than <b), it is determined to be non-corresponding. If it is equal to or more than the threshold a and less than the threshold b, the document search is repeated again, and the same determination is performed. A configuration for making a determination may be employed. in short,
When the document search result indicates that the correspondence between the categories is positive, any configuration may be adopted as long as it is determined that the categories correspond.

With such a configuration, the category correspondence determining means 17 determines whether or not there is a corresponding relation for all of the category pairs of the Japanese category and the English category (arbitrary combination of category A and category B). By doing so, it is possible to comprehensively determine the correspondence between the categories of the first category and the second category.

In this embodiment, the above-mentioned document "Hiro"
shi Masuichi, Raymond Flo
urnoy, Stefan Kaufmann an
dStanley Peters, “Query Tr
announcement Method for Cros
s Language InformationRet
rieval ", The Proceedings o
f Machine Translation Sum
mit VII'99 Workshopon Mac
hine Translation for Cros
s Language Information Re
trivalent, (1999), a multilingual document search method based on a vector space method using a parallel corpus as learning data is used, but a multilingual document search that can repeatedly add a search result to the learning data is used. A similar effect can be obtained by any method (see FIG. 1).

For example, a document written in a first language in category A is translated into a second language by a machine translation system, and a multilingual document is searched using a general monolingual document search technique. Similar effects can be obtained by a search method.

As an example of realizing a machine translation system using a parallel corpus as learning data, a document “Peter
F. Brown, Stephen A .; Della P
ietra, Vincent J .; Della Pie
tra, and Robert L. Mercer, "
The materials of statuses
physical Machine Transatio
n: Parameter estimation ", Co
Mputational Linguistics, 3
2: 263-311, 1993. ].

[0033]

As described above, according to the present invention, the problem of word ambiguity resolution can be avoided without preparing learning data for each field, and a plurality of category structures constructed for different languages can be used. The correspondence between categories can be determined with high accuracy.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a typical corresponding category search system according to the present invention.

FIG. 2 is a diagram showing a configuration of a corresponding category search system according to one embodiment of the present invention.

[Explanation of symbols]

 11 Category structure holding means 12 Learning data holding means 13 Word vector generating means 14 Document vector generating means 15 Multilingual search means 16 Search result holding means 17 Category correspondence determining means

Claims (6)

[Claims] 1. A first language generated for a first language.
Category structure holding means for holding a category structure of the second language and a second category structure generated for the second language, learning data holding means for holding learning data when performing a multilingual document search, and learning data. Using the learning data held in the holding means, a multilingual document search is performed on the categories in the first category structure and the categories in the second category structure held in the category structure holding means, Similar first
Multilingual document search means for determining a bilingual document pair of the second language and the second language, and a search result for holding the document pair obtained by the multilingual document search and adding the document pair to the learning data holding means A correspondence category search system comprising: a holding unit; and a category correspondence determining unit that determines a correspondence between categories by referring to a document pair held in the search result holding unit. 2. A first language generated for a first language.
Category structure holding means for holding the category structure of the first language and the second category structure generated for the second language, and learning data holding means for holding a translation document pair as learning data when performing a multilingual document search And using the learning data held in the learning data holding means to target the categories in the first category structure and the categories in the second category structure held in the category structure holding means by the vector space method. A multilingual document search means for performing a multilingual document search based on the first language and a similar bilingual document pair of a second language and a document pair obtained by the multilingual document search; A search result holding unit that adds a document pair to the learning data holding unit, and a correspondence between categories by referring to the document pair held in the search result holding unit. And a category correspondence determining means for determining. 3. A first language generated for a first language.
Category structure holding means for holding the category structure of the second language and a second category structure generated for the second language, and learning data holding means for holding a translation document pair as learning data when performing a multilingual document search. Using the learning data held in the learning data holding means to convert a document written in a first language belonging to a category in the first category structure held in the category structure holding means into a second language Translated and obtained translated document set and second
A multilingual document search means for performing a document search for a set of documents belonging to a category in the category structure of the first language and determining a similar bilingual document pair of a first language and a second language; A search result holding unit that holds the obtained document pair and adds the document pair to the learning data holding unit; and a category that determines the correspondence between the categories by referring to the document pair held in the search result holding unit. And a correspondence relation determining means. 4. A first language generated for a first language.
A category structure holding step of storing a category structure of the second language and a second category structure generated for the second language; a learning data holding step of storing learning data when performing a multilingual document search; Using the learning data stored in the holding step, a multilingual document search is performed on the categories in the first category structure and the categories in the second category structure stored in the category structure step to find similarities. A multilingual document search step for determining a bilingual document pair of the first language and the second language; holding the document pair obtained in the multilingual document search step; and adding the document pair as learning data Search result holding step and the document pair stored in the search result holding step A corresponding category determining step of determining a corresponding relationship. 5. A first language generated for a first language.
A category structure holding step of storing a category structure of the second language and a second category structure generated for the second language; a learning data holding step of storing learning data when performing a multilingual document search; Using the learning data stored in the holding step, a multilingual document search is performed on the categories in the first category structure and the categories in the second category structure stored in the category structure step to find similarities. A multilingual document search step for determining a bilingual document pair of the first language and the second language; holding the document pair obtained in the multilingual document search step; and adding the document pair as learning data Search result holding step and the document pair stored in the search result holding step And a category correspondence determining step of determining a correspondence. 6. A first language generated for a first language.
Category structure holding means for holding a category structure of the second language and a second category structure generated for the second language, learning data holding means for holding learning data when performing a multilingual document search, and learning data. Using the learning data held in the holding means, a multilingual document search is performed on the categories in the first category structure and the categories in the second category structure held in the category structure holding means, Similar first
Multilingual document search means for determining a bilingual document pair of the second language and the second language; category correspondence determining means for determining correspondence between categories based on the document pair obtained by the multilingual document search; A corresponding category search system comprising: