JP2010009237A - Multi-language similar document retrieval device, method and program, and computer-readable recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform similarity retrieval even to a retrieval target document written in a natural language different from an inputted similarity-retrieving keyword or similarity-retrieving document. <P>SOLUTION: This multi-language similar document retrieval device previously calculates a concept vector of a word to n (n≥2) pieces of natural languages in each natural language, and stores it into a word concept base. The multi-language similar document retrieval device previously calculates a concept vector of a retrieval target document to a retrieval target document group by use of the word concept base to the natural language written with the retrieval target document, and stores it into a retrieval target document concept base. When an input key is inputted, the multi-language similar document retrieval device translates it into another natural language, creates a translation key, calculates an input document translation concept vector to the created translation key by use of the word concept base to each natural language, compares the input translation concept vector and the concept vector of the retrieval target document of the retrieval target document concept base to each natural language, and calculates similarity. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a similar document search technique that makes it possible to search a search target document similar to an input keyword or document, and in particular, a natural language different from the input similar search keyword or similar search document. The present invention relates to a multilingual similar document search apparatus and method, a program, and a computer-readable recording medium that can perform a similar search even on a search target document written in the above.

  Conventional techniques related to similarity search across languages include the following.

  (1) Synonyms are developed for keywords described in a certain natural language, and each search target document is searched using the translation result as a query.

(2) A text corpus of a natural language is placed in a multi-dimensional space based on the tendency of co-occurrence relationships between all words in both languages using a language parallel corpus consisting of a translation into another natural language. To do. The similarity of words in different languages is calculated based on the similarity (distance) in the multidimensional space (see, for example, Non-Patent Document 1). Based on the similarity between words, a similarity search between multiple languages is realized.
Masuichi, H., Flournoy, R., Kaufmann, S., and Peters, S., "A Bootstrapping method for Extracting Bilingual Text Pairs", Coling 2000, pp.1066-1070 (2000).

  However, the above conventional techniques have the following problems.

  A: In the above method (1), similar documents for each language can be obtained, but the similarity between them cannot be calculated.

  B: In the method (2), the similarity across languages can be calculated, but a parallel corpus written in a plurality of languages must be prepared, and a large number of man-hours are required for preparing the corpus.

  The present invention has been made in view of the above points, and is capable of performing a similar search even for a search target document written in a natural language different from the input similar search keyword or similar search document. It is an object of the present invention to provide an inter-similar document retrieval apparatus and method, a program, and a computer-readable recording medium.

  FIG. 1 is a principle configuration diagram of the present invention.

The present invention (Claim 1) is a multilingual similar document search apparatus for obtaining a similarity in a case where languages are different between an input search key and a search target document,
For n (n ≧ 2) natural languages,
A word concept base storage processing means 1 for calculating a concept vector of a word for each natural language and storing it in the word concept base storage means 8;
Using the word concept base storage unit 8 for the natural language in which the search target document is written, the concept vector of the search target document is calculated for the search target document group and stored in the search target document concept base storage unit 10. Search target document concept base storage processing means 2 for storing;
A word group or document (hereinafter referred to as “input key”) 11 used for a search written in a certain natural language is translated into another natural language, and the word group or document translated into the other natural language (hereinafter, referred to as “input key”). Machine translation means 3 for creating “translation key”),
An input document translation concept vector calculation means 4 for calculating an input document translation concept vector using the word concept base storage means 8 for each natural language for the translation key created by the input key 11 and the machine translation means 3 ,
Similarity for calculating similarity by comparing the input translation concept vector obtained by the input document translation concept vector calculation means 4 with the concept vector of the search target document in the search target document concept base storage means 10 for each natural language Degree calculation means 5.

  The present invention (Claim 2) further includes similar document display means for displaying search target documents in descending order of similarity calculated by the similarity calculation means 5.

  Further, the present invention (Claim 3) includes translation means for translating the result in a natural language different from the input into the same natural language in the similar document display means 5.

  FIG. 2 is a diagram for explaining the principle of the present invention.

The present invention (Claim 4) is a multilingual similar document search method for obtaining a similarity when the input search key and the search target document have different languages,
For n (n ≧ 2) natural languages,
A word concept base storage processing unit that calculates a word concept vector in advance for each natural language and stores it in the word concept base storage unit (step 1);
The retrieval target document concept base storage processing unit calculates a concept vector of the retrieval target document in advance using the word concept base storage unit for the natural language in which the retrieval target document is written for the retrieval target document group. Then, the word concept base and the search target document concept base are constructed in advance by two steps of the search target document concept base storage step (step 2).
When a word group or a document (hereinafter referred to as “input key”) used for a search written in a natural language is input, the machine translation means translates the input key into another natural language, A machine translation step (step 3) for creating a word group or a document (hereinafter referred to as a “translation key”) translated into a natural language;
The input document translation concept vector calculation means calculates the input document translation concept vector using the word concept base storage means for each natural language for the input key and the translation key created in the machine translation step (step 3). Input document translation concept vector calculation step (step 4),
The similarity calculation means compares the input document translation concept vector obtained in the input document translation concept vector calculation step (step 4) with the concept vector of the search target document in the search target document concept base storage means for each natural language. Then, the similarity calculation step (step 5) for calculating the similarity is performed.

  The present invention (Claim 5) further performs a similar document display step of displaying search target documents in descending order of similarity calculated in the similarity calculation step (Step 5).

  The present invention (Claim 6) further includes a translation step of translating the result in a natural language different from the input into the same natural language in the similar document display step.

  The present invention (Claim 7) is a multilingual similar document search program for causing a computer to function as each means constituting the multilingual similar document search apparatus according to any one of claims 1 to 3. .

  The present invention (Claim 8) is a computer-readable recording medium storing the multilingual similar document search program according to Claim 7.

  As described above, according to the present invention, the following effects can be obtained.

  A: Unlike the conventional method (1), the similarity between languages can be calculated on the same scale by using the similarity based on the concept vector.

  B: Instead of a bilingual corpus as in the conventional technique (2), it is sufficient to prepare a corpus for each language, and a large man-hour is not required for preparing the corpus.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows a configuration of a multilingual similar document search apparatus according to an embodiment of the present invention.

  The multilingual similar document search apparatus shown in FIG. 1 includes a word concept base storage processing unit 1, a search target document concept base storage processing unit 2, a machine translation unit 3, an input document translation concept vector calculation unit 4, and a similarity calculation unit 5. , A similar document display unit 6, a plurality of word concept bases 8, and a plurality of search target document concept bases 10, and in terms of hardware, a CPU and a memory.

  The word concept base storage processing unit 1 has a corpus 7 written in one of n natural languages ("-i" in the figure represents a natural language being described. For example, "-1" represents Japan The word concept vector of each natural language is calculated and stored in the word concept base 8. It is assumed that the corpus 7 input to the word concept base storage processing unit 1 is read in the word concept base storage processing unit 1 in a database format.

  The search target document concept base storage processing unit 2 refers to the search target document 9 written in any one of the n natural languages and refers to the word concept base 8 of each natural language to search target document concept vectors. Is stored in the search target document concept base 10. The input search target document 9 is stored in the database format and is read by the search target document concept base storage processing unit 2.

  The machine translation unit 3 translates the input key 11 which is a word group or a document used for the search input by the user into another natural language, and creates a translation key. For example, if the input is in Japanese, it is translated into another natural language such as English or French, and the obtained result is used as a translation key.

  The input document translation concept vector calculation unit 4 uses the word concept base 8 for each natural language with respect to the translation key for the input key 11 obtained by the input key 11 and the machine translation unit 3, and uses the input document translation concept vector. Calculate

  The similarity calculation unit 5 compares the input document translation concept vector obtained by the input document translation concept vector calculation unit 4 with the search target document concept base 10 for each natural language, and calculates the similarity.

  The similar document display unit 6 displays the search target document 9 in descending order of similarity calculated by the similarity calculation unit 5.

  The word concept base 8 stores a word concept vector for every n natural languages.

  The search target document 9 is a document for each of n natural languages.

  The search target document concept base 10 stores a search target document vector for every n natural languages.

  The operation in the above configuration will be described below.

  FIG. 4 is a flowchart of the operation in one embodiment of the present invention.

  In the following description, it is assumed that the word concept base 8 for n natural languages has been constructed in advance, and that the search target document concept base 10 for n natural languages has also been constructed using these.

  The construction method of the word concept base 8 and the search target document concept base 10 is not particularly limited. For example, Reference 1 “Schutze,“ Dimensions of Meaning ”, Proceedings of Supercomputing 92, pp. 787-796”, or Reference 2 “Kasahara, Matsuzawa, Ishikawa,“ Difference of everyday words using a Japanese language dictionary, ”Information Processing Society Journal, Vol. 38, No. 7, pp. 1272-1284 (1997). Can be constructed.

  Step 101) The machine translation unit 3 translates the input key 11 which is a word group or a document described in a certain language into (n-1) other natural languages, and creates a translation key. Further, the input key and the translation key are sent to the input document translation concept vector calculation unit 4.

  Step 102) The input document translation concept vector calculation unit 4 calculates the concept vector of the input key using the word concept base 8 constructed from the same natural language as the input key 11.

  Step 103) Further, the input document concept vector calculation unit 4 calculates the concept vector of the translation key for all the translation keys using the word concept base 8 constructed from the same natural language as each natural language.

  Step 104) The input document translation concept vector calculation unit 4 divides the processing depending on whether or not the input key and the translation key are words.

  Step 105) When the input key and the translation key are words, the obtained concept vector is set as an input document translation concept vector for each description language.

  Step 106) When the input key and the translation key are a word group or a document, the centroid of the concept vector group obtained for each description language is calculated and set as an input document translation concept vector for each description language.

  Step 107) The obtained input document translation concept vector for each description language is sent to the similarity calculation unit 5. The similarity calculation unit 5 receives input document translation concept vectors for n natural languages.

  Step 108) The similarity calculation unit 5 calculates the similarity between the input document translation concept vector for the input key 11 and all search target document concept vectors in the search target document concept base 10 of the corresponding natural language. .

  Step 109) The similarity calculation unit 5 calculates the similarity between the input document translation concept vector corresponding to the translation key and all search target document concept vectors in the search target document concept base 10 corresponding to each natural language. To do.

  Step 110) Then, the similarity calculation unit 5 sends the calculation results obtained in steps 108 and 109 to the similar document display unit 6. That is, a calculation result corresponding to each similarity degree is sent to the similar document display unit 6 for n natural languages ((n-1) natural languages of the natural language translation key of the input key).

  Step 111) In the similar document display unit 6, the calculation results sent from the similarity calculation unit 5 are temporarily stored in a memory (not shown) and sorted in descending order of the similarity, regardless of the type of natural language. The concept vectors of the search target document concept base 10 are rearranged in descending order of similarity. Further, the corresponding document is extracted from the search target document 9 described in the corresponding natural language and displayed.

  At this time, as shown in FIG. 5, a similar document machine translation unit 62 is provided in the similar document display unit 6, and machine translation is performed on (n-1) natural language results different from the input. , It may be displayed after being converted to the same natural language as the input.

  The operation of the constituent elements of the multilingual document search apparatus 13 shown in FIG. 3 is constructed as a program and installed in a computer used as the multilingual document search apparatus, or is executed or via a network. It can be distributed.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a similar search technique for searching for a document similar to a keyword or a document, in particular, a similar search technique between multiple languages.

It is a principle block diagram of this invention. It is a figure for demonstrating the principle of this invention. It is a block diagram of the multilingual similar document search apparatus in one embodiment of this invention. It is a flowchart of the operation | movement in one embodiment of this invention. It is a structural example of the similar document display part in one embodiment of this invention.

Explanation of symbols

1 word concept base storage processing means, word concept base storage processing section 2 search target document concept base storage processing means, search target document concept base storage processing section 3 machine translation means, machine translation section 4 input document translation concept vector calculation means, input Document translation concept vector calculation unit 5 Similarity calculation unit, similarity calculation unit 6 Similar document display unit 7 Corpus 8 Word concept base storage unit, word concept base storage unit 9 Search target document 10 Search target document concept base storage unit, search target Document concept base 11 Input key 12 Cross-language similarity document search result 13 Multi-language similar document search device 61 Similar document display control unit 62 Similar document machine translation unit

Claims (8)

A multi-language similar document search device for obtaining a similarity in a case where a language is different between an input search key and a search target document,
For n (n ≧ 2) natural languages,
A word concept base storage processing means for calculating a word concept vector for each natural language and storing it in a word concept base storage means;
For the search target document group, the concept vector of the search target document is calculated using the word concept base storage means for the natural language in which the search target document is written, and stored in the search target document concept base storage means. Search target document concept base storage processing means,
A word group or a document (hereinafter referred to as “input key”) used for a search written in a certain natural language is translated into another natural language, and the word group or document (hereinafter, “ A machine translation means for creating a translation key),
An input document translation concept vector calculation means for calculating an input document translation concept vector using a word concept base storage means for each natural language for the translation key created by the input key and the machine translation means,
Similarity calculation that calculates the similarity by comparing the input translation concept vector obtained by the input document translation concept vector calculation means with the concept vector of the search target document in the search target document concept base storage means for each natural language Means,
A multilingual similar document search apparatus characterized by comprising: The multilingual similar document search device according to claim 1, further comprising similar document display means for displaying search target documents in descending order of similarity calculated by the similarity calculation means. Similar document display means
The multilingual similar document search device according to claim 2, further comprising a translation unit that translates a result in a natural language different from the input into the same natural language as the input. A multi-language similar document search method for obtaining a similarity in a case where a language is different between an input search key and a search target document,
For n (n ≧ 2) natural languages,
A word concept base storage processing unit that calculates a word concept vector in advance for each natural language and stores it in the word concept base storage unit;
The retrieval target document concept base storage processing unit calculates a concept vector of the retrieval target document in advance using the word concept base storage unit for the natural language in which the retrieval target document is written for the retrieval target document group. The word concept base and the search target concept base are constructed in advance by two steps of the search target document concept base storage processing step stored in the search target document concept base storage means,
When a word group or a document (hereinafter referred to as “input key”) used for a search written in a natural language is input, the machine translation means translates the input key into another natural language, A machine translation step for creating a word group or document (hereinafter referred to as a “translation key”) translated into a natural language of
The input document translation concept vector calculation means calculates the input document translation concept vector using the word concept base storage means for each natural language for the input key and the translation key created in the machine translation step. Concept vector calculation step and
The similarity calculation means compares the input translation concept vector obtained in the input document translation concept vector calculation step with the concept vector of the search target document in the search target document concept base storage means for each natural language. A similarity calculation step for calculating
A method for retrieving similar documents between multiple languages. 5. The multilingual similar document search method according to claim 4, further comprising a similar document display step of displaying search target documents in descending order of similarity calculated in the similarity calculation step. In the similar document display step,
6. The multilingual similar document search method according to claim 5, further comprising a translation step of translating a result in a natural language different from the input into the same natural language as the input.   A multilingual similar document search program for causing a computer to function as each means constituting the multilingual similar document search device according to any one of claims 1 to 3.   A computer-readable recording medium storing the multilingual similar document search program according to claim 7.

Images