JP2017091192A - Method and device for learning between documents in different languages using images, and method and device for searching cross-lingual document - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for learning between documents in different languages that indirectly uses images.SOLUTION: A first training data set including a pair group of a first language document and an image is prepared, and a second training data set including a pair group of a second language document and an image is prepared. In the first training data set, a first feature vector and a second feature vector are extracted from the first language document and from the image, respectively. In the second training data set, a third feature vector and the second feature vector are extracted from the second language document and from the image, respectively. The first, second, and third feature vectors are then used for conducting a generalization canonical correlation analysis so that the first and third feature vectors are mapped using the second feature vector.SELECTED DRAWING: Figure 3

Description

The present invention relates to cross-language document retrieval.

Cross-language document search or cross-language document search is a technique for inputting a Japanese document and searching for related / similar English documents. In the prior art, a large amount of bilingual corpora (for example, a document set written in both Japanese and English) is required to learn the system, and it is generally difficult to obtain such data itself. Therefore, feasibility was poor.

Specifically, manual translation of a large amount of documents requires a great deal of labor. It is also possible to crawling a bilingual document from the Web and using it as learning data, but many documents on the Web are closed to one language. Therefore, collecting a sufficient amount of multilingual documents is not easy, especially for minor languages.

Therefore, we focused on multimedia information that exists abundantly on the Web in recent years, especially documents and image pairs. If two documents written in different languages both contain images and the image features are similar, it can be assumed that the text contained in the documents will also be similar. Moreover, in addition to being able to understand the semantic content contained in the image regardless of the mother tongue, the image has the advantage of being a universal expression since the document can be included in any national document. .

Machine learning in traditional image recognition is very poor compared to machine learning in the field of natural language processing. However, in recent years, the accuracy of image recognition is rapidly approaching the human level due to the breakthrough of deep learning (non-patent document 12).
Douglas J Carroll. 1968. Generalization of canonical correlation analysis to three or more sets of variables.In Proceedings of the 76th Annual Convention of the American Psychological Association, volume 3, pages 227-228. Jon Robers Kettenring. 1971. Canonical Analysis of Several Sets of Variables. Biometrika, 58 (3): 433-451. Michel Velden and Yoshio Takane. 2012.Generalized Canonical Correlation Analysis with Missing Values.Computational Statistics, 27 (3): 551-571. Jan Rupnik, Andrej Muhic, and Primo Skraba. 2012. Cross-Lingual Document Retrieval through Hub Languages. In Neural Information Processing Systems Workshop. Harold Hotelling. 1936. Relations between Two Sets of Variants. Biometrika, 28: 321-377. David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004. Canonical Correlation Analysis: an Overview with Application to Learning Methods. Neural Computation, 16 (12): 2639-2664. Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. 2010. A New Approach to Cross-modal Multimedia Retrieval.Proceedings of the International Conference on Multimedia, pages 251-260. Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. 2014.A Multiview Embedding Space for Modeling Internet Images, Tags, and their Semantics.International Journal of Computer Vision, 106 (2): 210-233. Alexei Vinokourov, John Shawe-Taylor, and Nello Cristianini. 2002. Inferring a Semantic Representation of Text via Cross-Language Correlation Analysis. Advances in Neural Information Processing Systems, pages 1473-1480. Yaoyong Li and John Shawe-Taylor. 2004. Using KCCA for Japanese-English Cross-language Information Retrieval and Classification.In Learning Methods for Text Understanding Raghavendra Udupa and Mitesh M Khapra. 2010. Improving the Multilingual User Experience of Wikipedia Using Cross-Language Name Search.Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 492-500. Hao Fang, Saurabh Gupta, Forrest Iandola, K. Rupesh Srivastava, Li Deng, Piotr Doll´ar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. 2015. From Captions to Visual Concepts and Back.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the Devil in the Details: Delving Deep into Convolutional Nets. In Proceedings of the British Machine Vision Conference, pages 1-11. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2013.DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition.In Proceedings of the International Conference on Machine Learning, pages 647-655 . Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014.Caffe: Convolutional Architecture for Fast Feature Embedding. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision. Christian Szegedy, Scott Reed, Pierre Sermanet, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going deeper with convolutions. CoRRabs / 1409.4842. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, pages 1097-1105. Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying Conditional Random Fields to Japanese Morphological Analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 230-237. Cyrus Rashtchian, Peter Young, Micah Hodosh, Julia Hockenmaier, and North Goodwin Ave. 2010.Collecting Image Annotations Using Amazon's Mechanical Turk.Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 139-147. Florent Perronnin, Jorge S´anchez, and Thomas Mensink. 2010. Improving the Fisher kernel for large-scale image classification.In Proceedings of the European Conference on Computer Vision, pages 143-156. David G Lowe. 1999. Object recognition from local scale-invariant features.In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 2, pages 1150-1157.

An object of the present invention is to propose a learning method and apparatus between different language documents using an image indirectly, and to perform a document search between different languages using the learning model.

The learning method between different language documents mediated by images according to the present invention is as follows:
Prepare a first training data set consisting of pairs of first language documents and images,
Prepare a second training data set consisting of pairs of second language documents and images,
In the first training data set, a first feature vector is extracted from the first language document, a second feature vector is extracted from the image,
In the second training data set, a third feature vector is extracted from the second language document, a second feature vector is extracted from the image,
Performing generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, thereby mapping the first feature vector and the third feature vector through the second feature vector;
Is.

In one aspect, the first feature vector and the third feature vector are extracted using Bag of words.
Bag of words is one representative example for extracting a feature amount of a document, and is not limited to this.

In one aspect, the second feature vector is extracted using a convolutional neural network.
Examples of CNN include AlexNet, CaffeNet, GoogLeNet, and VGG net.

In one aspect, the first feature vector, the second feature vector, and the third feature vector are dimensionally reduced by the dimension reduction means.
A typical example of the dimension reduction means is principal component analysis (PCA).

In one aspect, the first training data set includes multimedia data obtained by crawling from a web in a first language;
The second training data set includes multimedia data acquired by crawling from the Web in the second language.

In one aspect, a third training data set comprising a pair group of a first language document and a second language document is further prepared,
Extracting a first feature vector from text in a first language, extracting a third feature vector from text in a second language in a third training data set;
In the generalized canonical correlation analysis, the first feature vector and the third feature vector extracted from the third training data set are further used.
This mode corresponds to Few-Shot learning described later.

A learning device between different language documents mediated by an image according to the present invention,
A first training data set consisting of pairs of first language documents and images;
A second training data set consisting of pairs of second language documents and images;
First feature vector extraction means for extracting a first feature vector from a first language document;
Second feature vector extraction means for extracting a second feature vector from the image;
Third feature vector extraction means for extracting a third feature vector from the second language document;
Generalized canonical correlation analysis means;
With
The generalized canonical correlation analysis means performs the generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, and thereby the first feature vector Mapping the third feature vector;
Is.

The cross-language document search method according to the present invention uses a learning model obtained by the learning method between the different language documents,
In the learning model, the first projection coefficient from the first language space (the “first feature vector” space) to the canonical space, and the second language space (the “third feature vector” space) to the canonical space. A second projection coefficient is defined,
Extracting a first feature vector from a first language query document;
Projecting the extracted first feature vector onto a canonical space using the first projection coefficient to obtain a first projected feature vector;
Extracting a third feature vector from the second language target document candidate;
Projecting the extracted third feature vector onto the canonical space using the second projection coefficient to obtain a third projected feature vector;
The target document is determined using the similarity between the first projection feature vector and the third projection feature vector.

The cross-language document search device according to the present invention uses a learning model obtained by the learning method between the different language documents,
In the learning model, a first projection coefficient from the first language space to the canonical space and a second projection coefficient from the second language space to the canonical space are defined,
Means for extracting a first feature vector from a first language query document;
Means for projecting the extracted first feature vector into a canonical space using a first projection coefficient to obtain a first projected feature vector;
Means for extracting a third feature vector from the second language target document candidate;
Means for projecting the extracted third feature vector onto a canonical space using a second projection coefficient to obtain a third projected feature vector;
Means for determining a target document using the similarity between the first projected feature vector and the third projected feature vector;
It has.

In the present invention, learning between different language documents can be performed even when there is no bilingual corpus by mediating images (Zero-shot learning).
In the present invention, learning between different language documents can be performed even when there are few parallel corpora by mediating images (Few-shot learning).

In a study between different language documents, a bilingual corpus is also required in research (Non-Patent Document 4) that enables learning by using a document such as English as a hub.
Information across two languages is needed.
On the other hand, the present invention is characterized in that learning can be performed only with information closed in one language on the premise that multimedia information is attached (see FIG. 1).
Since the image is a universal expression, it is included in a document for each language (a document closed in one language) and plays a role as a bridge (mediation) in learning.

  The present invention focuses on multimedia information that has been abundant on the web in recent years, and by using a learning method that indirectly uses images, in combination with recent image recognition technology breakthroughs, Cross-language document retrieval is implemented without a corpus (or with a small bilingual corpus).

It is a conceptual diagram of image-mediated learning. A web document containing image data is shown. 1 is a schematic diagram of a learning system between different language documents mediated by images according to the present invention. 1 is a schematic diagram of a cross-language document search system according to the present invention. It is a schematic diagram of the learning system between different language documents concerning one embodiment of the present invention. It is a conceptual diagram which shows the projection to the canonical space of the feature vector of a Japanese document, the feature vector of an English document, and the feature vector of an image using generalized canonical correlation analysis. It is a conceptual diagram of the nearest neighbor search in the projected space (canonical space). It is a figure which illustrates the data set used for experiment. Five English sentences and five Japanese sentences corresponding to each English sentence are prepared for one image. A feature vector extracted from five Japanese sentences and a feature vector extracted from one image form a pair, and a feature vector extracted from five English sentences and a feature vector extracted from one image form a pair. Forming. The result of search accuracy experiment 1 is shown. Search accuracy experiment 1 was conducted by changing the number of samples in [train-E / I] and [train-I / J], and also changing the number of samples in [train-E / J]. Went. The result of the search accuracy experiment 2 is shown. Search accuracy experiment 2 is a comparative experiment using a plurality of types of image features.

[A] Outline of Image-Mediated Learning System An outline of the image-mediated learning system according to the present invention will be described with reference to FIG.
[A-1] Data used for learning First, a first training data set and a second training data set are prepared. The first training data set includes a pair group of a first language document and an image. The second training data set is composed of a pair group of a second language document and an image. In the experimental example to be described later, Japanese and English are used as the first language and the second language, but the first language and the second language can be selected from any different languages. In one aspect, completely different images are used for the images included in the first training data set and the images included in the second training data set so that no overlap occurs. It is not excluded that the same image is included.

In the first training data set, a first feature vector is extracted from the first language document, and a second feature vector is extracted from the image. In the second training data set, a third feature vector is extracted from the second language document, and a second feature vector is extracted from the image. Learning is performed using these feature vectors. The feature vectors extracted from the images are collectively referred to as the second feature vector. However, the second feature vector obtained from the first training data set is different from the second feature vector obtained from the second training data set. .

In the examples described later, UIUC Pascal Sentence Dataset (Non-Patent Document 19) is used, but various data can be considered as the source of the training data set. In one aspect, the first training data set includes multimedia data obtained by crawling from a first language web, and the second training data set is obtained by crawling from a second language web. Contains media data. SNS such as Twitter, news articles, blog articles, etc. have abundant data with multimedia data and can be obtained by crawling. The training data set is not limited to data acquired from the Web. For example, a pair of image data and document data may be acquired by scanning a paper medium (book, advertisement flyer) including an image such as a photograph and a document. Alternatively, a text / multimedia pair may be acquired from video and subtitles of a television program.

[A-2] Text Feature Extraction In one aspect, the first feature vector and the third feature vector are extracted using Bag of words (BoW). BoW is one typical example for extracting the feature amount of a document, and the following methods are exemplified as non-limiting methods other than BoW.
TF-IDF features;
Using word vectors obtained by Word2vec, adding all the words in the document and averaging them;
Document vector obtained from Paragraph2vec;
Features of N-gram language model;
BoW reduced in dimension using LSI (Latent Semantic Indexing, LSA: also called Latent Semantic Analysis);
BoW reduced in dimension by LDA (Latent Dirichlet Allocation).

[A-3] Image Feature Extraction In one aspect, the second feature vector is extracted using a convolutional neural network (CNN). The convolutional neural network (CNN) is deep learning that is most successful as image recognition (Non-Patent Documents 12 to 17). Examples of CNN include AlexNet, CaffeNet, GoogLeNet, and VGG net. CNN has various structures depending on the designer. CNN updates the weights that combine the layers so that classification is done correctly when learning. That is, weights are determined in the learned neural network, and when an image is input to the neural network, the value of the activation function is calculated based on the respective weights. The result is used as a feature value. There is no limitation on which layer value of the plurality of layers is used. It is understood by those skilled in the art that the best layer can be selected in experiments and the like, although the layer near the final layer of the network is generally considered to give good characteristics. The second feature vector that can be used in the present invention does not exclude feature quantities other than CNN, and includes, for example, a Fisher vector, Bag of Features (Bag of Visual Words), and the like.

[A-4] Dimensional reduction of feature vectors In one aspect, the first feature vector, the second feature vector, and the third feature vector may be dimensionally reduced by a dimensional reduction means. The dimension reduction of each feature vector is arbitrary, but considering the calculation time, it is desirable to perform dimension reduction when the data size is large. Typically, principal component analysis (PCA) is exemplified as the dimension reduction means, but independent component analysis (ICA), LDA, or LSI may be used.

[A-5] Generalized canonical correlation analysis (GCCA)
Generalized canonical correlation analysis is a canonical correlation analysis that handles two variable groups, generalized to handle more than two variable groups, and maximizes the sum of correlations between multiple modalities. Data can be mapped to By performing the generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, the first feature vector and the third feature vector are mapped using the second feature vector as a medium.

Learning using canonical correlation analysis (CCA) that handles two variable groups and its application to CLDR are known (Non-Patent Documents 5 to 11). GCCA is a generalized CCA for m modalities (m = 3 in this embodiment).

Typical examples of GCCA include those described in Non-Patent Document 1 (Carroll), Non-Patent Document 2 (Kettenring), Non-Patent Document 3 (Velden et al), and Non-Patent Document 4 (Rupnik et al). Are known. In Non-Patent Document 4, the technique of Non-Patent Document 2 (partly referred to in Non-Patent Document 1) is adopted. In the experiment described later, the GCCA of Non-Patent Document 2 was adopted, but it will be understood by those skilled in the art that other GCCA can be adopted to realize the present invention.

[A-6] Image-mediated cross-language learning model
GCCA determines a first projection coefficient from the first language space to the canonical space and a second projection coefficient from the second language space to the canonical space. That is, in the learning model obtained by the learning method between different language documents, the first projection coefficient from the first language space to the canonical space and the second projection coefficient from the second language space to the canonical space are defined. Yes. The first feature vector extracted from the first language document is converted into the first projected feature vector by the first projection coefficient. The third feature vector extracted from the second language document is converted into a third projected feature vector by the second projection coefficient. The first projected feature vector and the third projected feature vector can be compared in a canonical space (in a broad sense, “latent space”) that is a common space or joint space.

[A-7] Few-Shot Learning One ideal form of the present invention is Zero-Shot learning for realizing cross-language document search without any bilingual corpus, but using a small amount of bilingual corpus. Few-Shot learning may be performed. In this case, a third training data set including a pair group of the first language document and the second language document is prepared, and the first feature vector is extracted from the text of the first language in the third training data set, and the second language In the generalized canonical correlation analysis, the first feature vector and the third feature vector extracted from the third training data set are further used.

[A-8] Hardware Configuration The learning system according to the present invention includes one or a plurality of computers, and the computer includes processing means (CPU, etc.) as hardware, storage means (hard disk, RAM, ROM, etc.), input means, output means or display means, and a control program for operating the computer as software. A first training data set consisting of a pair group of a first language document and an image and a second training data set consisting of a pair group of a second language document and an image are stored in the storage means. The means for extracting features from the text data and the means for extracting features from the image data comprise processing means. The first feature vector extracted from the first language document, the second feature vector extracted from the image, and the third feature vector extracted from the second language document are stored in the storage means. The generalized canonical correlation analysis is executed by the processing means using the first feature vector, the second feature vector, and the third feature vector. By performing the generalized canonical correlation analysis, the second feature vector is used as a medium. Mapping of the first feature vector and the third feature vector is performed. Specifically, a first projection coefficient from the first language space to the canonical space and a second projection coefficient from the second language space to the canonical space are calculated, and the projection coefficients are stored in the storage means.

[B] Overview of Cross-Language Document Search System An overview of a cross-language document search system according to the present invention will be described with reference to FIG.
[B-1] Image-mediated interlingual learning model In the learning model obtained by the learning method between different language documents, the first projection coefficient from the first language space to the canonical space, the canonical from the second language space A second projection coefficient to space is defined. The first feature vector extracted from the first language document is converted into the first projected feature vector by the first projection coefficient. The third feature vector extracted from the second language document is converted into a third projected feature vector by the second projection coefficient. The similarity between the first language document and the second language document can be estimated from the similarity between the first projection feature vector and the third projection feature vector.

[B-2] When a search first language query document is input to the search system, a first feature vector is extracted from the input text data. The extracted first feature vector is projected onto the canonical space using the first projection coefficient to obtain the first projected feature vector.

On the other hand, a third feature vector is extracted from the second language target document candidate, and the extracted third feature vector is projected onto the canonical space using the second projection coefficient to obtain a third projected feature vector. A third projection feature vector corresponding to each second language document is extracted in advance and stored in the storage unit, and the similarity described below is calculated using the third projection feature vector stored in advance. May be.

The target document is determined using the similarity between the first projected feature vector and the second projected feature vector. The similarity is typically represented by a distance between vectors, and examples include a Eugrid distance, a Mahalanobis distance, and a Manhattan distance. Further, the cosine similarity may be used as the similarity. Typically, the candidate having the highest similarity is output as the second language target document. Alternatively, a plurality of candidates having a high degree of similarity may be output as the second language target document, and may be ranked and displayed according to the degree of similarity.

How to set the second language target document candidate is not particularly limited. In one aspect, all second language documents available at the time the first language query document is input are candidates. For example, all data acquired by crawling from the second language Web may be targeted.

[B-3] Hardware Configuration The search system according to the present invention includes one or a plurality of computers. The computer includes processing means (CPU, etc.) as hardware, storage means (hard disk, RAM, ROM, etc.), input means, output means or display means, and a control program for operating the computer as software. The user terminal is also composed of one or a plurality of computers, and the computer includes processing means, storage means, input means, output means or display means, a control program for operating the computer, and the like.

The search system and the user terminal are provided with transmission / reception means that can exchange information with each other via a computer network represented by the Internet, and are connected to each other via a computer network represented by the Internet. Yes. The search system is connected to an existing search engine via a computer network represented by the Internet. For example, a query screen is displayed on the screen of the user terminal, text data in the first language is input from the input means of the user terminal, and is transmitted to the search system as a search query. In addition, instead of text data input, document upload may be used, and in the case of a recommendation system that automatically extracts and recommends similar sentences on the search system side, there is no user side interaction and the search result Is displayed on the user terminal. In one aspect, a plurality of second languages can be selected, and one or a plurality of second languages are designated. In the search system, the similarity with the second language target document candidate is calculated based on the search query, and the search result can be viewed from the user terminal.

[C] Examples
[C-1] Data to be used In this embodiment, the different data division shown in Table 1 is used.
[train-E / I]: Learning document consisting of pairs of English text and images
[train-I / J]: Learning document consisting of pairs of Japanese text and images
[train-E / J]: Learning document consisting of pairs of English text and Japanese text
[test-E / J]: Test document consisting of pairs of English text and Japanese text
Each data division does not overlap. For example, the image data in [train-E / I] is different from the image data in [train-I / J].

As shown in Table 1, an ID is defined for each modality of each division. For example, E1 represents an English feature in the [train-E / I] division. A typical CLDR based on a parallel corpora uses only [train-E / J] as learning data and evaluates it using [test-E / J]. In the Zero-Shot learning scenario that does not use [train-E / J] data according to this experiment, only [train-E / I] and [train-I / J] are used as learning data. In the Few-Shot learning scenario, a few [train-E / J] samples are used. In this specification, these learnings are collectively referred to as image-mediated learning.

[C-2] System Overview Diagram FIG. 5 shows a system overview according to the embodiment. Learning is performed using three feature quantities: English, images, and Japanese. In learning, the first feature vector is extracted from English text, the second feature vector is extracted from the image, the third feature vector is extracted from Japanese, and the obtained features are dimensionally reduced by principal component analysis (PCA). Learn about reduced and reduced features by GCCA.

In the test, the features obtained from the query Japanese text are dimensionally reduced by principal component analysis (PCA). Note that, as indicated by the arrows in FIG. 5, the coefficients projected in a low dimension by the PCA projection are learned by the PCA in the learning phase. The reduced features are projected using the first projection coefficient obtained by GCCA, while the features obtained from the English text are dimensionally reduced by principal component analysis (PCA), and the reduced features are GCCA. Projection is performed using the second projection coefficient obtained by the above, and the nearest neighbor search from Japanese to English is performed in the joint space.

[C-3] Image Feature Extraction Image features are extracted using a convolutional neural network. In this embodiment, learning is performed in advance using an ILSVRC2012 dataset (Non-patent Document 15), and a CNN model provided in Caffe (Non-Patent Document 16) is applied. In the experiment, the features of the pool5 / 7x7 s1 layer of GoogLeNet (Non-Patent Document 17), the features of the fc6 layer of VGG (Non-Patent Document 13), and the fc6 layer of CaffeNet (Non-Patent Document 16, Non-Patent Document 18) The feature quantity was used as an image feature vector.

[C-4] Text Features Weighting by bag of words (BoW) and TF-IDF (term frequency-inverse document frequency) was used as English and Japanese text features. A MeCab library is used in dividing Japanese sentences into words by morphological analysis (Non-patent Document 18). In the experiment, no pre-processing such as stop word deletion or stemming was performed, but these are optional.

[C-5] Generalized canonical correlation analysis (GCCA)
By using GCCA, data can be mapped so as to maximize the sum of correlations between a plurality of modalities (see FIG. 6). In this example, GCCA of Non-Patent Document 2 was adopted. The calculation of GCCA itself is known, and the GCCA described below is an example, and does not limit the GCCA used in the present invention.
If E, I, and J are English, images, and Japanese, respectively, the feature vector
In
Represents a canonical variable.
Here, X _k bar is an average of feature vectors.
h _k is a projection coefficient.
k∈ {E, I, J}, and canonical variables (projection feature vectors) can be calculated from each feature vector and the corresponding projection coefficient.

GCCA is a maximization problem
The constraint
Equation (1) is derived by solving under
The projection coefficient h _k is calculated to maximize the total correlation of each pair of modalities obtained by solving the following generalized eigenvalue problem.
here,
And
Σ _ij is a covariance matrix of modalities i and j, and i, jε {E, I, J}. Σ _EJ is calculated by E ₃ and J ₃ , Σ _IJ is calculated by I ₂ and J ₂ , Σ _II is calculated by I ₁ and I ₂ , and Σ _JJ is calculated by J ₂ and J ₃ . Σ _EJ differs from Σ _EI and Σ _IJ , especially in the case of Zero-Shot learning, the number of training samples may be 0, in which case it does not contribute to the above maximization problem, so Σ _EJ Filled with zeros.

The canonical axis is standardized as follows.
A regularization term is added to prevent overlearning. That is,
And α is a regularization parameter.

[C-5] Nearest neighbor search in joint space When a query document in the first language is given, in order to search related documents in the second language, which is another language, the query document and the candidate document are searched in the joint space. What is necessary is just to calculate the distance. The feature vector in the joint space (projective feature vector) is
And h _k is obtained by GCCA.

For example, if the query document is Japanese and the target document is English, the nearest neighbor in the joint space can be calculated by the following equation.
Here, z ⁱ _E and z ^j _J are projected feature vectors of the target document and the query document, respectively, and d (•) is a distance function. In this embodiment, the distance function is the Eugrid distance.

[D] Experiment
[D-1] Data set used in the experiment
UIUC Pascal Sentence Dataset (Non-Patent Document 19) has 1000 images with 5 English annotations each describing the content. This data set was created for the study of document generation from images, but in order to use it for the image-mediated CLDR according to the present embodiment, a Japanese translation corresponding to each English sentence was prepared ( (See FIG. 8). In this experiment, five sentences corresponding to each image are treated as one text data. Therefore, in this setup, each document of the data set consisting of 1000 documents consists of one image, corresponding English text, and Japanese text. Each symbol in the conceptual diagram of FIG. 6 represents “a feature vector extracted from five Japanese sentences”, “a feature vector extracted from one image”, and “a feature vector extracted from five English sentences”. Yes.

[D-2] Data was randomly extracted from each data division in Evaluation Table 1 so as not to overlap. The experiment was performed by changing the sample size of [train-E / I] and [train-I / J]. Specifically, the number of samples was 100, 200, 300, and 400. Furthermore, the number of [train-E / J] samples was gradually increased from 0 to 100 to create a Few-Shot learning scenario. The size of the test data [test-E / J] was set to 100.
According to this setup, GCCA-based image-mediated CLDR was performed and compared with the results of conventional CLDR based on CCA using only [train-E / J] data.

The performance was evaluated for the first Japanese to English retrieval accuracy in the test data. When 100 test samples are given, the chance rate is 1%. In each trial, 50 trials were performed while changing data at random, and an average score was used. All features were reduced to 100 dimensions by PCA and α was set to 0.01.

As shown in FIG. 9, the experimental results show that the accuracy improves as the amount of text-image data increases in both the Zero-Shot learning scenario and the Few-Shot learning scenario. It is considered that further improvement in accuracy can be expected by increasing the amount of text-image data.

The results of Zero-Shot learning scenario (Zero-Shot learning accuracy) are summarized in Table 3. Image features were extracted by GoogLeNet, and text features used bag-of-words (BoW) and TF-IDF.
As shown in Fig. 9, in both GCCA and CCA, the performance increases as the sample size of [train-E / J] increases, but it can be expected that the sample size is in English text and Japanese text. When it becomes large enough to learn directly between texts, the performance of CCA exceeds that of GCCA. However, when the amount of [train-E / J] data is small, it exceeds the GCCA CCA baseline, and therefore image-mediated learning is useful even in the Zero-Shot learning scenario.

[D-3] Effect of Image Feature The effect of the image feature performance in this embodiment was verified (FIG. 10, Table 4). Three different features extracted using the following CNN were used as image features.
Features of the pool5 / 7x7 s1 layer of GoogLeNet (Non-Patent Document 17)
VGG (Non-Patent Document 13) fc6 layer features,
Features of fc6 layer of CaffeNet (Non-patent document 16, Non-patent document 18)

Furthermore, the Fisher vector (Non-patent Document 20), which was widely used before image feature extraction using deep learning, was also tested. For the Fisher vector, the SIFT descriptor (Non-Patent Document 21) was reduced to 64 dimensions by principal component analysis, and a mixed Gaussian distribution using 64 elements was used. Four spatial grids were used for final feature extraction.

Table 4 shows the accuracy of Zero-Shot learning using a plurality of image features. The sample size of [train-E / I] and [train-I / J] is 400. Text features were bag-of-words (BoW) and TF-IDF.
The order of accuracy when using each image feature is the order of known performance for the image features used, specifically GoogLeNet → VGG net → CaffeNet → FisherVector
This is consistent with the fact that the recognition accuracy is high in this order (Non-patent Document 12). The same ranking in the image-mediated CLDR means that a higher search accuracy can be obtained in the image-mediated CLDR if a better feature amount is used.

Claims (14)

Prepare a first training data set consisting of pairs of first language documents and images,
Prepare a second training data set consisting of pairs of second language documents and images,
In the first training data set, a first feature vector is extracted from the first language document, a second feature vector is extracted from the image,
In the second training data set, a third feature vector is extracted from the second language document, a second feature vector is extracted from the image,
Performing generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, thereby mapping the first feature vector and the third feature vector through the second feature vector;
A learning method between different language documents via images.   The learning method according to claim 1, wherein the first feature vector and the third feature vector are extracted using Bag of words.   The learning method according to claim 1, wherein the second feature vector is extracted using a convolutional neural network.   The learning method according to claim 1, wherein the first feature vector, the second feature vector, and the third feature vector are dimensionally reduced. The first training data set includes multimedia data obtained by crawling from the web in a first language;
The second training data set includes multimedia data acquired by crawling from the Web in a second language.
The learning method according to claim 1. Furthermore, a third training data set consisting of a pair group of the first language document and the second language document is prepared,
Extracting a first feature vector from text in a first language, extracting a third feature vector from text in a second language in a third training data set;
In the generalized canonical correlation analysis, a first feature vector and a third feature vector extracted from a third training data set are further used.
The learning method according to claim 1. A first training data set consisting of pairs of first language documents and images;
A second training data set consisting of pairs of second language documents and images;
First feature vector extraction means for extracting a first feature vector from a first language document;
Second feature vector extraction means for extracting a second feature vector from the image;
Third feature vector extraction means for extracting a third feature vector from the second language document;
Generalized canonical correlation analysis means;
With
The generalized canonical correlation analysis means performs the generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, and thereby the first feature vector Mapping the third feature vector;
A learning device between different language documents via images.   The learning apparatus according to claim 7, wherein the first feature vector extraction unit and the third feature vector extraction unit acquire Bag of words.   The learning apparatus according to claim 7, wherein the second feature vector extraction unit is a convolutional neural network. The learning device includes principal component analysis means,
The learning device according to claim 7, wherein the first feature vector, the second feature vector, and the third feature vector are dimensionally reduced by a dimension reduction means. The first training data set includes multimedia data obtained by crawling from the web in a first language;
The second training data set includes multimedia data acquired by crawling from the Web in a second language.
The learning device according to claim 7. And a third training data set comprising a pair group of the first language document and the second language document,
In the generalized canonical correlation analysis, a first feature vector and a third feature vector extracted from a third training data set are further used.
The learning device according to claim 7. A cross-language document search method using a learning model obtained by a learning method between different language documents according to any one of claims 1 to 6,
In the learning model, a first projection coefficient from the first language space to the canonical space and a second projection coefficient from the second language space to the canonical space are defined,
Extracting a first feature vector from a first language query document;
Projecting the extracted first feature vector onto a canonical space using the first projection coefficient to obtain a first projected feature vector;
Extracting a third feature vector from the second language target document candidate;
Projecting the extracted third feature vector onto the canonical space using the second projection coefficient to obtain a third projected feature vector;
Determining a target document using the similarity between the first and third projected feature vectors;
Cross-language document retrieval method. A cross-language document search apparatus using a learning model obtained by a learning method between different language documents according to any one of claims 1 to 6,
In the learning model, a first projection coefficient from the first language space to the canonical space and a second projection coefficient from the second language space to the canonical space are defined,
Means for extracting a first feature vector from a first language query document;
Means for projecting the extracted first feature vector into a canonical space using a first projection coefficient to obtain a first projected feature vector;
Means for extracting a third feature vector from the second language target document candidate;
Means for projecting the extracted third feature vector onto a canonical space using a second projection coefficient to obtain a third projected feature vector;
Means for determining a target document using the similarity between the first projected feature vector and the third projected feature vector;
A cross-language document search device.