JP2013105393A - Video additional information relationship learning device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video additional information relationship learning device which uses both image information and acoustic information included in a video and considers their mutual co-occurrence relation, thereby highly accurately learning relationship between those information and linguistic information.SOLUTION: In a video additional information relationship learning device, an accumulation image feature extraction unit 5 extracts a complete accumulation image feature and an incomplete accumulation image feature from each of a complete accumulation video and an incomplete accumulation video. An accumulation acoustic feature extraction unit 6 extracts a complete accumulation acoustic feature and an incomplete accumulation acoustic feature from each of the complete accumulation video and the incomplete accumulation video. An accumulation additional information feature extraction unit 4 extracts an accumulation additional information feature from each of the accumulation additional information. An accumulation latent variable extraction unit 8 extracts an accumulation latent variable group which is a group of variables for describing relationship between the video and the additional information. An inter-video/additional information relationship learning unit 9 learns a video/additional information relation model which is a model for describing relationship between the video and the additional information.

Description

  The present invention relates to a video additional information relationship learning apparatus, method, and program.

  Video search technology for searching for desired video based on given language information, and video recognition technology for automatically assigning language information describing the video to the given video are digital cameras, mobile phones, etc. With the widespread use of such imaging devices and the generalization of video sharing on the Internet, it has become a very important technology. Video can be considered as a time series of still images, and many technologies for video search and video recognition use image search and image recognition technology after dividing the video into still images (for example, Non-Patent Document 1).

  In recent years, a number of technologies such as Non-Patent Document 2 have been developed for image recognition and retrieval technology that realizes image retrieval and image recognition under the same framework.

  Furthermore, by developing these technologies, even when images that are not associated with linguistic information cannot be prepared sufficiently, the relationship between the images and linguistic information is properly learned to realize high-accuracy image recognition and retrieval. Technology has also been developed (Non-Patent Document 3).

Sivic, J. and Zisserman, A. “Video Google: A Text Retrieval Approach to Object Matching in Videos,” Proceedings of the International Conference on Computer Vision (2003) Nakayama, Harada, Kuniyoshi, Otsu “Ultra-high-speed image recognition / retrieval method using probabilistic structure learning corresponding to image / word concept”, IEICE Technical Report, PRMU2007-147, December 2007 Kimura, Nakano, Sugiyama, Kameoka, Maeda, Sakano “SSCDE: Semi-supervised canonical density estimation method for image recognition retrieval”, Proceedings of Symposium on Image Recognition and Understanding, OS8-1, July 2010

  These technologies realize video recognition and retrieval by using only image information. However, usually, an audio signal is also given to the video. There are a number of events in which this audio signal is very important for understanding the content of video. For example, not only events that are revealed only by observing acoustic information such as “singing” and “cheers”, but also acoustic information such as “soccer goal” and “explosion” There are cases where the details of the event become clearer by being observed.

  The present invention has been made in consideration of such circumstances. The purpose is to use both image information and audio information included in the video, and to learn the relationship between the information and language information with higher accuracy in consideration of their mutual co-occurrence relationships. It is an object to provide a video additional information relationship learning apparatus, method, and program capable of performing the above.

  In order to achieve the above object, a video additional information relationship learning apparatus according to the present invention learns the relationship between a video that is a moving image with sound and additional information that is information that describes the video. A relationship learning device that is a complete stored video that is an element of a fully stored video set that is a set of videos to which additional information is given in advance, and an incompletely stored video set that is a set of videos to which no additional information is given And a stored image feature extracting means for extracting a completely stored image feature and an incompletely stored image feature, which are vectors expressing the characteristics of the image, from each of the incompletely stored video as elements of the image, and an element of the complete stored video set Extraction of complete accumulated sound features and incompletely accumulated sound features, which are vectors representing acoustic characteristics, from each of the completely accumulated image and the incompletely accumulated image that is an element of the incompletely accumulated image set. The storage acoustic feature extraction means for extracting the stored additional information features that are vectors representing the characteristics of the additional information from each of the stored additional information that is an element of the stored additional information set that is a set of the added additional information Additional information feature extraction means, a completely stored image feature set that is a set of the completely stored image features, an incompletely stored image feature set that is a set of the incompletely stored image features, and a complete storage that is a set of the completely stored acoustic features To describe the relationship between video and additional information from an acoustic feature set, an incompletely stored acoustic feature set that is a set of the incompletely stored acoustic features, and a stored additional information feature set that is a set of the stored additional information features Storage latent variable extraction means for extracting a stored latent variable set that is a set of variables, the completely stored image feature set, the incompletely stored image feature set, and the completely stored acoustic feature Video / additional information relation model that is a model describing the relationship between video and additional information from the incompletely stored acoustic feature set, the stored additional information feature set, and the stored latent variable set. Additional information relationship learning means.

  The video additional information relationship learning method according to the present invention is a video used in a video additional information relationship learning device that learns the relationship between a video that is a moving image with sound and additional information that is information that describes the video. An additional information relationship learning method, wherein the accumulated image feature extraction means is a completely accumulated image that is an element of a completely accumulated image set that is a set of images to which additional information is given in advance, and an image for which no additional information is given Extracting a completely accumulated image feature and an imperfectly accumulated image feature, which are vectors representing the characteristics of the image, from each of the incompletely accumulated images that are elements of the incompletely accumulated image set that is a set of images, and accumulated acoustic feature extraction By means of the method, the acoustic characteristics are obtained from each of the completely accumulated image that is an element of the completely accumulated image set and the incompletely accumulated image that is an element of the incompletely accumulated image set. A step of extracting a completely accumulated acoustic feature and an incompletely accumulated acoustic feature which are present vectors, and a stored additional information feature which is a set of additional information added by a stored additional information feature extracting unit; A step of extracting a stored additional information feature, which is a vector expressing the characteristic of the additional information, and a stored latent variable extracting means, and a complete stored image feature set, which is a set of the completely stored image features, and the incompletely stored image An incompletely stored image feature set that is a set of features, a fully stored acoustic feature set that is a set of the completely stored acoustic features, an incompletely stored acoustic feature set that is a set of the incompletely stored acoustic features, and the stored additional information feature From the stored additional information feature set, which is a set of, a latent latent variable set, which is a set of variables for describing the relationship between video and additional information, is extracted And the step of learning the relationship between video and additional information, the complete stored image feature set, the incompletely stored image feature set, the complete stored acoustic feature set, the incompletely stored acoustic feature set, and the stored additional information feature Learning a video / additional information relationship model, which is a model describing the relationship between video and additional information, from the set and the accumulated latent variable set.

  According to the present invention, the accumulated image feature extraction means is a set of a completely accumulated image that is an element of a completely accumulated image set that is a set of images to which additional information is given in advance, and a set of images to which no additional information is given. A fully accumulated image feature and an incompletely accumulated image feature, which are vectors representing the characteristics of the image, are extracted from each of the incompletely accumulated images that are elements of the incompletely accumulated image set. Complete accumulation, which is a vector that expresses acoustic characteristics from each of the completely accumulated video, which is an element of the completely accumulated video set, and the incompletely accumulated video, which is an element of the incompletely accumulated video set, by the accumulated acoustic feature extraction means Extract acoustic features and imperfectly accumulated acoustic features.

  Then, the storage additional information feature extraction means extracts a storage additional information feature that is a vector representing the characteristic of the additional information from each of the storage additional information that is an element of the storage additional information set that is a set of the added additional information. To do. By the accumulated latent variable extraction means, a completely accumulated image feature set that is a set of the completely accumulated image features, an incompletely accumulated image feature set that is a set of the incompletely accumulated image features, and a complete accumulation that is a set of the completely accumulated acoustic features. To describe the relationship between video and additional information from an acoustic feature set, an incompletely stored acoustic feature set that is a set of the incompletely stored acoustic features, and a stored additional information feature set that is a set of the stored additional information features An accumulated latent variable set that is a set of variables is extracted.

  Then, by the relationship learning means between video and additional information, the complete stored image feature set, the incomplete stored image feature set, the complete stored acoustic feature set, the incomplete stored acoustic feature set, the stored additional information feature set, The video / additional information relationship model, which is a model describing the relationship between the video and the additional information, is learned from the accumulated latent variable set.

  As described above, the completely accumulated image feature and the completely accumulated acoustic feature extracted from each of the completely accumulated images, the incompletely accumulated image feature and the incompletely accumulated acoustic feature extracted from each of the incompletely accumulated images, and the accumulated latent variable. By learning a model that describes the relationship between video and additional information from the set, using both image information and acoustic information contained in the video, and taking into account their mutual co-occurrence relationship, The relationship between the information and the language information can be learned with higher accuracy.

  The program according to the present invention is a program for causing a computer to function as each unit of the video additional information relationship learning apparatus.

  As described above, according to the video additional information relationship learning apparatus, method, and program of the present invention, the completely accumulated image feature and the completely accumulated acoustic feature extracted from each of the completely accumulated image, and the incompletely accumulated image By learning a model that describes the relationship between video and additional information from incompletely stored image features and incompletely stored acoustic features extracted from each, and a set of stored latent variables, image information contained in the video By using both of the acoustic information and taking into account the mutual co-occurrence relationship, it is possible to learn the relationship between the information and the linguistic information with higher accuracy.

It is a block diagram which shows the example of 1 structure of the image | video additional information relationship learning apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the example of 1 structure of the accumulation latent variable extraction part of the image | video additional information relationship learning apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the model learning process routine in the image | video additional information relationship learning apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the example of 1 structure of the semi-supervised video search apparatus based on the 2nd Embodiment of this invention. It is a flowchart which shows the content of the video search processing routine in the semi-supervised video search apparatus based on the 2nd Embodiment of this invention. It is a block diagram which shows the example of 1 structure of the semi-teacher image | video recognition apparatus which concerns on the 3rd Embodiment of this invention. It is a flowchart which shows the content of the video recognition process routine in the semi-teacher video recognition apparatus which concerns on the 3rd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
<System configuration>
FIG. 1 is a block diagram showing a video additional information relationship learning apparatus 100 according to the first embodiment of the present invention. The video additional information relationship learning apparatus 100 is configured to store a complete accumulated video set that is a set of video (image signal + acoustic signal) to which additional information, which is information describing the video, is given in advance, and a video to which no additional information is given. A video / additional information relation model, which is a model that describes the relationship between video and additional information, by inputting an incompletely stored video set that is a set and a storage additional information set that is a set of additional information attached to the video. An output device, specifically, a computer comprising a CPU (Central Processing Unit), a RAM, and a ROM storing a program for executing a model learning processing routine to be described later. The configuration is as follows.

  The video additional information relationship learning device 100 includes an input unit 10, a calculation unit 20, and an output unit 30.

  The input unit 10 adds a complete stored video set that is a set of videos to which additional information, which is information describing the video, is added in advance, an incompletely stored video set that is a set of videos to which no additional information is given, and video An input of a stored additional information set that is a set of added additional information is received.

  The calculation unit 20 includes a storage additional information database 1, a complete storage video database 2, an incomplete storage video database 3, a storage additional information feature extraction unit 4, a storage image feature extraction unit 5, a storage acoustic feature extraction unit 6, a feature database 7, A storage latent variable extraction unit 8 and a video / additional information relationship learning unit 9 are provided.

  The accumulated additional information database 1 stores the inputted accumulated additional information set. The fully stored video database 2 stores the input complete stored video set. The incompletely stored video database 3 stores the input incompletely stored video set.

  The stored image feature extraction unit 5 inputs the complete stored video set and the incomplete stored video set, and inputs the complete stored video that is an element of the complete stored video set and the incomplete stored video that is an element of the incomplete stored video set. From each of them, a completely accumulated image feature that is a vector expressing the characteristics of the image signal included in each accumulated video and an incompletely accumulated image feature are extracted, and a complete accumulated image feature set that is a set of completely accumulated image features, An incompletely stored image feature set that is a set of completely stored image features is output.

  The extraction method of the completely accumulated image feature and the incompletely accumulated image feature is not particularly limited. For example, the complete accumulated image and the incompletely accumulated image of each frame of the image signal that is a component of the completely accumulated image are extracted. A method of extracting features from each of the incompletely stored images, which are each frame of the image signal as a constituent element, by the following method and any combination thereof can be considered.

・ Color histogram ・ Low frequency component of digital cosine transform of each small area in the image ・ Histogram of low frequency and / or high frequency component of Haar Wavelet ・ High-order local autocorrelation feature (Ref. 1 “N. Otsu and T. Kurita (See “A new scheme for practical flexible and intelligent vision systems,” Proc. IAPR Workshop on Computer Vision, pp.431-435, 1988.)
・ SIFT (see Reference 2 “D. Lowe,“ Distinctive image features from scale-invariant keypoints, ”International Journal of Computer Vision, Vol. 60, No. 2, pp. 91-110, 2004.) Improvement / Bag of Features (Reference 3 “G. Csurka, C. Bray, C. Dance and L. Fan“ Visual categorization with bags of keypoints, ”in Proc. Of ECCV Workshop on Statistical Learning in Computer Vision, pp. 59 -74, 2004.)

  In addition, it is possible to adopt a form in which a histogram created by using the same method as the Bag of Features, which is a feature extracted from each stored image included in the section corresponding to the stored video, by the above method is used as the stored image feature. (Reference 4 “K. Kashino, T. Kurozumi and H. Murase:“ A quick search method for audio and video signals based on histogram pruning ”, IEEE Transactions on Multimedia, Vol.5, No.3, pp.348- 357, 2003 ").

  In this manner, the accumulated image feature extraction unit 5 extracts the completely accumulated image feature and the incompletely accumulated image feature, and outputs a complete accumulated image feature set and an incompletely accumulated image feature set, which are their respective sets.

  The stored acoustic feature extraction unit 6 inputs the complete stored video set and the incomplete stored video set, and the complete stored video that is an element of the complete stored video set and the incomplete stored video that is an element of the incomplete stored video set. From each of them, a completely accumulated acoustic feature that is a vector expressing the characteristics of the acoustic signal included in each accumulated video and an incompletely accumulated acoustic feature are extracted, and a complete accumulated acoustic feature set that is a set of completely accumulated acoustic features, and a non-accumulated acoustic feature set. An incompletely stored acoustic feature set that is a set of completely stored acoustic features is output.

  The extraction method of the completely accumulated acoustic feature and the incompletely accumulated acoustic feature is not particularly limited. For example, an analysis window is applied to the acoustic signal as a component of each accumulated video, and the following method is used from each analysis window. And a method of extracting features by any combination thereof.

Mel frequency cepstrum coefficient (see Reference 5 “J. Foote“ Content-based retrieval of music and audio, ”In Multimedia Storage and Archiving Systems II, Proc. Of SPIE, volume 3229, pages 138-147, 1997.)
Delta cepstrum (Reference 6 “S. Furui,“ Speaker independent isolated word recognition using dynamic features speech spectrum, ”IEEE Transactions on Acoustics, Speech and Signal Processing, Vol. 34, No. 1, pp. 52-59, 1986 ."reference)
・ Bandpass filter bank (Ref. 7, “Sagano, Smith, Murase“ Fast acoustic signal search method using histogram features-Time-series active search method ”, IEICE Transactions, Vol.J82-D2, No. 9, pp.1365-1373, 1998 ")

  In addition, the stored acoustic feature extraction unit 6, similar to the embodiment shown in the stored image feature extraction unit 5, stores the histogram extracted from the features extracted by the above method using the same method as the Bag of Features. A form adopted as a feature is also possible.

  In this way, the accumulated acoustic feature extraction unit 6 extracts the completely accumulated acoustic feature and the incompletely accumulated acoustic feature, and outputs a complete accumulated acoustic feature set and an incompletely accumulated acoustic feature set, which are their respective sets.

  The stored additional information feature extraction unit 4 inputs a stored additional information set, extracts a stored additional information feature that is a vector expressing the characteristics of the additional information from each of the stored additional information that is an element of the stored additional information set, A stored additional information feature set that is a set of the stored additional information features is output.

  The method of extracting the accumulated additional information feature is not particularly limited, but in the present embodiment, a language label is assumed as the additional information, and a binary vector representing the presence or absence of the language label is accumulated. And That is, the accumulated additional information feature is configured as follows.

  The accumulated additional information feature is a vector having the same number of dimensions as the total number of language labels to be considered, and each dimension of the vector corresponds to a language label. Hereinafter, for convenience, the language label is expressed using an index of a corresponding dimension in the vector. If the accumulated additional information includes the i-th language label, the i-th dimension of the accumulated additional information feature is set to “1”. Otherwise, “0” is set. Alternatively, a multidimensional vector obtained by compressing the feature created by the above method using principal component analysis may be used as the accumulated additional information feature.

  In this way, the accumulated additional information feature extraction unit 4 extracts the accumulated additional information feature and outputs an accumulated additional information feature set that is a set of the accumulated additional information features.

  The feature database 7 stores the extracted completely accumulated image feature set, incompletely accumulated image feature set, completely accumulated acoustic feature set, incompletely accumulated acoustic feature set, and accumulated additional information feature set.

  The stored latent variable extraction unit 8 inputs from the feature database 7 a complete stored image feature set, an incomplete stored image feature set, a complete stored acoustic feature set, an incomplete stored acoustic feature set, and a stored additional information feature set. From the feature set, an accumulated latent variable set that is a set of variables for describing the relationship between the video and the additional information is extracted, and this accumulated latent variable set is output.

The stored latent variable set is a case where the image vector, sound vector, and additional information vector belonging to the same video are mapped to the space with the latent variable set by some method when each of the image, sound, and additional information is given as a vector. In order to be described with the same latent variable. Thus, the latent variable relates the image, sound, and additional information.
In reality, it is difficult for the values at the mapping destinations of each vector to completely match, so the image, sound, and additional information vectors are the mapping that maximizes the correlation between the combined vector, the image, An objective function such as a mapping that minimizes the square error of the latent variable at the point where each vector of acoustic and additional information is mapped can be calculated and calculated using optimization methods such as Lagrange's undetermined coefficient method and gradient method. .
At this time, a large number of sets of images, sounds, and additional information that are generally known to have been obtained from the same video cannot be obtained. Therefore, it is expected that the calculated map will be inaccurate. The present invention compensates for this problem by using incomplete accumulation features and accurately estimating the overall density of the entire latent variable space.
The method for extracting the accumulated latent variable set is not particularly limited, but in the present embodiment, the following method is used in which canonical correlation analysis, which is a kind of multivariate analysis, is improved.

  As shown in FIG. 2, the accumulated latent variable extraction unit 8 includes a complete accumulation feature set statistic calculation unit 81, an incomplete accumulation feature set statistic calculation unit 82, an integrated statistic calculation unit 83, and a feature compression function determination. Unit 84 and feature compression unit 85.

The completely accumulated feature set statistic calculation unit 81 is a complete accumulated feature set (completely accumulated image feature set X) that is a set of features that a complete accumulated image feature, a completely accumulated acoustic feature, and a corresponding accumulated additional information feature form a set. _C0 , a completely stored acoustic feature set X _C1 , a stored additional information feature set Y _C ) are input, a completely stored feature set statistic that is a statistic representing this completely stored feature set is calculated, and this completely stored feature set statistics Output quantity.

  The calculation method of the completely accumulated feature set statistics is not particularly limited, but in this embodiment, the self-covariance matrix and the mutual covariance matrix of the completely accumulated feature set are calculated as the completely accumulated feature set statistics. To do.

Here, the symbols necessary for the specific description of the method will be described. As shown in the following equation, the completely stored image feature set is described as X _C0 , the incompletely stored image feature set is described as X _I0, and the stored image feature set that is the union thereof is described as X ₀ . Similarly, a complete stored acoustic feature set is described as X _C1 , an incomplete stored acoustic feature set is described as X _I1, and a stored acoustic feature set that is the union thereof is described as X ₁ . The accumulated additional information set is described as Y.

Here, N is the number of elements in the complete accumulated feature set, and N _x is the number of elements in each of the accumulated image feature set and the accumulated acoustic feature set (Note: the number of elements in both sets means the same). In addition, each element x _{0, i} , x _{1, i} and y _j (i = 1, 2,..., N _x , j = 1, 2,..., N) has d _x0 dimension, d _x1 dimension, and d Let it be a _y- dimensional column vector. Accumulated features with the same subscript indicate that they correspond to each other. In the following description, when it is necessary to clearly distinguish between the complete accumulation feature set and the incomplete accumulation feature set, an alternative notation may be used as shown in the following equation.

  In the following, for the sake of simplicity, it is assumed that the average of each accumulated feature set is always a zero vector. If not, it is possible to obtain the same situation by obtaining each average vector in advance and subtracting the average vector from each accumulated feature.

At this time, as complete storage feature set statistic _{S C,} each storage feature set autocovariance matrix _{S Cx0x0, S Cx1x1, S yy} and cross-covariance matrix _{S Cx0x1, S Cx0y,} the _{S Cx1y,} following (1) It calculates | requires by Formula-(6) Formula.

However, ^{x T} denotes the transpose (vector or matrix).

In this way, the completely accumulated feature set statistic calculation unit 81 extracts and outputs the completely accumulated feature set statistic S _C = {S _Cx0x0 , S _Cx1x1 , S _yy , S _Cx0x1 , S _Cx0y , S _Cx1y }. .

Next, the incompletely stored feature set statistic calculation unit 82 includes an incompletely stored feature set (incompletely stored image feature set X) that is a set of combinations of incompletely stored image features and incompletely stored acoustic features. _I0 , incompletely stored acoustic feature set X _I1 ) is input, an incompletely stored feature set statistic that is a statistic representing the incompletely stored feature set is calculated, and this incompletely stored feature set statistic is output. . The method for calculating the incompletely accumulated feature set statistic is not particularly limited. Here, the incompletely accumulated feature set statistic is used as the incompletely accumulated feature set statistic according to the following equations (7) to (9). The set autocovariance matrices S _Ix0x0 and S _Ix1x1 and the mutual covariance matrix S _Ix0x1 are calculated.

As described above, the incompletely accumulated feature set statistic calculating unit 82 outputs the incompletely accumulated feature set statistics S _I = {S _Ix0x0 , S _Ix1x1 , S _Ix0x1 }.

  Next, the integrated statistic calculating unit 83 inputs the completely accumulated feature set statistic and the incompletely accumulated feature set statistic, calculates an integrated statistic that is a new statistic from these statistics, and the integrated statistic Output statistics. The method for calculating the integrated statistics is not particularly limited, but here, two types of integrated statistics calculated from the autocovariance matrix and the mutual covariance matrix are calculated.

  The first integrated statistic is calculated by the following equation (10). In addition, the character with _ attached to the lower part of the character in the formula is indicated by writing _ before the character in the sentence. That is, the first integrated statistic is described as _C.

Here, β is a constant determined in advance to satisfy 0 ≦ β ≦ 1, I _d is a d × d unit matrix, and 0 is a zero matrix. S _Cx0x0 is _{d x0} × _{d x0} square _{matrix, S Cx1x1} is _{d x1} × _{d x1} square _matrix, because the _{S yy} is _{d y} × _{d y} square matrix, the first integrated statistics _C is _(d x0 ₊ d _x1 + D _y ) × (d _x0 + d _x1 + d _y ) square matrix.

  On the other hand, the second integrated statistic is calculated by the following equation (11). In addition, a character with a 文字 at the top of the character in the formula is indicated with a ￣ in front of the character in the sentence. That is, the integrated statistic is described as ￣C.

Similarly to the first integrated statistic, the second integrated statistic ￣C is a (d _x0 + d _x1 + d _y ) × (d _x0 + d _x1 + d _y ) square matrix.

  As described above, the integrated statistic calculation unit 83 combines the first integrated statistic_C and the second integrated statistic C￣ to obtain an integrated statistic C = {_ C, ￣C}, and the integrated statistic. C is output.

  Next, the feature compression function determination unit 84 receives the integrated statistic C, determines a feature compression function that is a function for compressing the image feature, the acoustic feature, and the additional information feature, and outputs the feature compression function. The method for determining the feature compression function is not particularly limited, but here it is derived by solving the generalized eigenvalue problem using the first integrated statistic and the second integrated statistic.

  First, consider the generalized eigenvalue problem expressed by the following equation (12).

Here, w is a (d _x0 + d _x1 + d _y ) -dimensional vector. Solve the generalized eigenvalue problem shown in equation (12) above to find a set of a predetermined number of eigenvalues and eigenvectors, or a set of the maximum number of eigenvalues and eigenvectors whose sum of eigenvalues exceeds a predetermined threshold. Thus, the feature compression function can be determined.

Specifically, it is as follows. Each eigenvector w _i can be decomposed into a leading (d _x0 + d _x1 ) dimensional vector w _{x, i} and a subsequent _dy dimensional vector w _{y, i} . Then, using the decomposed eigenvectors w _{x, i} , w _{y, i} and the corresponding eigenvalues λ _i , the feature compression function is characterized (d _x0 + d _x1 ) × hat (^) d transformation matrices T _x and d _y × hat (^) d transformation matrix T _y is obtained as in the following equations (13) and (14).

Here, the hat (^) d is the number of eigenvalues and eigenvectors taken out, and satisfies the hat (^) d ≦ min (d _x , d _y ). Λ is a hat (^) d × hat (^) d diagonal matrix having the square root of the eigenvalue λ _i as a value for each diagonal component.

In this way, the feature compression function determination unit 44 calculates the transformation matrices T _x and T _y that characterize the feature compression function, and outputs these transformation matrices.

Next, the feature compression unit 85 receives the stored image feature set X ₀ (X _C0 , X _I0 ), the stored acoustic feature set X ₁ (X _C , X _I ), the stored additional information feature set Y, and the feature compression function. Then, each feature is compressed with a feature compression function, and a stored image compression feature set, a stored acoustic compression feature set, and a stored additional information compression feature set, which are sets of compressed features, are output. The stored image compression feature set hat (^) X ₀ , the stored acoustic compression feature set hat (^) X ₁ and the stored additional information compression feature set hat (^) Y are expressed by the following formulas (15) and (16): As described above, each feature is compressed by using a transformation matrix that characterizes the feature compression function.

As described above, the feature compression unit 85 derives the stored image compression feature set hat (^) X ₀ , the stored acoustic compression feature set hat (^) X ₁ , and the stored additional information compression feature set hat (^) Y, These compressed feature sets are output.

Finally, a multidimensional vector set obtained by synthesizing the hat (^) X ₀ , the hat (^) X ₁ , and the hat (^) Y according to the following expressions (17) and (18) is calculated, and accumulated latent variables It is used as a set Z = {z ₁ , z ₂ ,..., Z _Nx }.

However, each a _i is a constant given in advance. In this way, the accumulated latent variable extraction unit 8 extracts the accumulated latent variable set Z and outputs this accumulated latent variable set.

  The relationship learning unit 9 between the video and additional information includes a complete storage acoustic feature set, an incomplete storage acoustic feature set, a complete storage image feature set, an incomplete storage image feature set, a storage additional information feature set, and a storage latent variable set. From these sets, a video / additional information relationship model, which is a model describing the relationship between video and additional information, is learned, and this video / additional information relationship model is output. The learning method of the video / additional information relationship model is not particularly limited, but here, the latent variable model learning unit 91, the video / latent variable relationship model learning unit 92, and the additional information / latent variable relationship model learning are performed. A method using the unit 93 will be described.

  The latent variable model learning unit 91 receives a set of accumulated latent variables, learns a latent variable model that is a model describing the structure of the accumulated latent variables, and outputs the latent variable model. The learning method of the latent variable model is not particularly limited, but here, the occurrence probability p (z) of the latent variable z derived by the following equation (19) is adopted as the latent variable model.

Where δ _{a, b} is the Kronecker delta.

  In this way, the latent variable model learning unit 91 extracts and outputs the latent variable model p (z).

Next, the video / latent variable relationship model learning unit 92 inputs a stored acoustic feature set, a stored image feature set, and a stored latent variable set, and describes the relationship between the video and the latent variable using these sets. A video / latent variable relationship model as a model is learned, and the video / latent variable relationship model is output. The learning method of the video / latent variable relationship model is not particularly limited, but here, the image feature x ₀ obtained by the following equation (20) when the latent variable z is given. And the conditional occurrence probability p (x ₀ , x ₁ | z) of the acoustic feature x ₁ is adopted as the video / latent variable relationship model. In addition, a character with “˜” in the upper part of the character in the formula is indicated by “˜” before the character in the sentence.

However, to z are converted accumulated latent variables accumulated image feature x ₀ and accumulation acoustic feature x ₁ in the feature compression unit 85, gamma is a predetermined constant, I ^ _d is ^ d × ^ d It is an identity matrix. G (˜z; z _n , γI _{^ d} ) represents a multidimensional normal distribution of ~ z in which z _n is an average vector and γI _{^ d} is a covariance matrix.

  In this way, the video / latent variable relationship model learning unit 92 extracts the video / latent variable relationship model and outputs it.

  The additional information / latent variable relationship model learning unit 93 inputs a stored additional information feature set and a stored latent variable set, and uses these sets to describe the relationship between the additional information and the latent variable. The latent variable relationship model is learned, and this additional information / latent variable relationship model is output.

  The learning method of the additional information / latent variable relationship model is not particularly limited, but here, the latent variable z obtained by the following equations (21) to (24) is given. The conditional occurrence probability p (y | z) of the additional information feature y is adopted as the additional information / latent variable relationship model.

Here, mu is a constant satisfying 0 ≦ μ ≦ 1, y n , i is the i-th element of the accumulated additional information feature y _n. That is, the above relational expression first assumes that each language label occurs independently (the above equation (21)), and the occurrence probability of each language label is expressed as the empirical distribution of the language label in each sample n (the above) (22) (corresponding to δ _{yi, yn, i} in the equation) and the empirical distribution of language labels in all samples (corresponding to M _i / M in the above equation (22)) at a mixing ratio μ. Means.

  In this way, the additional information / latent variable relationship model learning unit 93 extracts the additional information / latent variable relationship model and outputs it.

  As described above, the video / additional information relationship learning unit 9 combines the latent variable model, the video / latent variable relationship model, and the additional information / latent variable relationship model into a video / additional information relationship model.・ Output additional information relation model.

<Operation of video additional information relationship learning device>
Next, the operation of the video additional information relationship learning apparatus 100 according to the present embodiment will be described. First, a complete stored video set to which additional information is given, a stored additional information set that is a set of the additional information, and an incompletely stored video set to which no additional information is given are input to the video additional information relationship learning apparatus 100. Then, the video additional information relationship learning device 100 stores the input stored additional information set in the stored additional information database 1 and the input complete stored video set in the fully stored video database 2. The input incompletely stored video set is stored in the incompletely stored video database 3. Then, the video additional information relationship learning apparatus 100 executes a model learning process routine shown in FIG.

  First, in step S101, the completely accumulated image feature and the incompletely accumulated image feature are extracted from each accumulated image of the completely accumulated image set and the incompletely accumulated image set, and stored in the feature database 7. Then, in step S102, the completely accumulated acoustic feature and the incompletely accumulated acoustic feature are extracted from each accumulated image of the complete accumulated image set and the incompletely accumulated image set, and stored in the feature database 7. In step S 103, the accumulated additional information feature is extracted from each additional information of the accumulated additional information set and stored in the feature database 7.

In the next step S104, with respect to the complete accumulation feature set that is a set of combinations of the completely accumulated image feature, the completely accumulated acoustic feature, and the accumulated additional information feature extracted in the above steps S101 to S103, ) To calculate auto-covariance matrices S _Cx0x0 , S _Cx1x1 , S _yy and cross-covariance matrices S _Cx0x1 , S _Cx0y , S _Cx1y .

In step S105, the above-described equations (7) to (9) are used for the incompletely stored image feature extracted in steps S101 to S102 and the incompletely stored feature set that is a set of incompletely stored acoustic features. Accordingly, the self-covariance matrices S _Ix0x0 and S _Ix1x1 and the mutual covariance matrix S _Ix0x1 are calculated.

In step S106, the self-covariance matrices S _Cx0x0 , S _Cx1x1 , S _yy and the mutual covariance matrices S _Cx0x1 , S _Cx0y , S _{Cx1y of} the complete accumulation feature set calculated in step S105 and the self-incomplete accumulation feature set self Based on the covariance matrices SI _x0x0 , S _Ix1x1 and the mutual covariance matrix S _Ix0x1 , the first integrated statistic _C and the second integrated statistic ￣C are calculated according to the above formulas (10) and (11). calculate.

In step S107, the generalized eigenvalue problem expressed by the above equation (12) is solved using the first integrated statistic_C and the second integrated statistic ￣C calculated in step S106. A set of a predetermined number of eigenvalues and eigenvectors is obtained. Using the set of the obtained eigenvalue and eigenvector, transformation matrices T _x and T _y characterizing the feature compression function are calculated according to the above equations (13) and (14).

In the next step S108, using the feature compression function determined by the transformation matrix _{T x} and _{T y} calculated in step S107, the equation (15), according to (16), storing the image compression feature set hat ( ^) X ₀ , stored acoustic compression feature set hat (^) X ₁ , and stored additional information compression feature set hat (^) Y are calculated. In step S109, the stored image compression feature set hat (^) X ₀ calculated in step S108, the stored acoustic compression feature set hat (^) X ₁ , and the storage addition according to the above equations (17) and (18). The information compression feature set hat (^) Y is synthesized to calculate an accumulated latent variable set.

In step S110, the latent variable model p (z) is learned according to the above equation (19) using the accumulated latent variable set calculated in step S109. In the next step S111, the image / latent variable relationship model p (x ₀ , x ₁ | z) is learned according to the above equation (20) using the accumulated acoustic feature set, the accumulated image feature set, and the accumulated latent variable set. To do.

In step S112, the additional information / latent variable relationship model p (y | z) is learned according to the above equation (21) using the stored additional information feature set and the stored latent variable set. In step S113, the latent variable model p (z) learned in step S110, the video / latent variable relationship model p (x ₀ , x ₁ | z) learned in step S111, and learned in step S112. The additional information / latent variable relationship model p (y | z) is output by the output unit 30 as a video / additional information relationship model, and the model learning processing routine is terminated.

  As described above, according to the video additional information relationship learning device according to the first embodiment, the completely accumulated image feature and the completely accumulated acoustic feature extracted from each of the completely accumulated image, and the incompletely accumulated image By learning a model that describes the relationship between video and additional information from incompletely stored image features and incompletely stored acoustic features extracted from each, and a set of stored latent variables, image information contained in the video By using both of the acoustic information and considering the mutual co-occurrence relationship, the relationship between the information and the language information can be learned with higher accuracy.

  Further, by calculating the integrated statistic by the above formulas (10) and (11) and by calculating the eigenvector by the above formula (12), the correlation between the three of the image information, the acoustic information, and the additional information included in the video is obtained. Relationships (co-occurrence relationships) can be easily learned. This makes it possible to learn from the stored information the relationship between the video signal and the additional information that cannot be obtained only by the image information, using the combination of the sound information and the sound information and the image information as a clue. Can improve the accuracy of video search and video recognition.

  Further, by extracting a latent variable for describing the relationship between the video and the additional information from both the video given the additional information and the video not given the additional information by the accumulated latent variable extraction unit, Even when only a small amount of video provided with additional information can be used, the relationship between the video and additional information can be learned with high accuracy.

  While it is difficult to collect a large amount of video with additional information, it is very easy to collect the video itself when it is not necessary to provide additional information. Is possible. Compared to the case where only a video with only a small amount of additional information is used by simultaneously using the video without additional information when learning the relationship between the video and the additional information. The relationship between video and additional information can be learned with high accuracy.

[Second Embodiment]
<System configuration>
Next, a second embodiment of the present invention will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The second embodiment is different from the first embodiment in that the video / additional information relation model is used to search for a video having high relevance to the input additional information. Further, a case where the present invention is applied to a semi-supervised video search apparatus that searches a set of videos related to input additional information will be described as an example.

  As shown in FIG. 4, the semi-supervised video search apparatus 200 according to the second embodiment includes additional information provided separately from a complete stored video set, an incomplete stored video set, a stored additional information set, and a stored additional information set. Is a device that inputs input additional information and outputs an additional information related video set that is a set of videos related to the input additional information. The semi-teacher video search apparatus 200 includes an input unit 10, a calculation unit 220, and an output unit 30.

  The input unit 10 receives input of a complete stored video set, an incompletely stored video set, and a stored additional information set, and receives input additional information as a query for searching for videos.

  The calculation unit 220 includes a storage additional information database 1, a complete storage video database 2, an incomplete storage video database 3, a storage additional information feature extraction unit 4, a storage image feature extraction unit 5, a storage acoustic feature extraction unit 6, a feature database 7, A storage latent variable extraction unit 8, a video / additional information relationship learning unit 9, an input additional information feature extraction unit 11, and a video search unit 12 are provided.

  The input additional information feature extraction unit 11 inputs the input additional information, extracts an input additional information feature that is a vector expressing the characteristics of the input additional information, and outputs the input additional information feature. The input additional information feature extraction method is the same as that of the stored additional information feature extraction unit 4.

  The video search unit 12 inputs the input additional information feature, the completely stored video set, the incompletely stored video set, and the video / additional information relation model, and gives the input additional information feature to the video / additional information relation model, thereby From the stored video set and the incompletely stored video set, an additional information related video that is a video having a high relationship with the input additional information is selected, and an additional information related video set that is a set of the additional information related video is output.

  The method for selecting the additional information related image is not particularly limited, but here, the following method will be described.

First, the posterior probabilities of the image feature x ₀ and the acoustic feature x ₁ when the input additional information feature y give is _given are set by the following equation (25).

The posterior probability _{p (x 0, x 1 |} y given) and were calculated for each combination of storage acoustic feature set and stored image feature set, a large a number of the stored images, wherein the accumulation audio feature set of a posteriori probabilities, or A set of accumulated image features and accumulated acoustic features whose posterior probabilities exceed a threshold is selected, and a set of images corresponding to the set of accumulated image features and accumulated acoustic features is set as an additional information related image set.

  In this way, the video search unit 12 selects the additional information related video set, and the output unit 30 outputs the additional information related video set.

<Operation of semi-teacher video search device>
First, when a complete stored video set, a stored additional information set, and an incomplete stored video set are input to the semi-supervised video search device 200, the input stored additional information set is stored by the semi-supervised video search device 200. The complete stored video set stored and input in the additional information database 1 is stored in the fully stored video database 2, and the input incompletely stored video set is stored in the incompletely stored video database 3. Then, the semi-teacher video search apparatus 200 executes the model learning processing routine shown in FIG. 3 as in the first embodiment.

  When the input additional information is input to the semi-teacher video search device 200 as a query for searching for the video, the video search processing routine shown in FIG.

  In step S201, input additional information is received. In step S202, input additional information features are extracted from the input additional information.

Then, in step S203, the posterior probabilities p (x ₀ , x ₁ | y _given ) for each combination of the stored image feature and the stored acoustic feature obtained from the stored image feature set and the stored acoustic feature set according to the above equation (25). ). In step S204, a combination of stored image features and stored acoustic features in which the posterior probability p (x ₀ , x ₁ | y _give ) calculated in step S203 is equal to or greater than a threshold value is extracted, and a complete stored video set and incomplete storage are extracted. A video corresponding to the combination of the extracted stored image feature and the stored sound feature is selected as the additional information related video from the video set.

  In step S205, the output unit 30 outputs the additional information related video set selected in step S204, and the video search processing routine is terminated.

  As described above, according to the semi-supervised video search apparatus according to the second embodiment, the model describing the relationship between the learned video and the additional information is used, so that it is highly related to the input additional information. Video search can be performed with high accuracy.

[Third Embodiment]
<System configuration>
Next, a third embodiment of the present invention will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The third embodiment is different from the first embodiment in that additional information that is highly relevant to an input video is output using a video / additional information relationship model. Also, a case where the present invention is applied to a semi-supervised video recognition apparatus that outputs a set of additional information describing an input video will be described as an example.

  As shown in FIG. 6, the semi-supervised video recognition apparatus 300 according to the third embodiment includes a complete stored video set, an incomplete stored video set, a stored additional information set, a complete stored video set, and an incomplete stored video set. This is an apparatus for inputting an input video that is a separately provided video and outputting a video-related additional information set that is a set of additional information corresponding to the input video. The semi-teacher video recognition apparatus 300 includes an input unit 10, a calculation unit 320, and an output unit 30.

  The input unit 10 receives input of a complete stored video set, an incompletely stored video set, and a stored additional information set, and also receives an input of an input video that is a video recognition target.

  The calculation unit 320 includes a storage additional information database 1, a complete storage video database 2, an incomplete storage video database 3, a storage additional information feature extraction unit 4, a storage image feature extraction unit 5, a storage acoustic feature extraction unit 6, a feature database 7, An accumulated latent variable extraction unit 8, a video / additional information relationship learning unit 9, an input image feature extraction unit 13, an input acoustic feature extraction unit 14, and an image recognition unit 15 are provided.

  The input image feature extraction unit 13 inputs an input video, extracts an input image feature that is a vector representing characteristics of an image signal included in the input video, and outputs the input image feature. Since the input image feature extraction method is the same as that of the accumulated image feature extraction unit 5, a description thereof will be omitted.

  The input acoustic feature extraction unit 14 inputs an input video, extracts an input acoustic feature that is a vector expressing the characteristics of an acoustic signal included in the input video, and outputs the input acoustic feature. The input acoustic feature extraction method is the same as that of the accumulated acoustic feature extraction unit 6, and thus the description thereof is omitted.

  The video recognition unit 15 inputs the input image feature, the input acoustic feature, and the video / additional information relation model, and gives the input image feature and the input acoustic feature to the video / additional information relation model, thereby relevance with the input picture. The video related additional information which is high additional information is extracted, and this video related additional information is output. The method for selecting video-related additional information is not particularly limited, but the following method will be described here.

First, the a posteriori probability p (y | x _{0, given} , x _{1, given} ) of the additional information feature y when the input image feature x _{0, given} and the input acoustic feature x _{1, given} are _given is _expressed as (26 ) To set.

Here, from the formulation of the video / latent variable relationship model p (x ₀ , x ₁ | z) and the additional information / latent variable relationship model p (y | z), the additional information feature y that maximizes the posterior probability Can be calculated by the following equation (27).

_{However, z Given,} using the processing shown in accumulating latent variable extraction unit 8, a latent variable calculated from the input image feature _{x 0, Given} and input acoustic feature _{x 1, Given.} Further, p (y _d = 1 | z _n ) is a conditional probability of the additional information feature y _d that becomes 1 when the latent variable z _n is given.

  Note that the additional information feature ˜y that maximizes the posterior probability generally does not become a binary vector. From the additional information features with the maximum posterior probability-y, select a certain number of elements with a large value or elements whose values exceed the threshold value, collect language labels corresponding to each element, and add video-related information Information.

  In this way, the video recognition unit 15 selects the video related additional information, and the output unit 30 outputs the video related additional information.

<Operation of semi-supervised video recognition device>
First, when a complete stored video set, a stored additional information set, and an incomplete stored video set are input to the semi-supervised video recognition device 300, the input stored additional information set is stored by the semi-supervised video recognition device 300. The complete stored video set stored and input in the additional information database 1 is stored in the fully stored video database 2, and the input incompletely stored video set is stored in the incompletely stored video database 3. Then, the semi-teacher video recognition apparatus 300 executes a model learning processing routine as in the first embodiment.

  When the input video to be recognized is input to the semi-teacher video recognition apparatus 300, the video recognition processing routine shown in FIG.

  In step S301, the input video input is received. In step S302, input image features are extracted from the input video. In step S303, input acoustic features are extracted from the input video.

In step S304, the posterior probabilities p (y | _x0 , _given , x1 _{, given} ) are calculated according to the above equation (27) using the input image features and input acoustic features extracted in steps S302 and S303. The maximum additional information feature -y is calculated.

  In the next step S305, an element whose value exceeds the threshold value is selected from the additional information features -y calculated in step S304. In step S306, language labels corresponding to the elements selected in step S305 are collected and output as video-related additional information by the output unit 30, and the video recognition processing routine ends.

  As described above, according to the semi-supervised video recognition apparatus according to the third embodiment, the model describing the relationship between the learned video and the additional information is used, so that the relevance with the input video is high. Additional information can be accurately obtained as a video recognition result.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program may be provided by being stored in a computer-readable recording medium.

1 accumulated additional information database 2 complete accumulated video database 3 incomplete accumulated video database 4 accumulated additional information feature extracting unit 5 accumulated image feature extracting unit 6 accumulated acoustic feature extracting unit 7 feature database 8 accumulated latent variable extracting unit 9 between video and additional information Relationship learning unit 11 Input additional information feature extraction unit 12 Video search unit 13 Input image feature extraction unit 14 Input acoustic feature extraction unit 15 Video recognition unit 15 Image recognition unit 44 Feature compression function determination unit 81 Complete accumulation feature set statistic calculation unit 82 Incompletely Accumulated Feature Set Statistics Calculation Unit 83 Integrated Statistics Calculation Unit 84 Feature Compression Function Determination Unit 85 Feature Compression Unit 91 Latent Variable Model Learning Unit 92 Video / Latent Variable Relationship Model Learning Unit 93 Additional Information / Latent Variable Relationship Model Learning Part 100 Video Additional Information Relationship Learning Device 200 Semi-Teacher Video Retrieval Device 300 Semi-Teacher Video Recognition Device

Claims (9)

A video additional information relationship learning device that learns a relationship between a video that is a moving image with sound and additional information that is information that describes the video,
Completely accumulated video that is an element of a fully accumulated video set that is a set of videos to which additional information is given in advance, and incompletely accumulated video that is an element of an incompletely accumulated video set that is a set of videos to which no additional information is given A stored image feature extracting means for extracting a completely stored image feature and an incompletely stored image feature, which are vectors expressing the characteristics of the image,
Fully stored sound features and incompletely stored sound that are vectors representing acoustic characteristics from each of the completely stored video that is an element of the complete stored video set and the incompletely stored video that is an element of the incompletely stored video set. Accumulated acoustic feature extraction means for extracting features;
A stored additional information feature extracting means for extracting a stored additional information feature, which is a vector expressing the characteristics of the additional information, from each of the stored additional information that is an element of the stored additional information set that is a set of the additional information;
A completely stored image feature set that is a set of the completely stored image features, an incompletely stored image feature set that is a set of the incompletely stored image features, a fully stored acoustic feature set that is a set of the completely stored acoustic features, and the incomplete A storage latency that is a set of variables for describing the relationship between video and additional information from an incompletely stored acoustic feature set that is a set of stored acoustic features and a stored additional information feature set that is a set of stored additional information features. An accumulated latent variable extracting means for extracting a variable set;
From the complete stored image feature set, the incomplete stored image feature set, the complete stored acoustic feature set, the incomplete stored acoustic feature set, the stored additional information feature set, and the stored latent variable set, video and additional information, A video additional information relationship learning device comprising: a video / additional information relationship learning means for learning a video / additional information relationship model, which is a model for describing the relationship between the images. The accumulated latent variable extracting means includes
A complete accumulation feature set statistic that is a statistic representing a statistical property of a complete accumulation feature set that is a set of the combination of the perfect accumulation image feature, the perfect accumulation acoustic feature, and the corresponding accumulation additional information feature is calculated. A completely accumulated feature set statistic calculation means;
An incomplete accumulation feature set for calculating an incomplete accumulation feature set statistic that is a statistic expressing a statistical property of the incomplete accumulation feature set that is a set of a combination of the incomplete accumulation image feature and the incomplete accumulation acoustic feature A statistic calculation means;
An integrated statistic calculation means for calculating an integrated statistic by combining the complete accumulation feature set statistic and the incomplete accumulation feature set statistic;
Feature compression function determination means for determining a feature compression function that is a function for compressing the image feature, the acoustic feature, and the additional information feature using the integrated statistic;
Using the feature compression function, a stored image compression feature set obtained by compressing the stored image feature set, a stored acoustic compression feature set obtained by compressing the stored acoustic feature set, and a stored additional information compression obtained by compressing the stored additional information feature set. A feature compression unit that calculates a feature set, combines the stored image compression feature set, the stored acoustic compression feature set, and a stored additional information compression feature set to calculate the stored latent variable set;
The video additional information relationship learning device according to claim 1, comprising: An input additional information feature extracting means for extracting an input additional information feature which is a vector expressing the characteristic of the additional information from the input additional information;
Video search means for providing the input additional information feature to the video / additional information relation model and searching for a video having high relevance to the input additional information from the complete stored video set and the incompletely stored video set; The video additional information relationship learning device according to claim 1, further comprising: An input image feature extracting means for extracting an input image feature which is a vector expressing the characteristics of the image from the input video;
Input acoustic feature extraction means for extracting an input acoustic feature, which is a vector expressing acoustic characteristics, from the input video;
And a video recognition unit that gives the input image feature and the input acoustic feature to the video / additional information relation model and selects additional information highly relevant to the input video from the stored additional information set. Item 3. The video additional information relationship learning device according to Item 1 or 2. A video additional information relationship learning method used in a video additional information relationship learning device that learns a relationship between a video that is a moving image with sound and additional information that is information that describes the video,
By the stored image feature extraction means, a complete stored video that is an element of a complete stored video set that is a set of videos to which additional information is given in advance, and an incompletely stored video set that is a set of videos to which no additional information is given. Extracting from each of the incompletely stored images that are the elements a fully stored image feature and an incompletely stored image feature that are vectors representing the characteristics of the image;
Complete accumulation, which is a vector that expresses acoustic characteristics from each of the completely accumulated video, which is an element of the completely accumulated video set, and the incompletely accumulated video, which is an element of the incompletely accumulated video set, by the accumulated acoustic feature extraction means Extracting acoustic features and imperfectly accumulated acoustic features;
A step of extracting a stored additional information feature, which is a vector representing a characteristic of the additional information, from each of the stored additional information that is an element of the stored additional information set, which is a set of the added additional information, by the stored additional information feature extracting means; When,
By the accumulated latent variable extraction means, a completely accumulated image feature set that is a set of the completely accumulated image features, an incompletely accumulated image feature set that is a set of the incompletely accumulated image features, and a complete accumulation that is a set of the completely accumulated acoustic features. To describe the relationship between video and additional information from an acoustic feature set, an incompletely stored acoustic feature set that is a set of the incompletely stored acoustic features, and a stored additional information feature set that is a set of the stored additional information features Extracting an accumulated latent variable set that is a set of variables;
The complete accumulated image feature set, the incompletely accumulated image feature set, the complete accumulated acoustic feature set, the incompletely accumulated acoustic feature set, the accumulated additional information feature set, and the video / additional information relationship learning means A video additional information relationship learning method comprising: learning a video / additional information relationship model, which is a model describing a relationship between a video and additional information, from a stored latent variable set. Extracting the accumulated latent variable set by the accumulated latent variable extracting means;
A statistic that expresses the statistical properties of the complete accumulation feature set, which is a set of the combination of the complete accumulation image feature, the complete accumulation acoustic feature, and the corresponding accumulated additional information feature by the perfect accumulation feature set statistic calculation means. Calculating a complete accumulated feature set statistic;
The incompletely accumulated feature set statistic calculating means is an incompletely accumulated feature that is a statistic representing the statistical properties of the incompletely accumulated feature set that is a set of the combination of the incompletely accumulated image feature and the incompletely accumulated acoustic feature. Calculating an aggregate statistic;
A step of calculating an integrated statistic by combining the complete accumulated feature set statistic and the incompletely accumulated feature set statistic by an integrated statistic calculating means;
Determining a feature compression function, which is a function for compressing the image feature, the acoustic feature, and the additional information feature, using the integrated statistic by a feature compression function determination unit;
The feature compression means compresses the stored image compression feature set obtained by compressing the stored image feature set, the stored acoustic compression feature set obtained by compressing the stored acoustic feature set, and the stored additional information feature set using the feature compression function. Calculating the stored additional information compression feature set, combining the stored image compression feature set, the stored acoustic compression feature set, and the stored additional information compression feature set to calculate the stored latent variable set;
The video additional information relationship learning method according to claim 5, comprising: Extracting an input additional information feature which is a vector expressing the characteristic of the additional information from the input additional information by the input additional information feature extracting means;
The video additional means provides the input additional information feature to the video / additional information relationship model, and searches for a video having high relevance to the input additional information from the complete stored video set and the incompletely stored video set. The video additional information relationship learning method according to claim 5, further comprising: a step. Extracting an input image feature which is a vector expressing the characteristics of the image from the input video by the input image feature extracting means;
Extracting an input acoustic feature, which is a vector expressing acoustic characteristics, from the input video by an input acoustic feature extracting means;
Providing the input image feature and the input acoustic feature to the video / additional information relation model by a video recognition means, and selecting additional information highly relevant to the input video from the stored additional information set; The video additional information relationship learning method according to claim 5, further comprising:   The program for functioning a computer as each means of the image | video additional information relationship learning apparatus of any one of Claims 1-4.

Images