JP6993250B2 - Content feature extractor, method, and program - Google Patents

Description

The present invention relates to a content feature amount extraction device, a method, and a program, and more particularly to a content feature amount extraction device, a method, and a program for extracting a content feature amount.

Related to images and videos, such as object recognition that identifies objects contained in images and videos, object detection that identifies the positions of objects contained in images and videos, and generation of explanatory texts that describe the contents of images and videos. Image features, which represent the characteristics of images and videos, play an extremely important role in the performance of various tasks. In recent years, image feature extraction uses a large-scale data set consisting of a large number of images / videos and labels attached to each of those images / videos to learn a feature model represented by a convolutional neural network. Based on supervised learning. However, these supervised learning-based methods require very large labeled datasets. The work of assigning accurate labels to each of images and videos requires a great deal of human operation, which is one of the bottlenecks in feature quantity extraction.

In order to eliminate this bottleneck, a feature amount extraction method using a web image has been devised. The advantage of using a Web image is that although it is not as accurate as a label given manually, information equivalent to a label useful for feature learning can be obtained without human operation. For example, when collecting images using a web image search system, the query used in the search can be used as information corresponding to the label (Non-Patent Document 1). Further, when collecting images from a content sharing site, a text tag assigned on the sharing site can be used (Non-Patent Document 2).

Sukhbaatar, Bruna, Paluri, Bourdev and Fergus, “Training convolutional networks from noisy labels,” Proc. International Conference on Learning Representations (ICLR), 2015. Joulin, van der Maaten and Jabri, “Learning visual features from large weakly supervised data,” Proc. European Conference on Computer Vision (ECCV), 2016.

However, these existing technologies do not reflect the important properties of label-equivalent information obtained using web images. That is, unlike a label given manually, the fact that a particular label is not given to an image does not mean that the image does not contain the content associated with that label. be. For example, when collecting images using a web image search system, it is naturally assumed that the collected images contain contents other than the query used in the search, but labels that cover all of those contents are collected. That is extremely difficult. In the general feature amount learning method including the above-mentioned existing technology, discriminative learning is performed by equating the absence of a label with the fact that the image does not include the related content, so that appropriate image feature amount learning is performed. I can't do it.

The present invention has been made to solve the above problems, and is a content feature amount extraction device, a method, and a program capable of extracting feature amounts of contents in consideration of a digital content set prepared in advance. The purpose is to provide.

In order to achieve the above object, the content feature amount extraction method according to the present invention uses a plurality of digital content sets composed of digital contents prepared in advance, and the digital contents included in the digital contents set and the digital contents. In the content feature amount extraction method in the content feature amount extraction device that extracts at least one feature amount of the new digital content given separately from the set, the content basic feature amount extraction unit includes the digital content included in the digital content set. For each of the above, a step of extracting the basic content feature amount, which is the basic feature amount of the digital content, and a graph in which the graph construction unit expresses which digital content set each of the digital contents is included in. Based on the step of constructing a certain content graph, the content basic feature amount extracted from each of the digital contents by the link prediction model learning unit, and the content graph, the presence or absence of belonging to the digital content set is determined by the digital content. The step of learning the link prediction model, which is a model for predicting from the content basic feature amount, and the digital content included in the digital content set based on the learned link prediction model, and the content feature amount calculation unit. It is characterized in that at least one of the new digital contents is executed including a step of calculating a content feature amount which is a feature amount of the digital content.

The content feature amount extracting device according to the present invention uses a plurality of digital content sets composed of digital contents prepared in advance, and is a novel content given separately from the digital contents included in the digital contents set and the digital contents set. In the content feature amount extraction device that extracts at least one feature amount of the digital content, the content basic feature amount which is the basic feature amount of the digital content is extracted for each of the digital contents included in the digital content set. A content basic feature amount extraction unit, a graph construction unit that constructs a content graph that is a graph expressing which digital content set each of the digital contents is included in, and a content basic feature extracted from each of the digital contents. Learned with a link prediction model learning unit that learns a link prediction model, which is a model that predicts the presence or absence of belonging to the digital content set from the content basic feature amount of the digital content based on the quantity and the content graph. Based on the link prediction model, the content feature amount calculation unit that calculates the content feature amount, which is the feature amount of the digital content, for at least one of the digital content included in the digital content set and the new digital content. It is composed of including.

The program according to the present invention is a program for causing a computer to execute each step of the content feature amount extraction method of the present invention.

According to the content feature amount extraction device, method, and program of the present invention, it is possible to obtain the effect that the feature amount of the content in consideration of the digital content set prepared in advance can be extracted.

It is a block diagram which shows the structure of the content feature amount extraction apparatus which concerns on 1st Embodiment of this invention. It is a flowchart which shows the content feature amount extraction processing routine in the content feature amount extraction apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the content feature amount extraction apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the content feature amount extraction processing routine in the content feature amount extraction apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the content feature amount extraction apparatus which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the content feature amount extraction processing routine in the content feature amount extraction apparatus which concerns on 3rd Embodiment of this invention. It is a figure which shows an example of the statistics of the number of digital contents sets to which digital contents belong in an experimental result. It is a figure which shows an example of the statistics of the number of digital contents contained in each digital content set in the experimental result.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview of Embodiments of the Present Invention>

First, an outline of the embodiment of the present invention will be described.

The method according to the embodiment of the present invention provides, for example, a label prediction model that reflects the properties of label-equivalent information acquired from a web image, and also provides a new means for learning features using this label prediction model. It is a thing.

More specifically, (1) an image set that holds information equivalent to the same label is defined, (2) a graph that expresses whether or not an image belongs to this image set is constructed, and (3) a graph. Label prediction is performed by predicting the presence or absence of edges, and the image feature amount is obtained by learning using (4) the result of label prediction or the error function of the model for label prediction.

In the description so far, for the sake of clarity of explanation, the means of feature amount learning for images only have been described. However, in the embodiment of the invention described below, the target for extracting the feature amount using the learned label prediction model is not limited to the image, but various targets such as acoustic signals, texts, and sensor signals. Applicable. Hereinafter, the targets for extracting these features will be collectively referred to as digital contents. Further, a set of digital contents holding information corresponding to the same label is called a digital contents set.

An embodiment of the present invention is a content feature that extracts a feature amount of digital content included in a digital content set or a new digital content given separately from the digital content set using a plurality of digital content sets prepared in advance. It provides a means of quantity extraction.

<Structure of content feature amount extraction device according to the first embodiment of the present invention>

Next, the configuration of the content feature amount extraction device according to the first embodiment of the present invention will be described. As shown in FIG. 1, the content feature amount extraction device 100 according to the first embodiment of the present invention includes a CPU, a RAM, a program for executing a content feature amount extraction processing routine described later, and various data. It can be configured with a computer including a stored ROM. The content feature amount extraction device 100 functionally includes an input unit 10, a calculation unit 20, and an output unit 50 as shown in FIG.

The input unit 10 accepts a plurality of digital content sets composed of labeled digital contents for learning. Further, the input unit 10 accepts new digital contents whose label is unknown.

The calculation unit 20 includes a link prediction model storage unit 22, a content feature amount model storage unit 24, a content basic feature amount extraction unit 30, a content set basic feature amount extraction unit 32, a graph construction unit 34, and a link prediction model. It includes a learning unit 36, a content feature amount model learning unit 40, a content feature amount calculation unit 42, and a model alternating optimization unit 44.

The content basic feature amount extraction unit 30 extracts the content basic feature amount, which is the basic feature amount of the digital content, for each of the digital contents included in the plurality of digital content sets received by the input unit 10.

The method for extracting the basic content feature amount is not particularly limited, and various feature amount extraction methods can be selected according to the type of digital content. In the present embodiment, as an example thereof, an image is targeted. As the basic feature amount extraction method, the feature amount using VGG-net described in Non-Patent Document 3 below will be described.

(Non-Patent Document 3) Simonyan and Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint, arXiv: 1409.1556, http://arxiv.org/abs/1409.1556.

VGG-net is an object recognition model composed of 16-layer or 19-layer convolutional neural networks, and is learned by supervised learning using a large-scale object recognition data set. When this VGG-net is used as an image feature extraction model, the progress of the object recognition model learned in the object recognition data set, for example, the 14th layer (FC6: 4096 dimensions) of the 16-layer VGG-net (VGG16). , It is common to use the output of the 15th layer (FC7: 4096 dimensions) or the final layer (FC8: 1000 dimensions). The content basic feature amount extraction unit 30 inputs digital content into the object recognition model learned in this way and extracts the content basic feature amount.

The content set basic feature amount extraction unit 32 extracts the content set basic feature amount, which is the basic feature amount of the digital content set, for each of the plurality of digital content sets received by the input unit 10.

The method for extracting the basic feature amount of the content set is not particularly limited, and various feature amount extraction methods can be selected according to the digital content expressing the digital content set. In the present embodiment, as an example thereof, as a basic feature amount extraction method for text information included in a digital content set, a word / word sequence embedding method represented by the method described in Non-Patent Document 4 below is used. The method to be used will be described.

(Non-Patent Document 4) Mikolov, Sutsekver, Chen, Corrado and Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26 (NIPS 2013).

The method described in Non-Patent Document 4 is a method using a model that transforms a word or a word sequence into one multidimensional vector, and this model is learned by unsupervised learning using a large-scale document data set. When the text information included in the digital content set is one word or one word sequence, the output when this word or word sequence is given to the model can be used as it is as the content set basic feature quantity. When a plurality of words or word sequences are included, for example, the vectors obtained from all the words or word sequences are averaged and used as the content set basic feature quantity.

The graph construction unit 34 constructs a content graph which is a graph expressing which digital content set includes each of the digital contents.

Various formats of the content graph and its construction method can be considered, but in the present embodiment, each digital content and each digital content set are associated with a node, and a certain digital content _Ii is included in a certain digital content set _Gc . Sometimes an edge is created between the node corresponding to the digital content I _i and the node corresponding to the digital content set G _c . If the total number of digital contents is expressed as NI and the total number of digital _contents is expressed as _NC , this content graph is an _NC × _NI binary adjacency matrix.

It is expressed by, and it is shown that the link exists between the node pair (c, i) in which the elements a _{c and i} of this adjacency matrix are 1, and the link does not exist in the other node pairs. The content graph configured in this way becomes a bipartite graph, and one node set contains digital content and the other node set contains nodes corresponding to the digital content set.

The link prediction model learning unit 36 is a content basic feature amount extracted from each of the digital contents by the content basic feature amount extraction unit 30, and a content set basic extracted from each of a plurality of digital content sets by the content set basic feature amount extraction unit 32. Based on the feature amount and the content graph constructed by the graph construction unit 34, a link prediction model that predicts the presence or absence of belonging to the digital content set from the content basic feature amount and the content set basic feature amount is learned, and the link prediction model is used. It is stored in the storage unit 22.

The learning method of the link prediction model and the link prediction model is not particularly limited, but in the present embodiment, the linear model and the method based on the rank loss shown below will be particularly described.

First, the basic content features extracted from _NI digital contents are calculated.

, NC _Content set basic features extracted from digital content set

It is expressed as.

Using these basic features, the link prediction value is an index of whether or not there is a link between the node corresponding to the i-th digital content and the node corresponding to the c-th digital content set.

Is modeled by the following equation (1).

・・・（１）

... (1)

here,

Are all model parameters, and the link prediction model is characterized by these model parameters. Model parameters if content set basic features are available

By adding, the link prediction model is modified as shown in Eq. (2) below.

・・・（２）

... (2)

Further, a link prediction model that does not use the content latent variable z _Ii , which is a model parameter corresponding to each digital content, and the content set latent variable z _Cc , which is a model parameter corresponding to each digital content set, is also possible. When the content latent variable z _Ii is not used, the link prediction model is modified as shown in Eq. (3) below.

・・・（３）

... (3)

Similarly, when the content set latent variable z _Cc is not used, the model parameters related to the content set latent variable are omitted. Hereinafter, for the sake of simplification of the explanation, the link prediction model based on the equation (3) will be used. Even when other link prediction models are used, the treatment is almost the same.

Link prediction value modeled by equation (3)

Is close to the corresponding components _{ai and c} of the adjacency matrix A that characterizes the content graph, which is the main step of the link prediction model learning unit 36. Specifically, a model parameter that minimizes the rank loss function shown in Eq. (4) below is derived.

・・・（４）

... (4)

Here, l (・) is a loss function and Ω (・) is a regularization term, and it is necessary to select them so that they can be subdifferentiated with respect to the model parameters. As a regularization term, for example, a weighted linear sum of squared norms of each parameter matrix can be used. again,

Is.

Since it is difficult to minimize the rank loss function shown in Eq. (4) for all parameters at the same time, the stochastic gradient descent method is used to sequentially update each parameter individually so as to reduce the loss function. do. Since the rank loss function shown in equation (4) is subderivative with respect to the model parameters, the equation inside the sum of equation (4)

And regularization terms

By partially differentiating each model parameter individually, the update formula of each model parameter when the digital content i, the digital content set c including the digital content, and the digital content set c'excluding the digital content are fixed is obtained. Can be derived. Using this update equation, the model parameters are updated by using the stochastic gradient descent method. That is, the model parameters are updated for each triplet of the digital content i, the digital content set c including the digital content, and the digital content set c'not including the digital content.

The update of the model parameter is stopped when a predetermined condition is satisfied. Possible stop conditions include conditions such as the number of model parameter updates exceeding a predetermined number, the difference between the pre-update parameter and the post-update parameter being smaller than a predetermined threshold value with reference to the link prediction model storage unit 22. ..

The above is the learning process of the link prediction model of the link prediction model learning unit 36.

The content feature amount model learning unit 40 is a model for extracting the content feature amount of digital content from the link prediction model learned by the link prediction model learning unit 36 and the digital content included in the digital content set. The quantity model is learned and stored in the content feature quantity model storage unit 24.

The learning method of the content feature amount model is not particularly limited, but in the present embodiment, VGG-Net is taken as an example of the content feature amount model and stored in the link prediction model storage unit 22 as an example thereof. The method by re-learning VGG-Net using the link prediction model will be explained.

As explained in the content basic feature quantity extraction unit 30, VGG-Net is a model composed of 16-layer or 19-layer convolutional neural networks, and is usually by supervised learning using a large-scale object recognition data set. learn. That is, the model parameter is set so as to reduce the difference between the prediction of the model and the correct answer label, using the label indicating which object is included in the given image among the plurality of types of objects prepared in advance as a teacher. Update.

On the other hand, in the present embodiment, first, the digital content used for learning the link prediction model is input to VGG-Net, and the content basic feature amount is extracted in the same manner as the content basic feature amount extraction unit 30. Subsequently, the content basic feature amount is input to the trained link prediction model, and the ranking loss is calculated by the ranking loss function shown in Eq. (4). Finally, the VGG-Net model parameters are updated so that this rank loss is reduced. In this procedure, since the link prediction model is fixed, the loss function in VGG-Net training can be regarded as different from the commonly used function, and training of general neural networks such as error backpropagation can be considered. The method can be used as it is.

In the training of VGG-Net model parameters, the model parameters may be set to random initial values and training may be started. However, the model is trained in advance using a large-scale object recognition data set, and the model parameters are trained. Can also be used as the initial value. Further, in the learning of the model parameters of VGG-Net in the present embodiment, the model parameters of all the layers may be updated, but the convolution layer (from the first layer to the thirteenth layer) or one of these convolution layers is input. It is also possible to fix the model parameters of layers close to (1st layer to 10th layer, 1st layer to 6th layer, etc.) and update other model parameters.

The content feature amount calculation unit 42 calculates the content feature amount from the digital contents included in the digital content set by using the content feature amount model learned by the content feature amount model learning unit 40.

The content feature amount calculation unit 42 may or may not use the content feature amount model, but in the present embodiment, the content feature amount model is used. When the content feature amount model is used, the process is the same as that of the content basic feature amount extraction unit 30, for example, the 14th layer (FC6: 4096 dimensions) and the 15th layer (FC6: 4096 dimensions) of the 16-layer VGG-net (VGG16). The output of FC7: 4096 dimensions) or the final layer (FC8: 1000 dimensions) can be used.

The model alternating optimization unit 44 optimizes the link prediction model and the content feature amount model by alternately and repeatedly executing the processing of the link prediction model learning unit 36 and the processing of the content feature amount model learning unit 40. The model alternating optimization unit 44 determines whether or not the iterative end condition is satisfied, and if it is satisfied, the learning process is terminated. If not, the link prediction model learning unit 36 and the content feature amount model learning unit 40 are used. Repeat the process of.

The model alternating optimization unit 44 is not necessarily an indispensable component, but by adding this component, the link prediction model and the content feature amount model are refined, and a more useful content feature amount is extracted. It becomes possible.

The method of alternate optimization is not particularly limited, but in the present embodiment, a method when the re-learning of VGG-Net is used in the content feature amount model learning unit 40 will be described.

When the re-learning of VGG-Net is used in the content feature amount model learning unit 40, the content basic feature amount extraction unit 30 and the content feature amount calculation unit 42 have substantially the same functions. That is, in any processing unit, VGG-Net is adopted as a feature amount model, and the feature amount is extracted from the input digital content.

Therefore, the model alternating optimization unit 44 alternately optimizes the link prediction model and the content feature model by the following procedure.

(1) The link prediction model learning unit 36 learns a link prediction model using the content basic feature amount or the content feature amount.
(2) The content feature amount model learning unit 40 learns the content feature amount model using a fixed link prediction model.
(3) The content feature amount calculation unit 42 learns the content feature amount using a fixed content feature amount model.
(4) Repeat steps (1) to (3). The repetition is stopped when a predetermined condition is satisfied. As the repetition end condition, conditions such as the number of updates exceeding a predetermined number and the difference between the pre-update parameter and the post-update parameter for the link prediction model and the content feature amount model may be smaller than a predetermined threshold value.

By the above processing, the link prediction model and the content feature amount model are learned.

Using the content feature amount model learned as described above, the content feature amount calculation unit 42 calculates the content feature amount of the new digital content received by the input unit 10 and outputs it to the link prediction unit 46.

The link prediction unit 46 inputs the link prediction model stored in the link prediction model storage unit 22 and the digital content feature amount of the new digital content, and determines whether or not the new digital content belongs to each of the digital content sets. It makes a prediction and outputs the prediction result to the output unit 50.

<Operation of the content feature amount extraction device according to the first embodiment of the present invention>

Next, the operation of the content feature amount extraction device 100 according to the first embodiment of the present invention will be described. When the input unit 10 receives a plurality of digital content sets composed of labeled digital contents, the content feature amount extraction device 100 executes the content feature amount extraction processing routine shown in FIG. 2.

First, in step S100, the content basic feature amount, which is the basic feature amount of the digital content, is extracted for each of the digital contents included in the plurality of digital content sets received by the input unit 10.

In step S102, the content set basic feature amount, which is the basic feature amount of the digital content set, is extracted for each of the plurality of digital content sets received by the input unit 10.

In step S104, a content graph, which is a graph representing which digital content set each of the digital contents is included in, is constructed.

In step S106, the content basic feature amount extracted from each of the digital contents in step S100, the content feature amount extracted in step S110, the content set basic feature amount extracted from each of the plurality of digital content sets in step S102, and the step. Based on the content graph constructed in S104, a link prediction model that predicts the presence or absence of belonging to the digital content set from the content basic feature amount and the content set basic feature amount is learned and stored in the link prediction model storage unit 22.

In step S108, the link prediction model learned in step S106 and the content feature amount model, which is a model for extracting the content feature amount of the digital content from the digital content included in the digital content set, are learned, and the content feature amount is learned. It is stored in the model storage unit 24.

In step S110, the content feature amount is calculated from the digital content included in the digital content set by using the content feature amount model learned in step S108.

In step S112, it is determined whether or not the repetition end condition is satisfied, and if it is satisfied, the process proceeds to step S114, and if it is not satisfied, the processes of steps S106 to S110 are repeated.

In step S114, the content feature amount of the new digital content received by the input unit 10 is calculated using the content feature amount model learned in step S108.

In step S116, the presence or absence of new digital content belonging to each of the digital content sets is predicted using the link prediction model learned in step S106 and the content feature amount calculated in step S114, and the output unit 50 Output to and end the process.

As described above, according to the content feature amount extracting device according to the first embodiment of the present invention, for each of the digital contents included in the digital content set, the content basic which is the basic feature amount of the digital content. A content graph is constructed by extracting features and expressing which digital content set each digital content is included in, and based on the basic content features extracted from each of the digital contents and the content graph. , Learn the link prediction model, which is a model that predicts the presence or absence of belonging to the digital content set from the basic content features of the digital content, and based on the learned link prediction model, the digital content included in the digital content set, and By calculating the content feature amount, which is the feature amount of the digital content, for at least one of the new digital contents, the content feature amount considering the properties of the label can be extracted, and the label is accurately predicted for the new digital content. can do.

<Structure of content feature amount extraction device according to the second embodiment of the present invention>

Next, the configuration of the content feature amount extraction device according to the second embodiment of the present invention will be described. The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

In the second embodiment, the content feature amount is calculated using the learned link prediction model without using the content feature amount model learning unit and the model alternating optimization unit.

As shown in FIG. 3, the content feature amount extraction device 200 according to the second embodiment of the present invention includes a CPU, a RAM, a program for executing a content feature amount extraction processing routine described later, and various data. It can be configured with a computer including a stored ROM. The content feature amount extraction device 200 functionally includes an input unit 210, a calculation unit 220, and an output unit 250 as shown in FIG.

The input unit 210 accepts a plurality of digital content sets composed of labeled digital contents for learning. Further, the input unit 210 accepts a newly given new digital content having an unknown label.

The calculation unit 220 includes a link prediction model storage unit 22, a content basic feature amount extraction unit 30, a content set basic feature amount extraction unit 32, a graph construction unit 34, a link prediction model learning unit 36, and a content feature amount calculation. A unit 242 and a link prediction unit 46 are included.

The content feature amount calculation unit 242 calculates the content feature amount from the digital contents included in the digital content set based on the link prediction model 22 stored in the link prediction model storage unit.

When the content feature amount calculation unit 242 calculates the content feature amount based on the link prediction model, the calculation is performed by a method based on perspective coding. The method based on sparse coding is a method of expressing the content basic features by a sparse linear sum of the content set basic features (and, if available, the content set latent variables). The link prediction model modeled by equation (3) is the basic content feature converted by the model parameters.

And the vectors that characterize the digital content set

Dot product with is large. That is, when both vectors are similar, the link prediction value

・・・（５）
なお、第２の実施の形態の他の構成は第１の実施の形態と同様であるため、説明を省略する。 Shows that becomes larger. Focusing on this point, the weighting coefficient α of the linear sum obtained as the solution of the minimization problem of the following equation (5) is calculated as a new feature amount obtained from the content basic feature amount x, that is, the content feature amount.

... (5)
Since the other configurations of the second embodiment are the same as those of the first embodiment, the description thereof will be omitted.

<Operation of the content feature amount extraction device according to the second embodiment of the present invention>

Next, the operation of the content feature amount extracting device 200 according to the second embodiment of the present invention will be described. The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

When the input unit 210 receives a plurality of digital content sets composed of labeled digital contents, the content feature amount extraction device 200 executes the content feature amount extraction processing routine shown in FIG.

In step S200, based on the link prediction model stored in the link prediction model storage unit 22, the content feature amount, which is the feature amount of the digital content, is calculated for the new digital content according to the above equation (5).

As described above, according to the content feature amount extracting device according to the second embodiment of the present invention, for each of the digital contents included in the digital content set, the content basic which is the basic feature amount of the digital content. A content graph is constructed by extracting features and expressing which digital content set each digital content is included in, and based on the basic content features extracted from each of the digital contents and the content graph. , Learn the link prediction model, which is a model that predicts the presence or absence of belonging to the digital content set from the basic content features of the digital content, and based on the learned link prediction model, the digital content included in the digital content set, and By calculating the content feature amount, which is the feature amount of the digital content, for at least one of the new digital contents, the content feature amount considering the properties of the label can be extracted, and the label is accurately predicted for the new digital content. can do.
<Structure of content feature amount extraction device according to the third embodiment of the present invention>

Next, the configuration of the content feature amount extraction device according to the third embodiment of the present invention will be described. The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

The third embodiment is a case of predicting whether or not the digital content belongs to a new digital content set.

As shown in FIG. 5, the content feature amount extraction device 300 according to the third embodiment of the present invention includes a CPU, a RAM, a program for executing a content feature amount extraction processing routine described later, and various data. It can be configured with a computer including a stored ROM. The content feature amount extraction device 300 functionally includes an input unit 310, a calculation unit 320, and an output unit 350 as shown in FIG.

The input unit 310 accepts a plurality of digital content sets composed of labeled digital contents for learning. Further, the input unit 310 accepts a new digital content set, which is a new digital content set. Further, the input unit 310 accepts a newly given new digital content having an unknown label.

The calculation unit 320 includes a link prediction model storage unit 22, a content feature amount model storage unit 24, a content basic feature amount extraction unit 30, a content set basic feature amount extraction unit 32, a graph construction unit 34, and a link prediction model. It includes a learning unit 36, a content feature amount model learning unit 40, a content feature amount calculation unit 42, a model alternating optimization unit 44, and a new digital content set link prediction unit 340.

The new digital content set link prediction unit 340 uses the new digital content received by the input unit 310 as a latent variable prediction model for the new digital content set corresponding to the link prediction model learned by the link prediction model learning unit 36. , The presence or absence of new digital content belonging to the new digital content set is predicted based on the link prediction model.

The new digital content set link prediction unit 340 makes it possible to consider a new digital content set that was not considered in the link prediction model learning unit 36. If the digital content set is a set of digital content with a particular text label, then link prediction to the new digital content set gives the digital content a text label that was not considered during training. It corresponds to the process of deciding whether to do it.

The means for predicting the link to the new digital content set is not particularly limited, but in the present embodiment, a method assuming the following situations will be described.

Some text information is included in the new digital content set, and the digital content set basic feature amount can be extracted by using the method described in the content set basic feature amount extraction unit 32. Further, it is assumed that the new digital content set is an empty set, that is, the new digital content set does not contain any digital content. Further, although digital contents may be included, they are not used in the methods shown below.

The new digital content set link prediction unit 340 includes a latent variable prediction model learning unit 343, a latent variable prediction unit 344, and a link prediction unit 346.

The latent variable prediction model learning unit 343 is a part of the model parameters of the link prediction model from the link prediction model stored in the link prediction model storage unit 22 and the digital content set basic feature quantity used in the link prediction model learning unit 36. Learn a latent variable prediction model, which is a model for predicting latent variables.

The link prediction value, which is a score indicating whether or not the digital content i belongs to the digital content set c, is the digital content basic feature amount x _i , the digital content set basic feature amount y _c , and the digital as shown in the above equation (3). Calculated from the content set latent variable z _Cc .

However, in the new digital content set c', although the digital content set basic feature quantity y _c'can be calculated, the digital content set latent variable z _Cc' cannot be used. Therefore, using the digital content set basic feature quantity Y used in the link prediction model learning unit 36 and the digital content set latent variable Z _C which is the model parameter of the link prediction model learned in the link prediction model learning unit 36, the following Consider a model that predicts a digital content set latent variable from the digital content set basic feature quantity of Eq. (6), and obtain this latent variable prediction model f (・; θ) by learning.

・・・（６）

... (6)

Here, θ is a model parameter of the latent variable prediction model. That is, the above-mentioned latent variable prediction model f (.; θ) predicts z _Cc'from y _c' . In this learning, the conversion function f () from the content set basic feature quantity y to the digital content set latent variable z _Cc is obtained by using the digital content set basic feature quantity Y and the digital content set latent variable Z _C as training data. be.

The configuration of the latent variable prediction model is not particularly limited, but the following methods can be considered, for example.

A simple linear regression model is adopted, the link prediction model is fixed, and the model parameter θ of the following equation (7) is obtained by learning.

・・・（７）

... (7)

The formula obtained by substituting the linear regression model of the above formula (7) into the link prediction model of the formula (3) is adopted as a new link prediction model, and the link prediction model is relearned.

Further, the link prediction model is relearned by adding a new regularization term in consideration of the above linear regression model to the loss function (4) of the link prediction model. At this time, the loss function equation (4) of the link prediction model is modified as the following equation (8).

・・・（８）

... (8)

A nonlinear regression model such as support vector regression is adopted, the link prediction model is fixed, and the model parameters of the nonlinear regression model are obtained by learning.

A neural network model is adopted, the link prediction model is fixed, and the model parameters of the neural network are obtained by learning.

The latent variable prediction unit 344 inputs the latent variable prediction model learned by the latent variable prediction model learning unit 343 and the new digital content set, extracts the content set basic feature quantity from the new digital content set, and extracts the new digital content set. The content set latent variable is predicted from the content set basic feature quantity and the latent variable prediction model, and the content set latent variable for the new digital content set is output.

The latent variable prediction unit 344 first extracts the content set basic features from the new digital content set. The extraction method is the same as the method described in the content set basic feature amount extraction unit 32.

The latent variable prediction unit 344 subsequently obtains the content set of the following equation (9) from the new content set basic feature amount y _c'which is the extracted content set basic feature amount and the latent variable prediction model f (・; θ). Predict latent variables.

・・・（９）

... (9)

The link prediction unit 346 includes a link prediction model stored in the link prediction model storage unit 22, a predicted content set latent variable which is a predicted content set latent variable, a digital content feature amount for new digital content, and a new content set basic. Using the feature amount as an input, the presence or absence of belonging of the new digital content to each of the digital content sets including the new digital content set is predicted, and the prediction result is output to the output unit 250.

The prediction of the presence or absence of the new digital content i'belonging to the new digital content set c'can be realized by using the link prediction model (3). However, since the content set latent variable related to the new digital content set is unknown in the equation (3), the predicted content set latent variable in the following equation (10) is used instead.

・・・（１０）

... (10)

Link predicted value obtained by this equation (10)

When a value larger than a certain value is taken, it is determined that the digital content i'belongs to the new digital content set c'.

Through the processing of each of the above units, the new digital content set link prediction unit 340 predicts whether or not the digital content belongs to the new digital content set, and outputs the prediction result for the new digital content set.
<Operation of the content feature amount extraction device according to the third embodiment of the present invention>

Next, the operation of the content feature amount extraction device 300 according to the third embodiment of the present invention will be described. The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.
<Operation of the content feature amount extraction device according to the third embodiment of the present invention>

Next, the operation of the content feature amount extraction device 300 according to the third embodiment of the present invention will be described. The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

When the input unit 310 receives a plurality of digital content sets composed of labeled digital contents, the content feature amount extraction device 300 executes the content feature amount extraction processing routine shown in FIG.

In step S302, a latent variable that is a part of the model parameter of the link prediction model is predicted from the link prediction model stored in the link prediction model storage unit 22 and the digital content set basic feature quantity used for learning the link prediction model. Learn the latent variable prediction model that is the model to do.

In step S304, the latent variable prediction model learned in step S302 and the new digital content set are input, the content set basic feature amount is extracted from the new digital content set, and the content set basic feature amount for the new digital content set, And the content set latent variable is predicted from the latent variable prediction model, and the content set latent variable for the new digital content set is output.

In step S306, the link prediction model stored in the link prediction model storage unit 22, the predicted content set latent variable which is the predicted content set latent variable, and the digital content feature amount for the new digital content are input, and new from these. It predicts whether or not new digital content belongs to each of the digital content sets including the digital content set, and outputs the prediction result.

As described above, according to the content feature amount extracting device according to the third embodiment of the present invention, for each of the digital contents included in the digital content set, the content basic which is the basic feature amount of the digital content. A content graph is constructed by extracting features and expressing which digital content set each digital content is included in, and based on the basic content features extracted from each of the digital contents and the content graph. , Learn the link prediction model, which is a model that predicts the presence or absence of belonging to the digital content set from the basic content features of the digital content, and based on the learned link prediction model, the digital content included in the digital content set, and By calculating the content feature quantity, which is the feature quantity of the digital content, and predicting the latent variable of the new digital content set, the content feature quantity considering the nature of the label can be extracted for at least one of the new digital content. , Labels can be predicted accurately for new digital contents.

<Experimental results>

In order to verify the embodiments shown so far, a data set was independently collected, and a link prediction model and a content feature amount model were trained using this data set. This dataset is calibrated from a set of 65,000 digital contents and 1.5 million digital contents (still images). FIG. 7 shows statistics on the number of digital content sets to which each digital content belongs, and FIG. 8 shows statistics on the number of digital contents included in each digital content set.

In the verifications shown below, the output of the 15th layer (FC7) of VGG-Net, 4096 dimensions, was used as the content basic feature quantity, and the text-embedded 300 dimensions described in Non-Patent Document 4 was used as the content set basic feature quantity. A method of learning a link prediction model using each of these basic features and a content graph constructed from a dataset, a method of extracting content features directly from the link prediction model, and a content feature using a link prediction model. Two methods, a method of extracting content features after learning the model, were carried out.

In order to evaluate the performance of the content features extracted in this way, an experiment using a public image data set was conducted. The data set used is as follows.

-Data set for classification of meal images: UECFOOD100, UECFOOD256
-Data set for classification of clothing images: Apparel, Hipster
・ Data set for the purpose of predicting the impression (positive or negative) received from the image: Instagram

In addition to these public datasets, we also prepared a dataset originally collected from Pinterest, which is one of the SNSs for the purpose of sharing images, for evaluation. This dataset contains 32 class labels and 63,000 images, as well as various categories of class labels such as women's fashion, holiday's events, tattoos, science and nature, and sports. Also, the task is multi-label prediction, where each image may have multiple class labels.

In the evaluation experiment, the following seven methods were compared.

1.VGG: The basic feature amount of the content is used as it is as the feature amount of each task.
2.FT-GRP: Instead of using the link prediction model, VGG-Net is relearned using the digital content set as the correct label, and the relearned VGG-Net is used as the feature of each task.
3.FT-WORD: Instead of using the link prediction model, VGG-Net is relearned using the text word contained in the digital content set as the correct label, and VGG-Net after this relearning is used as the feature of each task. use.
4.PROP-SC: A method of extracting content features directly from a link prediction model using sparse coding without using a content feature model in the above embodiment.
5.PROP-FT: A method of learning a content feature amount model and extracting a content feature amount from this content feature amount model in the above embodiment.
6.VGG + SC: A method of using PROP-SC content features and VGG content features in combination.
7.VGG + FT: A method of concatenating and using PROC-FT content features and VGG content features.

As evaluation indicators, Instagram dataset with binary classification as a task and Pinterest with multi-label prediction as a task have mean average precision (MAP), and UECFOOD100 / 256 with multi-class classification as a task, Apparel, Hipster are classified correctly. The rate (ACC) was used. In each index, 0 is the minimum value and 1 is the maximum value, and the larger the value, the better the task performance.

Table 1 shows the results of the verification experiment.

VGG trained using large-scale object recognition data sets showed excellent performance in each data set, but PROP-SC and PROP-FT trained using web image data also obtained similar performance. This shows the usefulness of the method shown in the above embodiment. In addition, VGG + SC and VGG + FT outperformed VGG. This means that the content features extracted by the method shown in the above-described embodiment can be extracted with characteristics that are significantly different from the features learned by the large-scale object recognition data set, and it can be extracted in various domains. Shows that it is applicable.

The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

For example, in each of the above-described embodiments, the content feature amount extraction in which the learning of the link prediction model and the content feature amount model, the extraction of the content feature amount, and the link prediction are performed by the same device has been described, but the present invention is limited to this. It's not something. For example, it may be divided into a device for learning a link prediction model and a content feature amount model, and a device for extracting a content feature amount and performing link prediction.

10, 210, 310 Input unit 20, 220, 320 Calculation unit 22 Link prediction model storage unit 24 Content feature amount model storage unit 30 Content basic feature amount extraction unit 32 Content set basic feature amount extraction unit 34 Graph construction unit 36 Link prediction model Learning unit 40 Content feature amount model Learning unit 42 Content feature amount calculation unit 44 Model alternating optimization unit 46, 246 Link prediction unit 50, 250, 350 Output unit 100, 200, 300 Content feature amount extraction device 243 Content feature amount calculation unit 340 New Digital Content Set Link Prediction Unit 342 Latent Variable Prediction Model Learning Unit 344 Latent Variable Prediction Unit

Claims (7)

Using a plurality of digital content sets consisting of digital contents prepared in advance, at least one feature amount of the digital contents included in the digital contents set and the new digital contents given separately from the digital contents set is extracted. , In the content feature amount extraction method in the content feature amount extraction device,
A step in which the content basic feature amount extraction unit extracts the content basic feature amount, which is the basic feature amount of the digital content, for each of the digital contents included in the digital content set.
A step in which the graph construction unit constructs a content graph, which is a graph expressing which digital content set each of the digital contents is included in.
The link prediction model learning unit predicts the presence or absence of belonging to the digital content set from the content basic feature amount of the digital content based on the content basic feature amount extracted from each of the digital contents and the content graph. Steps to learn the link prediction model, which is a model,
Based on the learned link prediction model, the content feature amount calculation unit calculates the content feature amount, which is the feature amount of the digital content, for at least one of the digital content included in the digital content set and the new digital content. Steps to do and
Content feature extraction method including. The content feature amount extraction method further comprises
The content set basic feature amount extraction unit comprises a step of extracting the content set basic feature amount, which is the basic feature amount of the digital content set, for each of the plurality of digital content sets.
The step of the link prediction model learning unit is based on the content basic feature amount extracted from each of the digital contents, the content set basic feature amount extracted from each of the plurality of digital content sets, and the content graph. The content feature amount extraction method according to claim 1, wherein a link prediction model for predicting the presence or absence of belonging to a content set from the content basic feature amount and the content set basic feature amount is learned. The content feature amount extraction method further comprises
The content feature amount model learning unit includes a step of learning a content feature amount model which is a model for extracting the content feature amount of the digital content from the link prediction model and the digital content included in the digital content set.
The step of the content feature amount calculation unit is a request to calculate the content feature amount from at least one of the digital content included in the digital content set and the new digital content by using the learned content feature amount model. The content feature amount extraction method according to item 1 or claim 2. The content feature amount extraction method further comprises
The model alternating optimization unit optimizes the link prediction model and the content feature amount model by alternately and repeatedly executing the steps of the link prediction model learning unit and the steps of the content feature amount model learning unit. With steps,
The step of the link prediction model learning unit is based on the content feature amount extracted from each of the digital contents using the learned content feature amount model and the presence or absence of belonging to the digital content set based on the content graph. The content feature amount extraction method according to claim 3, wherein the link prediction model, which is a model for predicting the content from the content feature amount of the digital content, is learned. The content feature amount extraction method further comprises
New digital content set The link prediction unit predicts the latent variables included in the link prediction model, which has been learned in advance for the new digital content set, which is a newly given digital content set, based on the latent variable prediction model. Claims 1 to include a step of predicting a latent variable for a new digital content set and predicting whether or not the digital content belongs to the new digital content set based on the predicted latent variable and the link prediction model. The content feature amount extraction method according to any one of claims 4. Using a plurality of digital content sets consisting of digital contents prepared in advance, at least one feature amount of the digital contents included in the digital contents set and the new digital contents given separately from the digital contents set is extracted. , In the content feature amount extraction device
For each of the digital contents included in the digital content set, a content basic feature amount extraction unit for extracting the content basic feature amount which is the basic feature amount of the digital content, and a content basic feature amount extraction unit.
A graph construction unit that constructs a content graph, which is a graph expressing which digital content set each of the digital contents is included in, and a graph construction unit.
A link prediction model, which is a model for predicting the presence or absence of belonging to the digital content set from the content basic feature amount of the digital content based on the content basic feature amount extracted from each of the digital contents and the content graph. Link prediction model learning department to learn,
A content feature amount calculation unit that calculates a content feature amount, which is a feature amount of the digital content, for at least one of the digital content included in the digital content set and the new digital content based on the learned link prediction model. ,
Content feature quantity extraction device including. A program for causing a computer to execute each step of the content feature amount extraction method according to any one of claims 1 to 5.

Images