JP5545877B2 - Content recognition model learning apparatus, content recognition model learning method, and content recognition model learning program - Google Patents

Description

  The present invention learns a content recognition model for estimating meaning from media data (contents) such as audio signals, sound signals, still images and moving images (videos), and text information manually assigned thereto. Regarding technology. Here, “meaning” is information obtained by combining information such as an audio signal, an acoustic signal, an object, an action, an action, and a scene included in a still image or video.

  2. Description of the Related Art Conventionally, video recognition technology has been developed that automatically gives language information that describes a given video to the language. In recent years, with the widespread use of imaging devices such as digital video cameras and mobile phones, and generalization of video sharing on the Internet, such video recognition technology has become a very important technology.

  Also, learning a topic model, which is a statistical model that links two observation information using latent variables, and using this topic model, an image labeling that automatically assigns an appropriate text label to a given image, There has been proposed a technique that uniformly handles image acquisition for finding an appropriate image from a given text label (see, for example, Non-Patent Documents 1 and 2).

  On the other hand, a method (dynamic learning method) that updates the model for understanding video by imitating the process of human understanding of video, presenting questions about the contents of images (video), and obtaining answers from users ) Has been proposed (for example, Non-Patent Document 3).

Nakayama, Harada, Kuniyoshi, Otsu “Ultra-high-speed image recognition / retrieval method using stochastic structure learning corresponding to image / word concept”, IEICE Technical Report, PRMU 2007-147, December 2007 Kimura, Nakano, Sugiyama, Kameoka, Maeda, Sakano "SSCDE: Semi-supervised canonical density estimation method for image recognition search", Image Recognition and Understanding Symposium MIRU2010, OS8-1, July 2010 Siddiquie, B. and Gupta, A .: Beyond active noun tagging: Modeling contextual interactions for multi-class active learning, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2979-2986, 2010.

  However, in the video recognition in the methods of Non-Patent Documents 1 and 2, there is a problem that learning accuracy is lowered due to the influence of noise included in the additional information. Here, the noise included in the additional information means that a word indicating an object that does not exist in the content is included in the additional information (text sentence) or an expression corresponding to it in the text sentence even though it exists in the content. (Label) does not exist. In order to improve the learning accuracy, a large amount of teacher data is required, but there is a problem that it is very costly to prepare additional information manually so as not to include the noise as described above.

In addition, according to the technique of Non-Patent Document 3, since the computer automatically generates a question regarding a region that the computer wants to study in the content and proceeds with learning while obtaining an answer about the region, the learning accuracy is improved. Can reduce the burden on people. Non-Patent Document 3 proposes the following three types of questions (ac).
(A) Question that asks the user to specify which area in the content corresponds to the object (for example, “bot”) that the computer wants to learn (b) Area that the computer could not recognize (indeterminate area) ) Is a question that asks the user using the relative relationship with the area (determined area) that the computer can recognize (for example, “what is above water?” Or “what is bright than what?”)
(C) A question that asks for a word (label) representing the relative relationship between two objects that can be recognized by the computer (eg, “what is the relation between boat and water?”)

  As described above, the technique of Non-Patent Document 3 is based on the “What” format question that identifies what the uncertain region is, and the degree of freedom of answers that can be input by the user is very high. For this reason, there are problems that different labels are given to the same object (indeterminate region) and that the types of labels become enormous.

  The present invention has been made in view of such circumstances, and based on the certainty of recognition of an indeterminate region obtained from already learned knowledge, not only “What” but also “Which” or “Is this” format. Content recognition model learning apparatus, content recognition model learning method, and content capable of performing high-accuracy learning while limiting the degree of freedom of user answers and solving the problems of the prior art The purpose is to provide a recognition model learning program.

  The present invention includes a content recognition model storage unit that stores content recognition model information for recognizing content data, a content acquisition unit that acquires content data including an audio signal or a video signal, and the content recognition model storage unit. Using the content recognition model stored in the content data, additional information estimation means for estimating additional information indicating the meaning of the audio signal or video signal included in the content data to be added to the content data, and the additional information estimation means The certainty factor calculating means for obtaining the certainty factor of the additional information to be given to the content data estimated by the above and the question for determining the additional information to be given to the content data are obtained by the certainty factor calculating means. Quality based on the certainty and the selected question to display A model for updating information on the content recognition model stored in the content recognition model storage unit based on information on a response acquired by a display unit, an answer acquisition unit that acquires an answer to the question, and the answer acquisition unit And updating means.

  In the present invention, when the certainty factor is high, the question display means displays only a question asking whether or not the additional information to be added to the content data matches the estimated additional information. When the degree is medium, only the questions that narrow down the additional information to be added to the content data from the estimated additional information candidates are displayed. When the certainty level is low, the additional information should be given to the content data A question asking what the additional information is is displayed.

  The present invention includes a content recognition model storage unit storing content recognition model information for recognizing content data, a content acquisition unit, an additional information estimation unit, a certainty factor calculation unit, a question display unit, an answer A content recognition model learning method in a content recognition model learning apparatus comprising an acquisition unit and a model update unit, wherein the content acquisition unit acquires content data including an audio signal or a video signal; and The additional information estimating means estimates additional information indicating the meaning of the audio signal or video signal included in the content data to be added to the content data, using the content recognition model stored in the content recognition model storage means. Additional information estimating step, and the certainty factor calculating means A certainty factor calculating step for obtaining a certainty factor of additional information to be added to the content data estimated in the additional information estimating step; and the question display means for determining the additional information to be added to the content data. A question display step for selecting the question based on the certainty factor obtained by the certainty factor calculation step, displaying the selected question, and an answer obtaining step for the answer obtaining means to obtain an answer to the question; The model update means includes a model update step for updating the content recognition model information stored in the content recognition model storage means based on the answer information obtained by the answer obtaining step. To do.

  In the present invention, in the question display step, when the certainty factor is high, only the question asking whether the additional information to be added to the content data matches the estimated additional information is displayed. When the degree is medium, only the questions that narrow down the additional information to be added to the content data from the estimated additional information candidates are displayed. When the certainty level is low, the additional information should be given to the content data A question asking what the additional information is is displayed.

  The present invention includes a content recognition model storage unit storing content recognition model information for recognizing content data, a content acquisition unit, an additional information estimation unit, a certainty factor calculation unit, a question display unit, an answer A content recognition model learning program for causing a computer on a content recognition model learning apparatus including an acquisition unit and a model update unit to perform a content recognition model learning process, and acquiring content data including an audio signal or a video signal Using the content acquisition step and the content recognition model stored in the content recognition model storage means, extra information indicating the meaning of the audio signal or video signal included in the content data to be added to the content data is estimated. Additional information estimation step and the additional information A certainty factor calculating step for determining the certainty factor of the additional information to be added to the content data estimated in the determination step, and a question for determining the additional information to be added to the content data are obtained by the certainty factor calculating step. Based on the information of the answer acquired by the question display step for selecting based on the given confidence and displaying the selected question, the answer acquiring step for acquiring the answer to the question, and the answer acquiring step. A model update step of updating information of the content recognition model stored in the content recognition model storage means is performed by the computer.

  According to the present invention, by changing the type of question according to the certainty of the estimated additional information, the misidentification rate can be improved as compared with the case where only a single type of question is performed. In addition, since the system automatically generates a question for extracting information useful for improving the accuracy of learning, an effect of reducing a human load can be obtained.

It is a block diagram which shows the structure of one Embodiment of this invention. It is a flowchart which shows operation | movement of the apparatus shown in FIG. It is explanatory drawing which shows operation | movement of the apparatus shown in FIG. It is explanatory drawing which shows operation | movement of the apparatus shown in FIG. It is explanatory drawing which shows operation | movement of the apparatus shown in FIG. It is explanatory drawing which shows an example of the screen which displays a question. It is explanatory drawing which shows the experimental result of the apparatus by this invention. It is explanatory drawing which shows the experimental result of the apparatus by this invention.

  Hereinafter, a content recognition model learning apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In the following description, content refers to media data such as audio signals, audio signals, still images and moving images (video), and a set of additional information refers to information indicating the content of the content (describes the content and structure of the video) And text data representing information on the time and place where the video was shot). In FIG. 1, reference numeral 1 denotes an initial content feature set X = {x1, x2,... For a set of N contents (this is called an initial content set) G = {g1, g2,. , XN} is an initial content feature set storage unit. Reference numeral 2 denotes an initial additional information feature set storage unit in which an initial additional information feature set Y = {y1, y2,..., YN} is stored.

  Reference numeral 3 denotes an initial model learning unit that learns the initial value of the content recognition model from the initial content feature set X and the initial additional information feature set Y for the initial content set G. Reference numeral 4 denotes a content recognition model storage unit that stores content recognition model information. Reference numeral 5 denotes a new content acquisition unit that acquires new content gN + 1 that is not necessarily included in the initial content set G and calculates its feature xN + 1. The new content acquisition unit 5 ends the process when there is no new content.

  Reference numeral 6 denotes an addition for estimating additional information for the new content gN + 1 and the initial contents g1, g2,..., GN acquired by the new content acquisition unit 5 using the content recognition model stored in the content recognition model storage unit 4. It is an information estimation part. Reference numeral 7 denotes a content in the initial content set G similar to the new content acquired by the new content acquisition unit 5 as a neighborhood sample, and a set of neighborhood samples ^ G (^ is attached to the head of G. Are similar samples) = {^ g1, ^ g2,..., ^ GH} (H is the number of neighboring samples).

  Reference numeral 8 denotes a label (additional information feature) ^ y1, ^ y2,..., ^ YH estimated by a learned content recognition model for each neighboring sample included in the neighboring sample set ^ G, and the degree of variation in the estimated label. It is a question selection unit that selects a question to be presented to the user regarding the content (additional information) of the input new content gN + 1 according to (this is called confidence). Reference numeral 9 denotes a display unit constituted by a display device such as a display for displaying the question information selected by the question selection unit 8. Reference numeral 10 denotes an input unit configured from an input device such as a keyboard or a mouse for inputting answer information to a question displayed on the display unit 9. Note that an input device that acquires audio signals, Web information, and the like may be used instead of the keyboard and mouse.

  Reference numeral 9 acquires the answer information input from the input unit 10, determines a new additional information feature yN + 1 for the input new content gN + 1 based on the obtained answer information, and sets the feature xN + 1 for the input new content as the initial content set And the additional information feature set Y (1) = Y∪ {yN + 1} obtained by adding the new additional information feature YN + 1 determined for the input new content and the content feature set X (1) = X∪ {xN + 1} This is an answer acquisition unit to output. Here, X∪ {xN + 1} represents the union of the set X and {xN + 1}, and the number (1) on the right shoulder of X and Y represents the number of repetitions of the update process. Reference numeral 12 denotes a content feature set storage unit that stores information on the content feature set X output from the answer acquisition unit 11. No. 13 is an additional information feature set storage unit that stores information of the additional information feature set Y output from the answer acquisition unit 11.

  Reference numeral 14 denotes the number of new contents (number of new content features or new additional information features) newly added to the learning data (content feature set X and additional information feature set Y) at the time of creating the content recognition model. ) And when the number of added new contents reaches a predetermined number, the model update control unit shifts the process and updates the content recognition model. When the number of new contents does not satisfy the predetermined standard, control is performed so as to shift the process to the new content acquisition unit and repeat the process.

  Reference numeral 15 denotes an updated content feature set X (i) and additional information feature set Y (i) stored in the content feature set storage unit 12 and the additional information feature set storage unit 13 based on an instruction from the model update control unit 14. ) To update the content recognition model stored in the content recognition model storage unit 4. The model update process is initial except that the input data uses X (i) (i is a natural number) and Y (i) (i is a natural number) instead of the initial content feature set and the initial additional information feature set. This is the same processing as the model learning unit 3. When the process of the model update unit 15 is completed, the process proceeds to the new content acquisition unit 5 and the update of the content recognition model is repeated for new content.

  Note that the model update control unit 14 may be omitted, and in this case, the content recognition model is sequentially updated every time one new content is added. The model update control unit 14 can reduce the number of times of learning by updating the content recognition model at a stage when the answers to the new content are gathered to some extent. A simple content recognition model can be created.

  Next, an operation in which the initial model learning unit 3 learns the content recognition model based on information stored in the initial content feature set storage unit 1 and the initial additional information feature set storage unit 2 illustrated in FIG. 1 will be described. A content set G = {g1, g2,..., GN} given in advance for learning the first content recognition model is set as an initial content set. A set of additional information A = {a1, a2,..., AN} associated with each content included in the initial content set G is set as an initial additional information set. A set of feature amounts extracted for each content included in the initial content set G becomes an initial content feature set X = {x1, x2,..., XN}, and a set of feature amounts extracted for each additional information in the initial additional information set A The initial additional information feature set Y = {y1, y2,..., YN}.

  When the target content is an image, the initial content features include, for example, a color histogram, an arbitrary component of digital cosine transform, an arbitrary component of Haar wavelet, a higher-order local autocorrelation feature (N. Otsu and T. Kurita "A new Scheme for practical flexible and intelligent vision systems, "Proc. IAPR Workshop on Computer Vision, pp.431-435, 1988.), Bag of Features representation of feature points selected by an arbitrary method (Li Fei-Fei et al. , 2005.) can be used. In addition, as the initial additional information feature, a binary vector expressing the presence or absence of a word (label) included in the additional information can be used. This is a vector having the same number of dimensions as the total number of possible words, and each dimension of the vector corresponds to the possible word. If the word i is included in the additional information, the vector i The value of the element is 1, and when it is not included, the value is 0.

  Alternatively, a vector expressing the number of occurrences of words included in the additional information (word occurrence vector) or a vector expressing the appearance probability of each topic can be used with the number of topics specified in advance as the number of dimensions. . This vector can be calculated using topic models such as probabilistic latent semantic analysis (pLSA) and latent dirichlet allocation (LDA). Roger Levy, Nuno Vasconcelos: “A new approach to cross-modal multimedia retrieval.” See ACM Multimedia 2010, pp.251-260.

  Further, when the target content is an audio signal or a video signal, a method such as a time-series active search method (TAS, see Japanese Patent No. 30653314) or a division match search method (DAL, see Japanese Patent No. 4327202). The feature quantity of the audio or video signal extracted by using can be used.

  First, the initial model learning unit 3 stores the initial content feature set X = {x1, x2,..., XN} stored in the initial content feature set storage unit 1 and the initial additional information feature set storage unit 2. Initial additional information feature set Y = {y 1, y 2,..., YN}, and using the information of these sets, the most appropriate additional information representing the content of the additional information that is unknown is used. Learn the content recognition model to determine.

The model learning process may use a known technique described in Non-Patent Documents 1 and 2. For example, first, a latent variable Z = {z1, z2,..., ZN} is generated from the feature quantity set (X, Y). The latent variables are the method using canonical correlation analysis (Non-Patent Document 1), the method using probabilistic canonical correlation analysis (reference: Nakayama et al.) Image annotation retrieval method for large-scale Web images from Web collective intelligence Toward autonomous image knowledge acquisition ", Image Recognition / Understanding Symposium MIRU 2009, OS2-4, July 2009), a method using semi-supervised canonical correlation analysis (Non-Patent Document 2), etc. . Next, Kernel density estimation (KDE), literature: Parzen, E .: On estination of a probability density function and mode, The annuals of Mathematical Statistics, vol. 33, No. 3, pp. 1065-1076 1962.) or semi-supervised kernel density estimation (SSKDE, see Non-Patent Document 2) or the like, the content recognition model can be learned. Specifically, the model learning means the conditional occurrence probability p (x | z _i ) of the content feature x when the latent variable z _i is given, and the conditional occurrence probability p (y |) of the additional feature y. It means obtaining a model parameter of z _i ).

  z1 to zN are called latent variables, and can be considered as, for example, category labels to which content belongs. The content recognition model after learning returns a latent variable zi (1 ≦ i ≦ N) when a feature xi of a given content gi and a feature yi of additional information are input. Here, since xi and yi are each represented by a multidimensional vector, a multidimensional vector corresponding to zi can be obtained by linear transformation using xi and yi.

ここで、Ｄｙは、付加情報特徴ｙｉ（ｉ＝１，２，…，Ｎ）を表すベクトルの次元数（要素数）である。 Next, the operation when the content recognition model learning apparatus shown in FIG. 1 acquires new content will be described with reference to FIG. First, the new content acquisition unit 5 acquires new content from the outside (step S1), and outputs it to the additional information estimation unit 6 and the neighborhood sample extraction unit 7. In response to this, the additional information estimation unit 6 uses the content recognition model stored in the content recognition model storage unit 4 for the new content gN + 1 acquired by the new content acquisition unit 5 and the initial contents g1, g2,. It is used to estimate additional information (step S2). Specifically, for the feature xN + 1 of the content gN + 1, the additional information feature ^ yN + 1 is estimated by the expression (1).

Here, Dy is the number of dimensions (number of elements) of the vector representing the additional information feature yi (i = 1, 2,..., N).

  Next, the neighborhood sample extraction unit 7 estimates the latent variable zN + 1 estimated for the new content gN + 1 acquired by the new content acquisition unit 5 and the latent variable zi (i = 1, 2,...) Estimated for each of the initial contents. , N), the initial content having a latent variable whose similarity exceeds a predetermined threshold (high similarity) or whose similarity is higher than the upper predetermined rank is set as a neighborhood sample, and a set of neighborhood samples ^ G = {^ g1, ^ g2,..., ^ GH} (H is the number of neighboring samples) is extracted (step S3). The similarity of latent variables is defined by, for example, the reciprocal of the distance between multidimensional vectors (for example, Euclidean (L2) distance, Mahalanobis distance, Manhattan (L1) distance).

  Next, the question selection unit 8 labels (additional information feature) ^ y1 for each neighboring sample included in the neighboring sample set ^ G = {^ g1, ^ g2,. , ^ Y2, ..., ^ yH, and a question to be presented to the user about the content (additional information aN + 1) of the new content gN + 1 acquired by the new content acquisition unit 5 according to the estimated degree of label variation (confidence) Is generated. The confidence of the label of the new content is estimated for each element in the additional information feature ^ yN + 1 estimated for the new content gN + 1 and the neighborhood sample set ^ G = {^ g1, ^ g2,. This is an index indicating how much the additional information feature ^ yj (j = 1, 2,..., H) is consistent. Then, the question selection unit 8 determines whether or not the certainty factor is one of “high,“ medium ”, and“ low ”(step S4).

  As a result of this determination, the question selection unit 8 is convinced that there is a sufficient number of learning data (initial content) in the vicinity of the new content, and the additional information (label) of these learning data is consistent. A question (for example, “Is this A?”) Asking whether or not the label for the new content is “A” is generated and displayed on the display unit 9 (step S5). Here, “A” is a label for a neighboring sample set and is consistent, so there is only one type (A) (see FIG. 3).

  Further, when the determination result shows that the certainty factor is medium, the question selection unit 8 has a sufficient number of learning data in the vicinity of the input sample, but sufficient consistency is seen in the labels of the learning data. No, that is, there are multiple candidate labels for the training data and it is difficult for the computer to automatically determine which one is appropriate, and among the labels estimated for each element in the neighborhood sample set, The most frequent label is “A”, the second most frequent label is “B”, and a question asking whether the label for the new content is A or B (for example, “Which is this, A or B ? ") Is generated and displayed on the display unit 9 (step 6: see FIG. 4).

  Further, when the determination result shows that the certainty factor is low, the question selection unit 8 has a small number of learning data in the vicinity of the input sample, and the reliability of the labels of the learning data is low, that is, the label of the learning data is A question (for example, “What is this?”) That asks what is the label for the new content is generated and displayed on the display unit 9 (see step S7: FIG. 5). .

  In general, content often includes a plurality of objects. Therefore, it is desirable that the label to be given to the content is a combination of a plurality of labels (a combination of labels corresponding to each object). In this case, what is necessary is just to produce | generate a question with the following procedures.

  First, a label (word) that does not exist in the additional information corresponding to the neighboring sample is deleted from the additional information feature yN + 1 estimated from the new content gN + 1. That is, the feature amount corresponding to this label is set to zero. If no label remains in ^ yN + 1 (in the case of a zero vector, that is, if no neighboring sample has a label corresponding to each element of ^ yN + 1), an addition estimated from the new content Since it can be determined that the information feature is not always appropriate or the reliability of the estimated additional information feature is low and corresponds to the above-mentioned “in case of low confidence”, a question asking what is the label for the new content ( For example, “What is this?”) Is generated.

  On the other hand, if a label remains in ^ yN + 1 (if it is not a zero vector), the latent variable ratio ^ ZN + 1 for the new content gN + 1 and each latent variable ratio ^ Zj (j = 1, 2,... Calculate the average distance to H). If the average distance exceeds a preset threshold value, it can be determined that the number of learning data in the vicinity of the input sample is extremely small, and corresponds to the above-mentioned “when confidence is low”. Generate a question that asks what is (eg, “What is this?”).

  When the average distance is equal to or smaller than a preset threshold value, possible combinations of individual labels corresponding to the neighboring sample set are generated. Then, the similarity of the generated label combination is calculated. The similarity of the combination is calculated by the average of the association degree of all the label sets in the combination. For example, when yA is an additional information feature corresponding to the label A and S (yA, yB) is the association of the label set (A, B), the similarity of the output of the multi-label ABC is S (yA , YB), S (yB, yC), and S (yC, yA). S (yA, yB) is the number of contents to which labels yA and yB are assigned at the same time.

  The calculated similarity is compared with a preset threshold value, and if there is only one combination whose similarity exceeds the threshold value, it corresponds to the above-mentioned “when the certainty factor is high”. The question (for example, “Is this A?”) That asks whether or not is generated is generated. Here, A includes a label included in the combination whose similarity exceeds the threshold value.

  On the other hand, if there are two or more combinations whose similarity exceeds the threshold value, it corresponds to the above-mentioned “when the certainty level is medium”, so it is asked whether the label for the new content is A or B. Create a question (eg, “Which is this, A or B?”). Here, in A and B, a label included in the combination having the highest similarity and a label included in the combination having the second highest similarity are entered.

  In this way, when the certainty level is high, only the question to confirm whether or not it matches the estimated label is performed, and when the certainty level is medium, the estimated label candidates are narrowed down. Since only the question is asked, and the information about the label is displayed without using the estimated label information only when the certainty level is low, the risk of giving a different label to the same object or the same concept is displayed. By avoiding this, it is possible to construct a content recognition model more precisely.

  Next, the answer acquisition unit 11 acquires an answer input by the user from the input unit 10 with respect to the question displayed on the display unit 9, and generates an additional information feature yN + 1 corrected according to the answer. When the certainty factor is high, a question as to whether or not the label is A is displayed. Therefore, the answer from the user is input as Yes (positive) or No (negative) (see FIG. 6). Note that FIG. 6 is an example of display, and the user can directly input a Yes or No label without being limited to the format of selecting the Yes and No radio buttons. In the case of No, the user can freely input the correct answer label. The answer acquisition unit 11 determines whether the answer is Yes or No (step S8). If a positive answer (Yes) is obtained, the label is assumed to be A and corresponds to A. The additional information feature yN + 1 is generated with the feature amount of the label to be set as 1 and the remaining as 0 (step S9).

On the other hand, if a negative answer (No) is obtained, it is determined that the system prerequisite knowledge (learned knowledge) is incorrect, and the additional information feature is corrected in accordance with the user's answer. That is, the process proceeds to step S7 in order to obtain a correct label from the user, and to generate an additional information feature y _{N + 1 in} which the feature amount corresponding to the label is 1 and the remaining is 0. At this time, if the correct label input by the user in the case of No is an unknown label for the system, the dimension of the additional information feature yN + 1 is increased in order to express the additional information feature corresponding to the label.

  In addition, when the certainty level is medium, since the question as to whether the label is A or B is displayed, the answer from the user is input as either A or B. (See FIG. 6). The answer acquisition unit 11 determines what the answer was (step S10), and when a positive answer (A or B) is obtained, the label is assumed to be A or B and is selected by the user. An additional information feature yN + 1 is generated in which the feature quantity corresponding to the label is 1 and the rest is 0 (steps S11 and S12).

  On the other hand, if a negative answer (neither) is obtained, the process proceeds to step S7 as in the case of a negative answer when the certainty factor is high, and additional information features are matched to the user's answer. To correct.

Further, when the certainty factor is low, a question as to what the label is is displayed, so that the correct label information (text information) is input from the user (see FIG. 6). In this case, an additional information feature yN + 1 is generated in which the feature amount corresponding to the label information input from the input unit 10 is 1, and the rest is 0 (step S13). Also in this case, when the correct label input by the user is an unknown label for the system, the dimension of the additional information feature y _{N + 1} is increased in order to express the additional information feature corresponding to the label.

  The answer acquisition unit 11 adds the generated additional information feature yN + 1 to the additional information feature set storage unit 13. Further, the feature xN + 1 corresponding to the new content gN + 1 is added to the content feature set storage unit 12.

  Next, based on an instruction from the model update control unit 14, the model update unit 15 stores the content feature set and additional information feature set information updated by the answer acquisition unit 11 in the content recognition model storage unit 4. The set content recognition model is updated (step S14). Since this update process is the same process as the initial model learning unit 3, a detailed description of the treatment operation is omitted. The model update unit 15 updates the content recognition model by repeating the above processing while new content exists.

  Next, experimental results using the content recognition model learning apparatus shown in FIG. 1 will be described. As a content data set for learning, PASCAL Visual Object Challenge (VOC 2008, literature: M. Everingham et al.,: The PASCAL VOC Challenge 2008 Results, http://www.pascal-network.org/challenges/VOC/voc2008 /workshop/index.html) is used. Each image data includes objects included in 20 types of object categories such as people, animals, vehicles, and furniture. Of these, 3596 images are used as the initial content set, the initial model learning unit 3 learns the content recognition model stored in the content recognition model storage unit 4, and the model update unit 15 uses 1000 images as new content. The model was relearned (updated). The remaining 500 sheets were used for evaluation experiments.

  Two methods A and B were used for comparison with the method of the present invention. Method A is a method of asking a question “what is the label” in any case regardless of the certainty factor. Method B is a method in which a question is selected by randomly selecting a question type.

  For each of Method A, Method B, and the method of the present invention, the misidentification rate and user cost (user's answer input) of labels (additional information) estimated by the learned content recognition model for 500 images for evaluation experiments The results of examining (time) are shown in FIGS. Iteration on the horizontal axis represents the number of images (learning samples) input as new content. As the number of learning samples increases, the misidentification rate is improved in any method, but the improvement rate of the misidentification rate of the method of the present invention is particularly high (see FIG. 7). The user cost was evaluated based on the user's answer input time. The text input time was (number of input characters) / (average input time per character), and time for operations other than text input (click, mouse movement, etc.) was ignored. It can be seen that the method of the present invention can complete the input at the lowest cost (see FIG. 8).

  It should be noted that a program for realizing the functions of the processing units shown in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by a computer system and executed to recognize content. Model learning processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  Inferring meaning (information combining information on objects, actions, actions, scenes, etc. contained in images and videos) from media data such as images and videos and text information manually assigned to them Can be applied to indispensable uses.

  DESCRIPTION OF SYMBOLS 1 ... Initial content feature set storage part, 2 ... Initial additional information feature set storage part, 3 ... Initial model learning part, 4 ... Content recognition model storage part, 5 ... New content acquisition part , 6 ... additional information estimation unit, 7 ... neighborhood sample extraction unit, 8 ... question selection unit, 9 ... display unit, 10 ... input unit, 11 ... answer acquisition unit, 12 ... Content feature set storage unit, 13 ... Additional information feature set storage unit, 14 ... Model update control unit, 15 ... Model update unit

Claims (5)

Content recognition model storage means for storing information of a content recognition model for recognizing content data;
Content acquisition means for acquiring content data including an audio signal or a video signal;
Additional information estimation means for estimating additional information indicating the meaning of the audio signal or video signal included in the content data to be added to the content data using the content recognition model stored in the content recognition model storage means; ,
A certainty factor calculating means for obtaining a certainty factor of additional information to be given to the content data estimated by the additional information estimating means;
To determine the additional information to be given to the content data, questions the selected question of changing the question format in response to the confidence determined by the confidence factor computing means, for displaying the selected question Display means;
An answer acquisition means for acquiring an answer corresponding to the question in which the question format is changed ;
A content recognition model learning device comprising: model update means for updating information of the content recognition model stored in the content recognition model storage means based on information of the answer acquired by the answer acquisition means . The question display means includes
If the certainty factor is high, only the question asking whether the additional information to be given to the content data matches the estimated additional information is displayed,
When the certainty factor is medium, only the questions that narrow down the additional information to be added to the content data from the estimated additional information candidates are displayed,
2. The content recognition model learning device according to claim 1, wherein when the certainty factor is low, a question asking what additional information is to be added to the content data is displayed. Content recognition model storage means storing content recognition model information for recognizing content data, content acquisition means, additional information estimation means, confidence factor calculation means, question display means, answer acquisition means, A content recognition model learning method in a content recognition model learning device comprising a model update means,
A content acquisition step in which the content acquisition means acquires content data including an audio signal or a video signal;
The additional information estimation means uses the content recognition model stored in the content recognition model storage means to add additional information indicating the meaning of the audio signal or video signal included in the content data to be added to the content data. An additional information estimation step to estimate;
A certainty factor calculating step in which the certainty factor calculating means obtains a certainty factor of additional information to be added to the content data estimated in the additional information estimating step;
The question display means, in order to determine the additional information to be given to the content data, select a question of changing the question format in response to the confidence determined by the confidence factor computing step, it is selected A question display step for displaying the questions,
An answer obtaining step in which the answer obtaining means obtains an answer corresponding to the question in which the question format is changed ;
The model update unit includes a model update step of updating the content recognition model information stored in the content recognition model storage unit based on the information of the response acquired by the response acquisition step. Content recognition model learning method. The question display step includes
If the certainty factor is high, only the question asking whether the additional information to be given to the content data matches the estimated additional information is displayed,
When the certainty factor is medium, only the questions that narrow down the additional information to be added to the content data from the estimated additional information candidates are displayed,
4. The content recognition model learning method according to claim 3, wherein when the certainty factor is low, a question asking what additional information is to be added to the content data is displayed. A content recognition model learning program for causing a computer to execute the content recognition model learning method according to claim 3 or 4 .

Images