JP5953151B2 - Learning device and program - Google Patents

Description

  The present invention relates to a learning apparatus and a program for learning a discriminator that detects whether a detection target is included in a video.

  As a technique for searching for an image, there is a method of using a color histogram as an index. Specifically, a color histogram is calculated for a given query video, and a video section having a similar color histogram is searched from videos prepared in advance. In this method, a similar video section is specified by searching for a section having similar color histograms while shifting the position of the time window in the search target video. However, in the case of the search method using the color of the video, the content of the video is not considered at all, so even if the video of the same object is semantically, it cannot be detected if the color is different, or a completely different object Even if it is an event, if the colors are similar, they will be regarded as the same.

  Therefore, for advanced search and summarization of video, an index that takes into account the semantic content rather than surface features such as color and texture is necessary. Therefore, it is conceivable to use a discriminator for determining whether an object belongs to a certain classification for the search. In order to construct such a discriminator, learning is performed using learning data of positive examples and negative examples. A positive example represents data in which an object or event to be detected appears, and a negative example represents data in which an object or event to be detected does not appear.

  FIG. 7 is a diagram for explaining the construction of a discriminator using learning data. This figure shows a vector space in which vectors having feature amounts obtained from learning data as elements are arranged, and is actually multidimensional. Each point indicates the position of the feature vector of the learning data, a black point indicates a positive example, and a white point indicates a negative example. The construction of the discriminator corresponds to defining the boundary between the positive example and the negative example in this space, as indicated by a dotted line. Therefore, the more accurate the learning data near the boundary, the more accurate the classifier can be constructed.

  In order to assign a correct or negative label to learning data, the most reliable method is to confirm the contents of all learning data and assign a label. . However, since a large amount of data is required to construct a sufficiently general classifier, it is very difficult to create learning data for various types of objects and events by such a method.

  As a method for solving this problem, a procedure is performed in which a discriminator is learned using a part of learning data that has already been given a positive or negative label, and the label is corrected based on the detection result. There is an approach to construct a discriminator by repeating the above. In this method, it is important how to give the first label. This is because the learning of the discriminator and the correction of the learning data are repeated based on the label, and if the initial learning data is biased, a discriminator specialized for some data is constructed. It is. For example, when constructing a discriminator that detects a clock as a whole, if only initial watches are included in the initial learning data, a discriminator that cannot accurately detect a wall clock, a table clock, or the like is constructed. Alternatively, if the initial learning data includes only data shot at a specific angle, it may be possible to accurately detect only at that angle.

  As an initial learning data generation method, there is a method of learning a discriminator using results detected by a plurality of separately developed identification methods and generating initial learning data (for example, non-patent literature). 1).

Stephane Ayache and Georges Quenot, "Evaluation of Active Learning Strategies for Video Indexing", Signal Processing: Image Communication, Vol 22 / 7-8, pp 692-704, 2007.

  In the initial learning data creation method as in Non-Patent Document 1, there is a problem whether sufficient accuracy and diversity are ensured in the identification method used for generating the first learning data. There is also the problem of how to learn the classifier itself.

  The present invention has been made in view of such circumstances, and constructs a discriminator that detects a detection target such as a specific object or event from a video with high accuracy by learning using diverse learning data. A learning device and a program are provided.

[1] One aspect of the present invention is a video data storage unit that stores video data, a feature amount of the video data, and a positive example in which a detection target appears in the video data, or a negative example in which the detection target does not appear. A learning data storage unit for storing learning data including a label indicating whether or not, when initial learning data is registered in the learning data storage unit, and when learning data is added to the learning data storage unit A classifier constructing unit that constructs a discriminator using the learning data stored in the learning data storage unit, and a discriminator constructed by the classifier constructing unit and stored in the learning data storage unit. For the learning data, among the video sections of the input video data, the video section that looks similar to the current positive learning data, the video data of the video section that has similar audio features, Knowledge A discriminator detector that performs detection processing using a separate device, a determination unit that determines the accuracy of the discriminator based on a detection result by the discriminator detector, and the accuracy of the discriminator is predetermined in the determination unit When it is determined that the accuracy of the video data has not been reached, a part of the video data stored in the video data storage unit is selected, and a positive example label is attached to the feature amount of the selected video data. A learning data adding unit that adds the generated learning data to the learning data storage unit, and the learning data adding unit randomly selected from the video data stored in the video data storage unit Video data, video data similar to the video data obtained from the learning data of the positive example stored in the learning data storage unit, or a detection target similar to the classifier to be constructed Selecting a portion from the image data detected by the other discriminator, it is learning apparatus according to claim.
According to this aspect, the learning device constructs a discriminator that detects whether the video is related to the detection target from the initial learning data of the positive example and the negative example, and detects the learning data by the built discriminator. Process to check accuracy. When the accuracy is low, the learning device generates learning data from a part of the video data stored in the video data storage unit, and adds it as a positive example to the current learning data. The learning device repeats the construction of the discriminator using the learning data and the addition of the learning data until the accuracy becomes high.
Thereby, since the learning apparatus can generate learning data without bias, it is possible to construct a discriminator that detects a detection target such as a specific object or event from a video with high accuracy.

Further , according to this aspect, the learning device can select video data that is randomly selected from the registered video data, video data that is visually or auditorily similar to the positive video data, or Select a portion of the video data detected using a learned classifier that detects a detection target that is semantically similar to the object or event that is being detected by the target classifier. The learning data is generated from and added to the current learning data.
As a result, the learning device can add learning data so as to improve diversity, so that a more accurate classifier can be constructed.

[ 2 ] One aspect of the present invention is the learning device described above, wherein the discriminator detection unit is stored in the learning data storage unit when learning data is added to the learning data storage unit. The learning data is detected by the discriminator, a positive example or a negative example is set as a label of the learning data based on a detection result, and the discriminator construction unit stores the learning data in the learning data storage unit. When added, after the label is set by the discriminator detector, a discriminator is constructed using the learning data stored in the learning data storage unit.
According to this aspect, the learning device detects all the learning data including the learning data after the addition by the current classifier before constructing the classifier in the iterative process, and learns data based on the detection result. Rewrite the label.
Thereby, since the error of the label of learning data is corrected, the performance of the constructed discriminator can be improved.

[ 3 ] One aspect of the present invention is the learning device described above, wherein the initial learning data registered in the learning data storage unit or the learning data in which a label is set by the classifier detection unit is used. On the other hand, a learning data correction unit that corrects a label of the learning data based on a user input or a detection result of the learning data by another discriminator is further provided, and the discriminator construction unit initially stores the learning data storage unit. When the learning data is registered, and when the learning data is added to the learning data storage unit, the learning data stored in the learning data storage unit is corrected after the label correction by the learning data correction unit. The classifier is used to construct a classifier.
According to this aspect, the learning device corrects the positive example and negative example labels of the learning data based on the input from the user or the detection result by another classifier before the classifier is constructed.
Thereby, since the error of the label of learning data is corrected accurately, the performance of the constructed discriminator can be improved.

[ 4 ] One aspect of the present invention is the above-described learning device, in which text data representing audio of the video data includes a keyword representing a detection target by the classifier to be constructed and other keywords related to the keyword. An initial learning data generation unit that detects whether it is included, generates initial learning data by adding a positive example label to the feature amount of the video data corresponding to the detected text data, and registers it in the learning data storage unit Is further provided.
According to this aspect, it is possible to generate the initial learning data based on the content of the video data rather than the surface features such as the color and texture of the video.
Thereby, the learning device can generate a variety of initial learning data based on the content of the video data, and can construct a discriminator that can accurately search the video based on the semantic content.

[ 5 ] According to one aspect of the present invention, learning data including a feature amount of video data and a label indicating whether a detection target appears in the video data is a positive example or a negative example that does not appear. A learning data storage unit for storing, a learning data storage unit for storing learning data including a feature amount of a video and a label indicating whether the detection target is a positive example or a negative example, and an initial in the learning data storage unit When the learning data is registered, and when learning data is added to the learning data storage unit, a classifier construction unit that constructs a classifier using the learning data stored in the learning data storage unit The learning data stored in the learning data storage unit by the classifier constructed by the classifier construction unit is viewed as current learning data from the video section of the input video data. Is similar The classifier based and has video section, the image data of the image segment in which speech features are similar, the identifier detection unit that performs detection processing for detecting the use of the identification device, the detection result of the classifier detector A determination unit for determining the accuracy of the video data, and when the determination unit determines that the accuracy of the discriminator has not reached a predetermined accuracy, a part of the video data stored in the video data storage unit is selected. A learning data adding unit that adds learning data generated by assigning a positive example label to the feature amount of the selected video data to the learning data storage unit, and the learning data adding unit includes the video data Video data from which the video data selected at random among the video data stored in the storage unit and the learning data of the positive example stored in the learning data storage unit are obtained Video data similar, or a program to function to select a part among the image data detected by other identifier corresponding to the similar detection target and the identifier of the building object.

  According to the present invention, it is possible to generate a variety of learning data, and to construct a discriminator that detects a detection object such as a specific object or event from a video with high accuracy by learning using the generated learning data. it can.

It is a block diagram which shows the structure of the learning apparatus by one Embodiment of this invention. It is a figure which shows the example of the audio | voice text data by the embodiment. It is a figure which shows the example of the learning data by the embodiment. It is a figure which shows the processing flow of the learning apparatus by the embodiment. It is a figure which shows the experimental result using the learning apparatus by the embodiment. It is a figure which shows the experimental result using the learning apparatus by the embodiment. It is a figure for demonstrating construction | assembly of the discriminator using learning data.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a learning device 1 according to an embodiment of the present invention, and shows only functional blocks related to the present embodiment. The learning device 1 can be realized by a computer device such as one or a plurality of server computers, for example.

  The learning device 1 performs learning using positive and negative learning data generated from input video data (hereinafter referred to as “input video data”), and constructs a discriminator. Here, the positive example indicates that an object (object) or event (event) to be detected appears in the video, and the negative example indicates that it does not appear. The discriminator is an algorithm that receives video features and detects whether the video is related to an object or event to be detected.

  Therefore, first, the learning device 1 uses a text representing the sound of the input video data, and has a keyword representing a detection target of the classifier to be constructed, a synonym or a synonym of the keyword, or a semantic inclusion relation with the keyword. Video segments corresponding to other keywords are extracted. The learning device 1 constructs a discriminator using the extracted video section as initial learning data as a positive example, and if the constructed discriminator is not sufficiently accurate, the learning device 1 generates learning data as diverse as possible. Are added, corrected, and the process of constructing the classifier is repeated.

  When the learning device 1 adds learning data, the learning device 1 randomly selects a video segment from among video segments obtained by dividing the input video data by a certain unit, a video segment that is visually similar to the positive example, and is semantic. The video data of the video section detected using the learned discriminator corresponding to the keyword similar to is mixed with the learning data already generated as a positive example by a certain ratio. As a result, the learning data is not partially biased, and a highly accurate classifier can be constructed.

  As shown in the figure, the learning device 1 includes a storage unit 10, an input unit 11, a video segment dividing unit 12, an initial learning data generating unit 13, a learning data correcting unit 14, a discriminator constructing unit 15, and a discriminator detecting unit 16. , A discriminator determining unit 17 and a learning data adding unit 18.

The storage unit 10 is realized by a hard disk device, a semiconductor memory, or the like, and includes a video data storage unit 101, a learning data storage unit 102, and a discriminator storage unit 103.
The video data storage unit 101 stores input video data and audio text data. The input video data is moving image content data, and in this embodiment, a case where a broadcast program is used as a moving image will be described.
The audio text data includes text data indicating the audio of the input video data and synchronization data for specifying a video portion of the input video data corresponding to the text data. In the present embodiment, closed caption data indicating transcription of program audio or voice recognition data indicating a result of voice recognition of voice included in input video data is used as voice text data.

  The learning data storage unit 102 stores learning data for constructing a discriminator. The learning data indicates a correspondence between a video section in the input video data, feature data, and a label indicating whether the example is a positive example or a negative example. The feature data indicates the image feature amount in the video section.

  The classifier storage unit 103 stores an existing classifier and a new classifier constructed from learning data. The discriminator is an algorithm that detects whether or not the video data is related to the detection target by receiving the feature data that represents the image feature of the video data and is associated with the keyword that represents the detection target. The classifier uses, for example, a classification algorithm such as a support vector machine or a decision tree, and calculates a value that quantitatively represents the degree to which the video is related to the detection target from the input feature data.

The input unit 11 receives input of various data such as input video data, a keyword representing an object or event to be detected by a newly constructed discriminator, and information for specifying learning data to be rewritten as a label selected by the user.
The video section dividing unit 12 divides each input video data for each video section. In the present embodiment, the video segment dividing unit 12 divides the input video data for each shot. One shot is a section continuously shot by one camera, and represents a section sandwiched between camera switching points.

The initial learning data generation unit 13 generates initial learning data from input video data. The initial learning data generation unit 13 includes a closed caption extraction unit 131, a program audio recognition unit 132, a keyword expansion unit 133, a video section extraction unit 134, and a feature data extraction unit 135.
The closed caption extraction unit 131 extracts closed caption data from the input video data and sets it as audio text data. The program voice recognition unit 132 performs voice recognition processing on the program voice of the input video data to generate voice text data. The keyword expansion unit 133 uses a thesaurus or dictionary stored in the thesaurus storage device 5 connected to the learning device 1, a keyword similar to the input keyword, a synonymous keyword, or a keyword that is semantically contained. And so on. The video segment extraction unit 134 extracts the video segment corresponding to the input keyword or the keyword extracted by the keyword expansion unit 133 from the input video data using the audio text data. The feature data extraction unit 135 acquires feature data from the video data of the video section extracted by the video section extraction unit 134, and generates first learning data.

  The learning data correction unit 14 is based on the information input from the input unit 11 or based on the detection result of the discriminator constructed in the course of the iterative process when learning is performed. Modify the labels that represent examples or negative examples. The initial learning data generation unit 13 generates the first learning data using the video section corresponding to the keyword as a positive example. However, even if keywords are included in the program audio or closed caption, the target object or event does not always appear in the video, so the label of whether the first learning data is positive or negative is corrected. is necessary. Similarly, there is a possibility that errors and omissions are included in the learning data to which the labels are given based on the detection results of the discriminator constructed in the process of iterative processing when learning is performed. Therefore, it is necessary to correct the label of the learning data even in the process of iterative processing.

The classifier construction unit 15 constructs a classifier from the learning data.
The discriminator detector 16 applies the discriminator constructed by the discriminator constructing unit 15 to the learning data, and obtains a detection result. The discriminator detection unit 16 sets the learning data determined to be related to the detection target based on the detection result as the learning data of the next positive example.
The classifier determination unit 17 determines whether or not the classifier constructed by the classifier construction unit 15 has sufficient accuracy.

  The learning data adding unit 18 adds learning data when the classifier determination unit 17 determines that the accuracy of the classifier is not sufficient. Simply constructing an identifier again using the learning data determined to be related to the detection target based on the detection result of the discriminator as a positive example, only those similar to the learning data when learning this discriminator can be detected accurately. End up. Further, if there is a problem such as insufficient diversity in the initial data, there is a possibility that a discriminator that can detect only specific data with high accuracy is learned. Therefore, the learning data adding unit 18 adds, as positive learning data, video data selected from the video section of the input video data by a discriminator that has no dependency relationship with the discriminator currently being learned. This avoids learning a discriminator biased to specific data.

The learning data adding unit 18 includes a random data selecting unit 181, a similar video selecting unit 182, a similar classifier detecting unit 183, and a data mixing unit 184.
The random data selection unit 181 sets the video data of the video section randomly extracted from the video sections of the input video data as addition candidates to the learning data.
The similar video selection unit 182 selects, from the video sections of the input video data, video data of a video section that looks similar to the current learning data of the positive example and a video section that has similar audio characteristics, Add candidates to learning data.
The similar classifier detection unit 183 includes a keyword that is semantically similar to the input keyword from among the already learned classifiers stored in the classifier storage unit 103, related keywords, and semantics. A learned classifier corresponding to a keyword having a content relation is selected. The similar classifier detection unit 183 performs detection processing on the video section of the input video data using the selected learned classifier, and sets the video data of the detected video section as an addition candidate to the learning data.
The data mixing unit 184 adds video data determined as additional candidates in the random data selection unit 181, the similar video selection unit 182, and the similar classifier detection unit 183 as a positive example to the learning data by a certain ratio. When the addition is performed a plurality of times, the data mixing unit 184 does not add the data determined to be a negative example even once by the learning data correction unit 14 to the learning data as a positive example.

  FIG. 2 is a diagram showing an example of voice text data. The audio text data shown in the figure is closed caption data, and includes text data indicating program audio and synchronization data indicated by time code information corresponding to the text data.

  FIG. 3 is a diagram illustrating an example of learning data. As shown in the figure, the learning data associates video segment specifying data for specifying a video segment, feature data indicating an image feature amount in the video segment, and a label indicating whether it is a positive example or a negative example. Data. The video section specifying data is indicated by identification information of the input video data and the start position and end position of the video section in the input video data. The start position and end position are, for example, from the beginning of the input video data such as time code information. Indicated by the playback time.

FIG. 4 shows a flowchart of the processing procedure of the learning apparatus.
First, the input unit 11 of the learning device 1 receives input of input video data and a keyword representing an object or event to be detected by a newly constructed classifier. The input unit 11 writes the input video data to the video data storage unit 101 and outputs the keyword to the initial learning data generation unit 13.

  The video section dividing unit 12 reads the input video data stored in the video data storage unit 101 and divides each input video data in units of one shot. For example, the video section dividing unit 12 calculates a video difference between adjacent frames indicated by the input video data, detects a cut point using the calculated difference as an index, and converts the input video data at the detected cut point for each video section. Separated into The video segment dividing unit 12 writes the divided data indicating the start position and the end position of each video segment in the video data storage unit 101 in association with the input video data (step S105). Thereafter, the learning device 1 specifies a video section in the video data based on the divided data.

Subsequently, the initial learning data generation unit 13 generates initial learning data from the input video data (step S110).
First, when the closed caption is superimposed on the input video data, the closed caption extraction unit 131 extracts the closed caption from the input video data and writes it as audio text data in the video data storage unit 101.

  Subsequently, the program audio recognition unit 132 acquires audio data from the input video data on which the closed caption is not superimposed, and performs audio recognition on the program audio indicated by the acquired audio data. The program voice recognition unit 132 generates voice recognition data in which text data indicating the result of voice recognition of the program voice and the synchronization data representing the video portion of the input video data from which the voice recognized is obtained. Then, it is written as audio text data in the video data storage unit 101.

  The keyword expansion unit 133 searches a thesaurus or dictionary stored in the thesaurus storage device 5 provided outside or inside the learning device 1, and searches for keywords similar to the input keyword, synonymous keywords, semantically Read keywords that are contained. For example, when the input keyword is “car”, the keyword expansion unit 133 acquires “automobile”, “car”, “taxi”, “passenger car”, etc. as similar keywords or synonymous keywords, and semantically. “Land traffic”, “One box car”, “Light car”, etc. are acquired as keywords related to inclusion. Hereinafter, similar keywords, synonymous keywords, and keywords that are semantically contained based on the input keywords will be referred to as “related keywords”.

  The video segment extraction unit 134 searches the audio text data stored in the video data storage unit 101 to detect input keywords and related keywords, and acquires synchronization data corresponding to the detected input keywords and related keywords. This synchronization data represents the time in the program in which the keyword appears. If the time is t, t is given based on the time code information described in the closed caption or the time when the voice is recognized. It is what was done. The video segment extraction unit 134 extracts a video segment corresponding to the specified synchronization data from the input video data.

For example, the video segment extraction unit 134 selects a video segment from time t-δ to time t + δ for the keyword appearance time t. Note that δ is a predetermined time. The video section extraction unit 134 sets time t−δ as a start position and time t + δ as an end position.
Alternatively, the video segment extraction unit 134 selects a shot at time t. In this case, the video segment extraction unit 134 selects a video segment including the time t indicated by the identified synchronization data from the video segments indicated by the divided data added to the input video data.
The feature data extraction unit 135 generates feature data representing the feature amount of the video from the video data of the video segment selected by the video segment extraction unit 134.

  Since the feature quantity used as the feature data needs to correspond to various objects and events, a more general-purpose feature quantity is used instead of a feature quantity specialized for a specific object or event. Specifically, feature data is generated by combining color moments in the grid area, edge direction histograms, Gabor wavelets, Haar wavelets, local binary patterns, and the like. For example, “T. Ojala, M. Pietikaninen and T. Maenpaa,“ Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, ”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 7 , pp. 971-987, 2002. "(Reference 1).

Alternatively, it is conceivable to use a feature amount that is a combination of methods using an approach in general object recognition, such as creating a frequency histogram based on a luminance gradient histogram in a local region near a feature point such as a corner. For example, “G. Csurka, C. Bray, C. Dance and L. Fan,“ Visual categorization with bags of keypoints, ”in Proc. ECCV Workshop on Statistical Learning in Computer Vision, pp. 59-74, 2004. (Reference 2).
In addition, as a feature quantity considering the time direction, it is conceivable to consider a motion vector sequence between frames, a correlation between feature quantities between frames, or use a feature of speech.

  The feature data extraction unit 135 generates learning data in which the video segment specifying data indicating the extracted video segment, the feature data of the video segment, and the label indicating the positive example are set and written in the learning data storage unit 102, Register as the first learning data.

  Subsequently, the learning data correction unit 14 corrects the label of the first learning data for which all positive examples are currently set (step S115). The correction is most accurately performed manually. Therefore, the correction to the first learning data is about the learning data in which a person (user) determines whether the learning data is given a positive example label and whether it is a negative example based on the determination result. , Correct the label to a negative example.

  Specifically, the input unit 11 receives input of information for specifying learning data as a negative example among positive learning data currently stored in the learning data storage unit 102. The learning data correction unit 14 rewrites the label of the learning data specified by the information input by the input unit 11 from the positive example to the negative example.

  The discriminator construction unit 15 constructs a discriminator using the learning data currently stored in the learning data storage unit 102, and writes it into the discriminator storage unit 103 (step S120). Note that the classifier construction unit 15 uses machine learning such as a support vector machine or a random forest for construction of the classifier.

  The discriminator detection unit 16 applies the discriminator constructed in step S120 to the learning data currently stored in the learning data storage unit 102, and obtains the detection result using the feature data set in each learning data as an input. . This detection result indicates a value that quantitatively represents the degree to which each learning data is related to the detection target, and the rank of the degree of relation obtained based on the value. The discriminator detection unit 16 rewrites the negative example in the label of the learning data determined to be related to the detection target based on the detection result, and rewrites the positive example in the label of the learning data determined not to be related. If is set, it is rewritten as a negative example. The discriminator determination unit 17 determines whether or not the accuracy of the constructed discriminator is greater than or equal to a threshold value from the detection result by the discriminator constructed in step S125 (step S125).

  As an index for evaluating the accuracy of the discriminator, an average relevance ratio that is an evaluation index for the ranked search result can be used. The following formula (1) shows a formula for calculating the average precision for the top N detection results.

Here, r _k is 1 if rank k-th detection result correct, represents 0 if incorrect. Information about whether the answer is correct or incorrect is input to the input unit 11 by a person.
Further, p (k) in equation (1) represents the relevance ratio of each of the top N cases, and is calculated by the following equation (2).

  When the discriminator determining unit 17 determines that the accuracy (average relevance ratio) calculated by the equation (1) is less than the threshold (step S125: NO), the learning data adding unit 18 stores the learning data in the learning data storage unit 102. The additional learning data is mixed with the learning data that has been set (step S130).

  First, the random data selection unit 181 randomly extracts from the video sections of the input video data stored in the video data storage unit 101, and sets each extracted video section as a learning data generation candidate.

  Further, the similar video selection unit 182 identifies learning data in which a positive example is set in the label from the learning data storage unit 102, and the video segment of the input video data indicated by the video segment identification data included in the identified learning data To obtain similarity detection feature data. Further, the similar video selection unit 182 acquires the feature data for similarity detection for each video section of each input video data stored in the video data storage unit 101.

As the feature data for similarity detection, video features and audio features can be used. For example, video color histograms, textures, and the like can be used as video features, and audio frequency distributions, audio power distributions, and the like can be used as audio features.
In addition, as the image feature amount of the video section indicated by the similarity detection feature data, color, texture, or the like can be used. In addition, as described in Reference Document 2 above, as an image feature amount, it is also possible to use an approach of creating frequency histograms based on luminance gradient histograms in local regions near feature points such as corners. It is done.

  The similar video selection unit 182 quantitatively determines how similar the similarity detection feature data for each video segment of each input video data is with the similarity detection feature data for the video segment corresponding to the learning data of the positive example. Calculate the value to represent. Based on this calculated value, the similar video selection unit 182 identifies a video segment that looks similar to the current learning data or a video segment that has similar audio characteristics.

  For example, the similar video selection unit 182 adds the similarities calculated for the feature data obtained from the video section of the input video data and the feature data obtained from the video section corresponding to the learning data of the positive example. The similar video selection unit 182 learns a predetermined number of video section data from the total similarity calculated for each video section of each input video data equal to or higher than a predetermined threshold or from the highest ranking based on the total similarity. Identified as a data generation candidate.

  In addition, the similar classifier detection unit 183 searches the thesaurus or dictionary stored in the thesaurus storage device 5 and reads the related keywords for the input keyword. The similar discriminator detection unit 183 selects a learned discriminator whose detection target is a related keyword from the already learned discriminators stored in the discriminator storage unit 103. The similar classifier detection unit 183 acquires feature data for each video section of each input video data stored in the video data storage unit 101, and detects the selected feature classifier using the acquired feature data as an input. Execute the process. The similar classifier detection unit 183 identifies the video section detected as related by the learned classifier as a learning data generation candidate.

  The data mixing unit 184 selects a certain ratio from the video data of learning data generation candidates specified by the random data selection unit 181, the similar video selection unit 182, and the similar classifier detection unit 183. Note that the mixing ratio of the video sections of the learning data generation candidates specified by the random data selection unit 181, the similar video selection unit 182, and the similar classifier detection unit 183 can be made variable depending on the detection target.

  The data mixing unit 184 generates feature data from the video data of the selected video segment for which the feature data has not yet been generated. The data mixing unit 184 generates learning data by associating the video segment specifying data and the feature data of the selected video segment with the label set with the positive example, and additionally writes it to the learning data storage unit 102.

Note that even when learning data is added a plurality of times by iterative processing, the similar video selection unit 182 and the similar classifier detection unit 183 may first specify a video section of a learning data generation candidate once. In the second and subsequent learning data addition processing, the data mixing unit 184 selects a learning data generation target from the video sections of the specified learning data generation candidates.
In addition, when mixing is performed a plurality of times, the data mixing unit 184 does not add data that has been determined as a negative example even once by the learning data correction unit 14 to the learning data as a positive example.

  The discriminator detector 16 uses the currently constructed discriminator, reads all the learning data from the learning data storage unit 102, and executes a detection process on the read learning data. The discriminator detection unit 16 rewrites the negative example in the label of the learning data determined to be related based on the detection result, and sets the positive example to the label of the learning data determined not to be related. If so, rewrite it as a negative example. Then, the person determines whether or not the top N pieces of learning data obtained as a result of the detection are correct and inputs information for specifying the learning data that is the object of wrong determination. The input unit 11 receives input of information for specifying learning data to be corrected, and the learning data correction unit 14 becomes a negative example if a positive example is set for the learning data specified by the input information. If rewriting or a negative example is set, it is rewritten to a positive example (step S135).

  Although the accuracy increases as N increases, it is usually determined as a percentage of the total number of learning data, or is determined according to the time and number of people required for the correction work. When it is necessary to completely automate, there is a method of preparing a plurality of discriminators of algorithms with completely different approaches and determining the correctness by majority decision of those discriminators.

The classifier construction unit 15 reads all the learning data from the learning data storage unit 102, and constructs a classifier using the read learning data (step S140).
The discriminator detector 16 applies the discriminator constructed in step S140 to the learning data stored in the learning data storage unit 102, and executes detection processing. The discriminator detection unit 16 rewrites the negative example in the label of the learning data determined to be related to the detection target based on the detection result, and rewrites the positive example in the label of the learning data determined not to be related. Is set to a negative example (step S145).

  After the process of step S145, the learning device 1 repeats the process from step S125 for determining whether or not the accuracy of the constructed discriminator is equal to or higher than the threshold value from the detection result by the discriminator constructed in step S140. In step S125, when the discriminator determination unit 17 determines that the accuracy is equal to or higher than the threshold (step S125: YES), the learning device 1 ends the process.

5 and 6 are diagrams showing experimental results using the learning apparatus 1 according to the present embodiment.
FIG. 5 shows the relationship between the number of repetitions of classifier construction and the change in the average precision value for each keyword (object name) and the number of positive learning data (# of Pos). The average precision is calculated for the top 100 cases. As shown in the figure, for each keyword, the value of the average relevance ratio improves as the number of repetitions increases, and the accuracy is equal to or greater than the threshold after 3 to 6 repetitions.

FIG. 6 is a diagram showing the relationship between the number of repetitions of the classifier construction shown in FIG. 5 and the average change in the average precision for each keyword. As shown in the figure, the increase in the average precision starts to saturate when the number of repetitions is about 3, and exceeds 0.95 for the fifth time.
Thus, the learning device 1 can improve the detection accuracy of the classifier by learning the classifier while adding learning data.

According to the embodiment described above, the learning device 1 generates learning data to which positive and negative labels are attached from video data such as a television program, and detects a specific object or event. A discriminator is constructed based on the generated learning data.
The learning device 1 searches a keyword representing the detection target of a newly constructed classifier and an additional keyword selected using a dictionary such as a thesaurus from the recognition result or closed caption of the program audio, and finds a corresponding video section. Extracted as positive example video data. Thereby, it is possible to efficiently generate learning data for constructing a discriminator that enables a search based on the content of the video, not the surface features such as color and texture. In addition, it is possible to construct various classifiers for various detection targets from various programs regardless of the specific object or event, regardless of the program genre or broadcasting station.

  The learning device 1 corrects the label indicating whether the learning data is a positive example or a negative example, and then constructs a discriminator based on the data, and the learning data detected by the constructed discriminator is used as the next learning data. The process of making a positive example in is repeated. Thereby, the accuracy of the discriminator can be improved.

  In addition, the learning device 1 detects a video section randomly selected from the input video data, or a video section that is visually or auditorily similar to the video data of the positive example, and a classifier to be constructed Video data of a video section detected using a learned discriminator corresponding to a detection target that is semantically similar to the target object or event is mixed at a fixed rate as a positive example in the learning data during the iterative process. To do. As a result, it is possible to add learning data so as to improve diversity, and avoid building a discriminator biased to specific data by iterative processing.

  Note that, by using still image data with attribute data instead of video data, a discriminator for detecting whether still image data is related to a detection target can be constructed. In this case, the learning apparatus 1 uses the text information about the still image described in the attribute data instead of the voice text data, and performs the same processing assuming that one still image data corresponds to one video section. . In this case, the feature data represents the feature amount of the still image.

  The learning device 1 described above has a computer system inside. Then, the input unit 11, the video section dividing unit 12, the initial learning data generation unit 13, the learning data correction unit 14, the classifier construction unit 15, the classifier detection unit 16, the classifier determination unit 17, and the learning data of the learning device 1 The process of the operation of the adding unit 18 is stored in a computer-readable recording medium in the form of a program, and the above-described processing is performed by the computer system reading and executing this program. The computer system here includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage unit such as a hard disk built in the computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 Learning apparatus 5 Thesaurus memory | storage device 10 Memory | storage part 11 Input part 12 Image | video division | segmentation part 13 Initial learning data generation part 14 Learning data correction part 15 Classifier construction part 16 Classifier detection part 17 Classifier determination part 18 Learning data addition part 101 Video data storage unit 102 Learning data storage unit 103 Discriminator storage unit 131 Closed caption extraction unit 132 Program audio recognition unit 133 Keyword expansion unit 134 Video section extraction unit 135 Feature data extraction unit 181 Random data selection unit 182 Similar video selection unit 183 Similar Discriminator detector 184 Data mixer

Claims (5)

A video data storage unit for storing video data;
A learning data storage unit that stores learning data including a feature amount of the video data and a label indicating whether the detection target appears in the video data as a positive example or a negative example that does not appear;
A discriminator using the learning data stored in the learning data storage unit when initial learning data is registered in the learning data storage unit and when learning data is added to the learning data storage unit A classifier construction unit for constructing
With the classifier constructed by the classifier construction unit, the learning data stored in the learning data storage unit is compared with the current positive learning data from the video section of the input video data. A discriminator detector for performing a detection process for detecting video data of a video segment that is similar to the video segment that has similar audio characteristics using the discriminator;
A determination unit that determines the accuracy of the classifier based on a detection result by the classifier detection unit;
When the determination unit determines that the accuracy of the discriminator does not reach a predetermined accuracy, a part of the video data stored in the video data storage unit is selected, and the characteristics of the selected video data A learning data adding unit for adding learning data generated by giving a positive example label to the quantity to the learning data storage unit;
Equipped with a,
The learning data adding unit obtains the video data selected at random from the video data stored in the video data storage unit, and the learning data of the positive example stored in the learning data storage unit. A part of the video data detected by another classifier corresponding to the classifier that is similar to the classifier to be constructed or the classifier to be constructed,
A learning apparatus characterized by that. When the learning data is added to the learning data storage unit, the classifier detection unit detects the learning data stored in the learning data storage unit by the classifier,
Based on the detection result, set a positive example or a negative example in the label of the learning data,
When the learning data is added to the learning data storage unit, the classifier construction unit uses the learning data stored in the learning data storage unit after the label is set by the classifier detection unit. Build,
The learning apparatus according to claim 1 . For the initial learning data registered in the learning data storage unit or the learning data for which the label is set by the discriminator detection unit, user learning or the detection result of the learning data by another discriminator A learning data correction unit for correcting the label of the learning data based on the learning data;
When the initial learning data is registered in the learning data storage unit and when learning data is added to the learning data storage unit, after the correction of the label by the learning data correction unit, the classifier construction unit, Building a discriminator using the learning data stored in the learning data storage unit;
The learning apparatus according to claim 2 , wherein: Detect whether the text data representing the audio of the video data includes a keyword representing a detection target by the classifier to be constructed and other keywords related to the keyword, and the video data corresponding to the detected text data An initial learning data generation unit that adds a positive example label to the feature quantity to generate initial learning data and registers the learning data in the learning data storage unit is further provided.
The learning device according to any one of claims 1 to 3 , wherein The computer used for the learning device
A video data storage unit for storing video data;
A learning data storage unit that stores learning data including a feature amount of video data and a label indicating whether the detection target is a positive example or a negative example that does not appear in the video data;
A discriminator using the learning data stored in the learning data storage unit when initial learning data is registered in the learning data storage unit and when learning data is added to the learning data storage unit The classifier construction part,
With the classifier constructed by the classifier construction unit, the learning data stored in the learning data storage unit is compared with the current positive learning data from the video section of the input video data. A discriminator detector that performs a detection process for detecting video data of a video segment that is similar and a video segment that has similar audio characteristics using the discriminator,
A determination unit that determines the accuracy of the classifier based on a detection result by the classifier detection unit,
When the determination unit determines that the accuracy of the discriminator does not reach a predetermined accuracy, a part of the video data stored in the video data storage unit is selected, and the characteristics of the selected video data A learning data adding unit for adding learning data generated by giving a positive example label to the quantity to the learning data storage unit,
To function as,
The learning data adding unit obtains the video data selected at random from the video data stored in the video data storage unit, and the learning data of the positive example stored in the learning data storage unit. A program that functions to select a part of the video data that is similar to the video data that has been detected or the video data that has been detected by another classifier corresponding to the classifier that is similar to the classifier to be constructed .

Images