JP2013009248A - Video detection method, video detection device, and video detection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately and efficiently detect advertisement video.SOLUTION: A video detection device 1 comprises: a structure analysis part 101 for analyzing a section structure of an input video and dividing it into partial sections; a candidate section detection part 102 for detecting one or more candidate sections including an advertisement video on the basis of characteristics of the partial sections, and finding its start time and its completion time; a shot characteristic extraction part 103 for analyzing at least one of an image and a sound in the partial section existing inside the start time and the completion time of the candidate section, and extracting a characteristic amount; a coding part 104 for expressing the partial sections by codes on the basis of the characteristic amount; a recursive code sequence detection part 105 for detecting a group of the partial sections to be code sequences recursively appearing from the code sequences constituted by one or more partial sections expressed by the codes; and a filter part 106 for outputting only the partial sections satisfying a predetermined condition as an advertisement video section among the group of the detected partial sections.

Description

  The present invention relates to a video detection method, a video detection apparatus, and a video detection program for analyzing an input video to be processed and detecting an advertisement video.

  As represented by VOD (Video On Demand), IPTV (Internet Protocol Television), digital terrestrial broadcast retransmission, etc., the use of video distribution services using communications has become active, and a huge amount of video content has been created over multiple channels. It came to be delivered.

  For producers, in addition to the existing broadcasts and screen screenings in movie theaters, multi-channel sales using the video distribution service are now possible. In addition, this not only increases the number of simple sales channels, but also brings new business opportunities. The representative is video advertisement. Video advertising is an important promotional material that attracts interest in the object of promotion and motivates consumers to purchase. Therefore, video advertisements are required to be timely in line with market trends and user needs.

  However, the new video distribution service as described above does not always work. In other words, in broadcasts and movie theaters that have been mainstream until now, it was common for all users to view the same content at the same time (or at the same time). For this reason, since it was possible to grasp when and where, and in some cases, what kind of user group would view the content, it was possible to hit an appropriate advertisement video. On the other hand, in a new video distribution service, it is normal for each user to enjoy viewing at a free place at any time. For this reason, depending on the timing and time when the user views, there arises a problem that the inserted advertisement video becomes completely uninteresting or meaningless at all. In order to cope with such a problem, for example, an adaptive advertisement video distribution system is desired, such as switching to an appropriate advertisement video depending on the time when the user views.

  There are a variety of technologies required to realize such a system, but in order to replace the advertisement video from the old one to the new one, at least first, where the advertisement video is inserted in the original video. I need to know. If this information is obtained in advance, there will be no trouble, but it is not easy to know and manage the position of the advertisement video for all the video data in recent years when a large amount of video data is distributed. Therefore, there is a demand for a technology that automatically detects only advertising videos from a large amount of video data.

  As for advertisement video detection technology, many inventions have been made so far. Older digital video recorders, etc. used the fact that the number of audio channels differs between the advertisement video section and other video sections, but recently there is no difference in the number of audio channels, so this method detects it. Became difficult. For this reason, inventions relating to techniques for directly analyzing video and detecting advertisement video have been made. The existing prior art is roughly classified into the following two.

(1) When there is reference data (2) When there is no reference data The former assumes that the advertisement video to be detected has been determined and is obtained (this data is referred to as reference data). Yes, it is a technique for finding a video section that matches the reference data from the video data. On the other hand, the latter is a technique that assumes that no such reference data is obtained.

  As a technique belonging to the former, Patent Document 1 discloses a technique for holding a still image as reference data and detecting a video frame similar to the still image.

  As a technology belonging to the latter, Non-Patent Document 1 models unique features that appear in advertising videos using statistical models such as Support Vector Machine (SVM) and Hidden Markov Model (HMM). A technique for performing detection based on the “advertisement video model” has been disclosed. The video features include image frame similarity between adjacent shots included in the video (a series of video segments between cut points), average and variance of edge change rate, Type (speech, music, silence, etc.) and their time difference are used. These video features are modeled using SVM, and a section that seems to be an advertising video section is detected.

  Similarly, as a technique belonging to the latter, Non-Patent Document 2 discloses a technique for performing detection based only on shot similarity without using a model. In this technique, first, all videos are divided into shots. Next, the average and variance of the color of the image frame included in each shot, the average and variance of the directional luminance, the edge, and the position and size of the face included in the image are extracted as shot feature quantities, and between all shots The similarity is calculated based on the feature amount. Finally, only shots having a special tendency in the similarity distribution with other shots are detected as advertisement video sections.

  Further, as a technique belonging to the latter, Patent Document 2 discloses a section in which one or more shots are detected and the start time and end time are silent, and the interval is either 15 seconds or 30 seconds. Is disclosed as an advertisement video section.

  Similar to Patent Document 2, Patent Document 3 discloses a technique using the time length of an advertisement video. In this technique, a time when the average volume value has dropped below 80 continuously for 100 milliseconds or more is detected as a blank, and when the interval of this blank matches the time length (for example, 15 seconds, 30 seconds) of the advertisement video, The section is detected as an advertisement video.

Japanese Patent No. 4421527 Japanese Patent No. 3407840 Japanese Patent No. 3513424

X.-S. Hua, L. Lu, and H.-J. Zhang. “Robust Learning-based TV Commercial Detection,” in Proceedings of IEEE International Conference on Multimedia & Expo., 2005. P. Duygulu, M.-Y. Chen, and A. Hauptmann. “Comparison and Combination of Two Novel Commercial Detection Methods,” in Proceedings of IEEE International Conference on Multimedia & Expo., Pp. 1267-1270, 2004.

  In the technology based on the reference data as described in Patent Document 1, it is necessary that the advertisement video to be detected is always known. However, there is a problem that it is practically impossible to obtain and store all advertisement videos in the past including newly published advertisement videos in advance and compare them.

  On the other hand, with a technique that does not assume reference data, the advertisement video to be detected does not necessarily have to be known. Therefore, although the above problems can be avoided, there are the following problems.

  In the technique described in Non-Patent Document 1, an advertisement video model is used. In order to obtain a general-purpose and high-accuracy model, it is necessary to obtain data including all advertisement videos in advance. However, in order to obtain such a model, a large amount of advertisement video data is required, so that there is a problem that it is difficult to obtain practical accuracy.

  In the technique of Non-Patent Document 2, since a model is not used, a large amount of data is not requested in advance. However, the amount of calculation becomes a problem. Usually, one hour of video data contains about 1,000 shots. To calculate the similarity of all these shots, 499,500 similarity calculations are required. Currently, due to multi-channel distribution etc., the time length of video delivered per hour is much larger than this, and it is expected that it will continue to increase in the future. Therefore, a technique with such a large amount of calculation is not practical.

  The techniques of Patent Literature 2 and Patent Literature 3 are extremely simple in that detection is performed only based on whether or not the boundary time interval matches the length of the advertisement video. Therefore, there are many problems of excessive detection (false detection) and low accuracy.

  As described above, conventionally, there has not been realized a video detection technique for efficiently and accurately detecting an advertising video from a large amount of video.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide a video detection method, a video detection device, and a video detection program capable of detecting an advertising video with high accuracy and efficiency. To do.

  In order to achieve the object, the invention according to the first aspect is a video detection method for detecting an advertising video from an input video to be processed, and analyzes the segment structure of the input video and divides it into partial sections. Based on the structure analysis process, one or more candidate sections including the advertisement video based on the characteristics of the partial sections, a candidate section detection process for obtaining a start time and an end time, a start time of the candidate sections, and Analyzing at least one of the image and sound of the partial section existing inside the end time and extracting the feature quantity, and a code for expressing the partial section with a code based on the feature quantity Code sequence detection that detects a sub-segment group that is a code sequence that recursively appears from a code sequence composed of one or more sub-intervals represented by the code And sense, of the detected subinterval group, characterized in that it comprises a filtering process for outputting only a predetermined condition is satisfied as an advertisement video section.

  The invention according to a second aspect is the invention according to the first aspect, wherein the structural analysis process analyzes a cut point that is a break of the input video, and at least a speech section or a music section among these cut points. It is characterized in that the time of no point and the time length to the next point satisfying the same condition are obtained.

  The invention according to a third aspect is the invention according to the first aspect, wherein in the feature extraction process, the luminance average value, variance, color average value for each partial region, variance, luminance histogram, color histogram, It is characterized in that at least one of the MFCCs is extracted as a feature quantity.

  The invention according to a fourth aspect is the invention according to the first aspect, wherein in the restart code sequence detection process, a restart code sequence is detected by repeatedly detecting a partial code sequence having a certain number of occurrences. The filtering process is characterized in that only a restart code sequence having a predetermined time length is output.

  In order to achieve the above object, the invention according to a fifth aspect is a video detection device that detects an advertising video from an input video to be processed, and analyzes the section structure of the input video to make a partial section. Based on the characteristics of the partial section, a structure analysis section to be divided, a candidate section detection section that detects one or more candidate sections including the advertisement video and obtains a start time and an end time thereof, and a start of the candidate section Analyzing at least one of the image and sound of the partial section existing inside the time and the end time, and extracting a feature amount, and expressing the partial section by a code based on the feature amount And a restart code sequence detection unit for detecting a partial section group that is a code sequence that recursively appears from a code sequence constituted by one or more partial sections expressed by the code , Among the detected partial section group, characterized in that it comprises a filter unit for outputting only a predetermined condition is satisfied as an advertisement video section.

  The invention according to a sixth aspect is the invention according to the fifth aspect, wherein the structural analysis unit analyzes a cut point that is a break of the input video, and at least a speech section or a music section among these cut points. It is characterized in that the time of no point and the time length to the next point satisfying the same condition are obtained.

  The invention according to a seventh aspect is the invention according to the fifth aspect, wherein the feature extraction unit includes a luminance average value for each partial region, a variance, a color average value for each partial region, a variance, a luminance histogram, a color histogram, It is characterized in that at least one of the MFCCs is extracted as a feature quantity.

  The invention according to an eighth aspect is the invention according to the fifth aspect, wherein the restart code sequence detection unit detects a restart code sequence by repeatedly detecting a partial code sequence having a certain number of occurrences. The filter unit outputs only a restart code sequence having a predetermined time length.

  In order to achieve the above object, an invention according to a ninth aspect is a video detection program that causes a computer to execute each process according to any one of the first to fourth aspects.

  According to the present invention, it is possible to provide a video detection method, a video detection device, and a video detection program capable of detecting an advertising video with high accuracy and efficiency.

It is a figure which shows the structural example of the image | video detection apparatus in embodiment of this invention. It is a flowchart which shows operation | movement of the image | video detection apparatus in embodiment of this invention. It is a figure which shows an example of the structure-analysis process in embodiment of this invention. It is a figure which shows an example of the candidate area detection process in embodiment of this invention. It is a figure which shows an example of the feature-value extraction position in embodiment of this invention. It is a figure which shows an example of the information memorize | stored in the memory | storage device in embodiment of this invention. It is a figure which shows an example of the restart code sequence detection process in embodiment of this invention. It is a figure which shows an example of the information memorize | stored in the memory | storage device in embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration example of a video detection device 1 according to an embodiment of the present invention. The video detection apparatus 1 is an apparatus that receives video stored in the video database 108, detects an advertising video from the input video, and stores the advertisement video in the storage device 107. The structure analysis unit 101, the candidate section detection unit 102, a shot feature extraction unit 103, an encoding unit 104, a restart code sequence detection unit 105, a filter unit 106, and a storage device 107. These processing units are realized by computer hardware and software programs including a CPU, a memory, an external storage device, and the like. In the present embodiment, the video database 108 is external to the video detection device 1 and is connected by a communication network that can communicate with each other. Further, the case where the storage device 107 is inside the video detection device 1 is illustrated. Of course, the video database 108 may be inside the video detection device 1, and the storage device 107 may be outside the video detection device 1.

  FIG. 2 is a flowchart showing the operation of the video detection apparatus 1.

  First, the structure analysis unit 101 executes a video structure analysis process including at least shot analysis on a video registered in the video database 108, and the video is made into a meaningful partial section (hereinafter simply referred to as “section”). Divide (step S201). Subsequently, the candidate section detection unit 102 receives the processing result of the structure analysis unit 101, detects only a section that is assumed to include the advertisement video to be detected as a candidate section, and the processing result of the structure analysis unit 101 At the same time, it is stored in the storage unit 107 (step S202). Subsequently, the shot feature extraction unit 103 extracts a feature amount for each shot obtained as a processing result of the structure analysis unit 101 with respect to the candidate section of the video registered in the video database 108 (step S203). Subsequently, the encoding unit 104 executes an encoding process for expressing a shot as a code based on the shot feature amount extracted by the shot feature extraction unit 103, and stores it in the storage unit 107 (step S204). Subsequently, the restart code sequence detection unit 105 reads the code sequence of the shot stored in the storage unit 107 and enumerates the code sequences that have been restarted (occurred twice or more) from the code sequence of the shot (step S205). . Finally, the filter unit 106 extracts a sequence satisfying the condition of the advertisement video from the listed reoccurrence code sequences, and stores it in the storage unit 107 (step S206). In this way, the video detection device 1 can detect an advertising video from a large amount of video with high accuracy and efficiency without using reference data, an advertising video model, and similarity between shots.

  Hereinafter, an example of each process executed by the video detection device 1 will be described in detail. The subsequent processes may be executed for all the videos registered in the video database 108 or may be executed for a specific part of the videos. The video database 108 may be attached with metadata indicating what kind of video each video is, but in the present embodiment, advertisement video is detected without assuming any metadata. To do. Also, it is assumed that the video to be processed is a series of data without losing generality.

[Step S201: Structural analysis processing]
The structure analysis unit 101 performs a structure analysis process on the video registered in the video database 108. The structural analysis here is to divide the video into meaningful section units, and in this embodiment, the shot between the cut points and the shots between the cut points, the advertisement video section and others (video main part etc.) The purpose is to find a boundary candidate with the video section. Therefore, it is necessary to analyze at least shots, that is, image frame breaks (cut points). Moreover, in order to obtain the latter boundary candidate, it is preferable to analyze the break of sound as described below.

  First, various known methods can be used for shot analysis. For example, the shot detection method described in Reference 1 may be used.

[Reference 1]: Y. Tonomura, A. Akutsu, Y. Taniguchi, and G. Suzuki, “Structured Video Computing”, IEEE Multimedia, vol.1, no.3, pp.34-43, 1994.
Only the information of this shot may be used, and the cut point may be immediately set as a boundary candidate. However, this alone is not necessarily a boundary between the advertising video section and other video sections. Usually, the advertisement video sections are sparsely distributed with respect to the entire video, so a considerable number of excessive detections are expected. Therefore, the boundary candidates are preferably narrowed down using sound breaks. This will be described with reference to FIG.

  As shown in this figure, based on a shot 302 obtained for the original video 301, a video break can be found. On the other hand, the advertisement video section and the other video sections are not continuous as a video and are irrelevant, so that the sound is always discontinuous (disconnected) at the boundary. Therefore, boundary candidates can be narrowed down by using information of sound breaks in combination.

  As a method for analyzing a sound break, sound in a video is analyzed, and for example, a voice frame that is neither a speech section nor a music section can be used as a break. Since a known music / speech segment detection technique described in Reference 2 or the like can be used to detect a speech segment and a music segment, a segment that does not apply to any of these may be used as a boundary.

[Reference 2]: K. Minami, A. Akutsu, H. Hamada, and Y. Tonomura, “Video Handling with Music and Speech Detection”, IEEE Multimedia, vol.5, no.3, pp.17-25, 1998.
A boundary candidate is found from the shot 302 thus obtained and the utterance or music section 303. That is, since a section that is a shot boundary (cut point) and is neither a speech section nor a music section is a candidate boundary, 304 boundary candidates are obtained as a result. As shown in this figure, by utilizing the sound break information, the boundary candidates 304 can be narrowed down more efficiently than the boundary obtained by the shot 302 alone. The boundary candidate time thus analyzed and the start or end time of each shot are output to the candidate section detection unit 102.

  Here, as in Patent Document 2 or Patent Document 3, it is possible to find a section where the interval between detected boundary candidates is 15 seconds or 30 seconds, and output this as an advertisement video. However, there is a problem in that the section that is not the advertisement video is excessively detected only under this condition, and the accuracy is extremely low. As will be described later, in the present embodiment, highly accurate detection is realized by introducing effective shot feature amount extraction and encoding, and efficient detection of a restart code sequence.

  The above is an example of the processing details of the structural analysis processing.

[Step S202: Candidate Section Detection Processing]
Subsequently, the candidate section detection unit 102 detects only sections that are assumed to include the advertisement video to be detected as candidate sections, and stores them in the storage unit 107 together with the processing result of the structure analysis process in step S201. In this candidate section detection process, sections that may be advertisement videos are left in the entire video, and other sections are thinned out.

  In general, the advertisement video is devised to attract the viewer's attention. For example, there are many cases in which shot division is made fine to increase screen changes, and sound effects and music characteristic of the sound are inserted. Therefore, a video section having no such feature is unlikely to be an advertising video. In the following, an example of candidate section detection processing based on this observation will be described with reference to FIG. In this example, the case where a candidate section is detected based on the frequency of shots (cuts) is described.

  As shown in FIG. 4, an image and a shot obtained through the structure analysis processing in step S201 are obtained correspondingly. First, a window having a window length W and a shift WS is provided for an image. The window length W represents the size (unit time range) of the window. The shift WS represents the time length for moving the window. W and WS may be designated by arbitrary values. For example, FIG. 4 illustrates a case where W is 30 seconds and WS is 30 seconds. The number of shots #S is counted in units of the window, and shots included in the window having a value greater than or equal to a certain value are detected as candidate sections. In the example of FIG. 4, shots included in a window where #S is 4 or more (even partially) are detected as candidate sections.

  In the example of FIG. 4, since W and WS have the same length, windows do not overlap each other, and #S is calculated without overlap. However, W and WS do not necessarily have to be the same time. If W is 30 seconds and WS is 15 seconds, the windows overlap each other for 15 seconds. In any case, the number of shots included in the window length W is counted and #S is not changed. However, it is preferable that the two closest windows are continuous and there is no blank section between them, and it is preferable that W ≧ WS.

  If a candidate section is obtained using only #S in window units, there are cases where accurate candidate section detection cannot be performed due to the influence of noise. In such a case, smoothing processing, for example, noise suppression processing such as k-neighboring or major voting may be introduced.

  Information indicating the time of the boundary candidate obtained in step S201, the start (or end) time and time length of the shot that became the candidate section, and whether the start (or end) time of each shot is a boundary candidate. Store in the storage unit 107.

  The above is an example of the processing details of step S202.

[Step S203: Shot Feature Extraction Processing]
Subsequently, the shot feature extraction unit 103 extracts a feature amount for each shot with respect to a video candidate section registered in the video database 108. Each candidate section detected in step S202 includes one or more shots. In this process, a feature amount representing the feature of the shot is extracted from each shot. As the feature amount to be extracted, a luminance value statistic, a color statistic, or the like can be used as image information. Alternatively, a Bag-of-Visual-Words histogram described in Reference 3 or GIST described in Reference 4 may be used.

[Reference 3]: J. Sivic and A. Zisserman. “Video Google: A Text Retrieval Approach to Object Matching in Videos”, In Proc. International Conference on Computer Vision (ICCV), pp. 1470-1477, 2003.
[Reference 4]: A. Oliva and A. Torralba. “Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope”. International Journal of Computer Vision (IJCV), vol. 42, no. 3, pp. 145-175, 2001.
As sound information, fundamental frequency, volume, or Mel Frequency Cepstrul Coefficients (MFCC) may be used. Moreover, you may utilize combining the feature-value regarding the said image and sound one or more.

  On the other hand, when many feature quantities are used, the calculation time is required. In particular, when processing a large amount of video, it is preferable that the calculation time be as short as possible. In this case, for example, (1) luminance average value and variance for each partial region, (2) color average value and variance for each partial region, (3) luminance histogram, (3) color histogram, and (4) MFCC, Any one of them is preferably adopted as the feature amount. Here, the partial area refers to each area when one image is divided into, for example, 3 × 3. These feature quantities are effective feature quantities that can be extracted at high speed and exhibit high accuracy in later detection.

  In addition, since a shot itself includes a plurality of image frames and a sound signal having a certain length, the image frame or the sound signal is arbitrarily extracted from which image feature amount is extracted. In this embodiment, when a shot is divided into n equal parts in the time axis direction, a feature amount is extracted from an image frame or sound signal in the middle of each. FIG. 5 is a diagram illustrating positions where image features are extracted when the image is divided into two equal parts. In this example, 14 image frames F1 to F14 are included. When dividing into two equal parts, seven image frames F1 to F7 are included in the preceding section, and seven image frames F8 to F14 are included in the following section. A feature amount is extracted from the frames F4 and F11 at the intermediate position. The method of selecting n is arbitrary, but for example, n = 1 may be set in order to shorten the calculation time. In this case, a feature amount is extracted from exactly one frame in the shot. The extracted feature amount is output to the encoding unit 104 as a shot feature amount.

  The above is an example of the processing details of step S203.

[Step S204: Encoding Process]
Subsequently, the encoding unit 104 executes an encoding process for expressing a shot as a code based on the shot feature amount extracted in step S <b> 203, and stores it in the storage unit 107. Although there are several methods for encoding, two methods will be described here. One is a method based on vector quantization, and the other is a method using a hash.

  In the method based on vector quantization, a code book for encoding shot feature quantities is prepared in advance, and shot feature quantities are encoded based on the code book. Various known methods can be used as a method for creating a codebook. For example, a codebook is created by applying a clustering method such as a k-means method. When encoding, the distance between the center vector of each cluster and the shot feature amount is calculated, and the id of the cluster having the closest distance is assigned as a code.

  In the method using a hash, for example, the following procedure is used. First, a vector of shot feature amounts (or portions thereof) is represented as f = (f1, f2,..., Fd). At this time, a plurality of threshold values are given as f′i (i = 1, 2,..., M), and converted to another value b based on whether or not the value of each dimension of the vector is greater than or equal to this threshold value. .

  For example, suppose f is three-dimensional and f = (0.8, 0.1, 0.5). At this time, it is assumed that three threshold values are prepared, and f′1 = 0.2, f′2 = 0.5, and f′3 = 0.7, respectively. At this time, for example, 0001 is assigned to a value less than f'1, 0010 is assigned to a value greater than or equal to f'1 and less than f'2, 0100 is assigned to a value greater than or equal to f'2 and less than f'3, and 1000 is assigned to a value greater than or equal to f'3. It shall be. Since f1 is 0.8, f′3 or more is 1000, and f2 is 0.1, so it is less than f′1, 0001, and f3 is 0.5, so it is f′2 or more, so 0100. Then, b at this time can be converted into a hash value of b = 100000010100. A sign can be obtained by this value. As this threshold value, an arbitrary value may be used. For example, a statistic of each vector element and a value obtained by coordinate transformation of the statistic can be used.

  As described above, since the sequence of shots included in each candidate section can be converted into a sequence of codes when the processing of step S201 to step S204 is completed, these are stored in the storage unit 107. FIG. 6 shows an example of information stored in the storage unit 107 at the end of step S204. As shown in FIG. 6A, the table 61 stores information on candidate sections (candidate section id, section start time, section end time). Further, as shown in FIG. 6B, the table 62 includes shot information (shot id, id of candidate section to which the shot belongs, shot start time, shot time length, and whether the shot is a boundary candidate. And information indicating a shot and a symbol indicating a shot) are stored. If shots are represented by codes A, D, E,... Instead of numerical vectors in this way, it is possible to significantly reduce the amount of calculation compared to the case of calculating shot similarity as in the prior art. Become.

  The above is an example of the processing details of step S204.

[Step S205: Restart Code Sequence Detection Process]
Subsequently, the restart code sequence detection unit 105 reads the code sequence of the shot stored in the storage unit 107 and enumerates the restart code sequence from the code sequence of the shot.

  The same advertisement video is repeatedly used over a plurality of channels and times. Therefore, by extracting a code sequence that appears again in all candidate sections, a section that seems to be an advertisement video can be obtained.

  However, on the other hand, it is known that such a process for obtaining a restart code sequence requires a lot of calculation time. Until now, there are some known techniques for solving the same problem, for example, the techniques described in Reference 5, Reference 6, and Reference 7 can be used.

[Reference 5]: Japanese Patent No. 3759438 [Reference 6]: R. Agrawal and R. Srikant. “Mining sequential patterns”, In Proc. International Conference on Data Engineering (ICDE), pp. 3-14, 1995 .
[Reference 7]: J. Pei, J. Han, B. Mortazavi-Asl, H. Pinto, Q. Chen, U. Dayal, and M.-C. Hsu. “Prefixspan: Mining sequential patterns by prefix-projected growth ”, In Proc. International Conference of Data Engineering (ICDE), pp. 215-224, 2001.
All of the above techniques are effective but are not optimized for advertising video detection. Therefore, in the present embodiment, more efficient processing is realized by considering the following viewpoints.

(1) List only continuous code sequences (2) Advertising video has a reference length (15 seconds, 30 seconds, 60 seconds, etc.)
It is preferable to use an efficient process which is based on the PrefixSpan described in Reference 7 of the depth priority search type, combined with the filter process in step S206 described later, and corrected. A depth-first search requires less memory than a width-first search. Hereinafter, an example of the process for obtaining the restart code sequence will be described with reference to FIG.

  First, a code sequence in which candidate sections and codes of shots included therein are arranged in time order is obtained from the storage unit 107. In FIG. 7, four candidate sections 1 to 4 are obtained, and each has the following code sequence.

Candidate section 1: {A, C, D, E, F, G, A, C, F}
Candidate section 2: {A, B, C, E, F, G, F, D}
Candidate section 3: {C, E, F, B, Y, E, B, C}
Candidate section 4: {A, D, E, H, G, E}
First, the number of codes (number of occurrences) appearing in these candidate sections is counted. In this example, A: 4, B: 3, C: 5, D: 3, E: 6, F: 5, G: 3, H: 1, Y: 1. Among them, a code having a certain number of occurrences is stored as a processing target. Thereafter, the processing is sequentially performed on these codes. Here, the processing is executed based on the depth-first search, and attention is first focused on E having the maximum number of occurrences. Among each candidate section, a (partial) candidate section is newly generated by a code sequence appearing after this at E. In the example of FIG. 7, the candidate section 1 is {A, C, D, E , F, G, A, C, F}, so the section {F, G, A, C, F} that appears after E Is a candidate section 1a. Thereafter, similarly, the following new (partial) candidate section is generated.

Candidate section 1a: {F, G, A, C, F}
Candidate section 2a: {F, G, F, D}
Candidate section 3a: {F, B, Y, E, B, C}
Candidate section 3b: {B, C}
Candidate section 4a: {H, G, E}
Next, when the number of occurrences of the code that appears first in the candidate sections 1a to 4a is counted, F: 3, B: 1, H: 1. The partial candidate section is generated by paying attention to F that has the maximum number of occurrences. At the same time, F is stored, and a restart code sequence {E, F} is found together with E stored previously.

  By repeating the above processing, it is possible to enumerate restart code sequences. In this example, since depth-first search is premised, the processing is first executed by paying attention to E and F that are the maximum number of occurrences, but the processing is similarly performed for other codes such as C, for example. It is preferable to repeat the same processing by paying attention to all codes that have a certain number of occurrences in each candidate section or partial candidate section.

  In addition, the code sequence may fluctuate even in the same advertisement video due to the influence of noise. As a countermeasure to such a case, by allowing a certain number of codes or less may be replaced, or by introducing a threshold value so as to be regarded as the same code sequence when the editing distance is less than a certain value, Can improve robustness. At the same time, the following effects can be obtained.

That is, there are usually multiple versions of advertisement video such as a 15-second version, a 30-second version, and a 60-second version, and there are cases where it is desired to manage these versions as an advertisement video for advertising the same one. These different versions of the advertising video are not completely different videos, they are partly in common with each other, and many shots are often inserted / deleted or replaced. Differences can be quantified and absorbed by code replacement and editing distance.
Through the above processing, all the found code sequence sections found are output to the filter unit 106.

  The above is an example of the processing details of step S205.

[Step S206: Filter Processing]
Finally, the filter unit 106 extracts a sequence satisfying the condition of the advertisement video from the enumerated restart code sequences and stores the sequence in the storage unit 107.

  In Japan, the advertisement video usually has a predetermined time length such as 15 seconds, 30 seconds, 60 seconds, or the like. Therefore, only those having a predetermined time length are filtered out (adopted) from the extracted recurring code sequences, and finally detected as advertisement video. The length of the detected restart code sequence can be obtained by referring to the time length of the shot stored in the storage unit 107. When the length of the restart code sequence is obtained, it is determined whether or not the time length of the advertisement video is applicable to a predetermined time length, and only the applicable one is adopted. However, at this time, a restart code sequence starting or ending with a shot (cut) determined not to be a boundary candidate in step S201 is not adopted. In addition, the time length of 15 seconds or the like may not always be accurate due to the influence of the time difference of the original advertisement video, the error of the structure analysis process, the rounding error, and the like. Therefore, it is preferable to provide a certain allowable range (for example, ± 0.5 seconds) and allow a repeated code sequence having a time length that falls within that range.

  In the flowchart shown in FIG. 2, after all the restart code sequences are enumerated once (step S205), the filtering process (step S206) is executed for all the restart code sequences listed. However, in practice, it may be possible to sequentially determine whether or not to adopt the restart code sequence by applying filter processing at the timing when one restart code sequence is found. By doing so, it is possible to forget a re-occurring code sequence that is not necessary successively, and thus it is possible to improve efficiency from the viewpoint of memory usage.

  The filtered re-occurrence code sequence information is stored in the storage unit 107 as an advertisement video section, and the process is terminated. For example, as shown in FIG. 8, the table 63 stores a start time or end time, a time length, and a restart code sequence. Such information of the advertisement video section may be output so that the user can refer to it as it is.

  The above is an example of the processing details of step S206.

  As described above, in the video detection apparatus 1 according to the embodiment of the present invention, a code sequence is obtained by analyzing video image / sound information, and a partial section group that is a code sequence that reoccurs is detected. ing. Therefore, it is possible to detect an advertising video from a large amount of video with high accuracy and efficiency without using reference data, an advertising video model, and similarity between shots.

  In the structural analysis process S201, cut points that are breaks in the input video are analyzed, and at least the time of a point that is neither a speech segment nor a music segment among these cut points, and the time length to the next point that satisfies the same condition Asking for. Therefore, it is possible to efficiently narrow down the boundary that separates the advertisement video from the other video.

  In the feature extraction process S203, at least one of the luminance average value, variance, color average value, variance, luminance histogram, color histogram, and MFCC for each partial region is extracted as a feature amount. Yes. Therefore, it is possible to extract feature amounts at a higher speed than in the conventional feature extraction process, and it is possible to detect an advertisement video with high accuracy.

  In the restart code sequence detection processing S205, a restart code sequence is detected by repeatedly detecting a partial code sequence having a certain number of occurrences. In the filter processing S206, only the restart code sequence having a predetermined time length is output. Like to do. Therefore, it is possible to detect a restart code sequence extremely efficiently as compared with the conventional restart code sequence detection process.

  In the reoccurrence code sequence detection process S205, a partial section group that becomes a recurring code sequence is detected. However, when a code sequence (reference data) is stored in advance, the same as that code sequence is stored. It is also possible to detect a partial section group that is a code sequence of. That is, when the advertisement video obtained once is detected from another new video, since the time length and code sequence of the advertisement video have already been obtained, this is stored in the storage device 107 as reference data. deep. In this case, it is not necessary to execute all steps S201 to S206. That is, after executing at least step S201, step S203, and step S204 to convert the video into a code sequence, the video section that matches the time length and code sequence of each advertisement video is determined based on the reference data stored in advance. Just scan. Thus, if the advertisement video obtained once is stored as reference data, the process can be executed more efficiently.

  The video detection device 1 according to the embodiment of the present invention has been described in detail above. It goes without saying that such a video detection method can be executed on a computer using a software program, and the present invention is not limited to an example of the described embodiment, and the technical description described in the claims. Various modifications can be made in the range. For example, the present invention can be used for various video distribution / communication services such as IPTV, digital signage, VOD (Video on Demand), and terrestrial digital broadcast retransmission.

DESCRIPTION OF SYMBOLS 1 ... Video | video detection apparatus 101 ... Structure analysis part 102 ... Candidate area detection part 103 ... Shot feature extraction part 104 ... Coding part 105 ... Reoccurrence code sequence detection part 106 ... Filter part 107 ... Memory | storage device 108 ... Video database

Claims (9)

A video detection method for detecting advertisement video from input video to be processed,
Analyzing the section structure of the input video and dividing it into partial sections;
Candidate section detection processing for detecting one or more candidate sections including the advertisement video based on the characteristics of the partial sections and obtaining the start time and end time;
A feature extraction process of analyzing at least one of the image and sound of the partial section existing inside the start time and end time of the candidate section, and extracting a feature amount;
An encoding process for expressing the partial section by a code based on the feature amount;
A recursive code sequence detection process for detecting a partial section group to be a recurring code sequence from a code sequence composed of one or more partial sections expressed by the code;
A filtering process for outputting only those satisfying a predetermined condition among the detected partial section groups as an advertising video section;
A video detection method comprising:   2. The video detection method according to claim 1, wherein in the structure analysis process, a cut point that is a break of the input video is analyzed, and a time of a point that is not at least an utterance section or a music section among these cut points, and A video detection method characterized in that a time length to the next point satisfying the same condition is obtained.   The video detection method according to claim 1, wherein in the feature extraction process, at least one of luminance average value, variance, color average value, variance, luminance histogram, color histogram, and MFCC for each partial region. A video detection method characterized by extracting one as a feature amount.   The video detection method according to claim 1, wherein in the restart code sequence detection process, a restart code sequence is detected by repeatedly detecting a partial code sequence having a certain number of occurrences, and in the filter process, a predetermined code sequence is detected. A video detection method characterized by outputting only a restart code sequence having a time length. A video detection device for detecting an advertising video from an input video to be processed,
A structure analysis unit that analyzes a section structure of the input video and divides the section structure into partial sections;
A candidate section detecting unit that detects one or more candidate sections including the advertisement video based on the characteristics of the partial section and obtains a start time and an end time;
A feature extraction unit that analyzes at least one of the image and sound of the partial section existing inside the start time and end time of the candidate section, and extracts a feature amount;
An encoding unit for expressing the partial section by a code based on the feature amount;
A recursive code sequence detection unit that detects a partial section group that becomes a recurring code sequence from a code sequence composed of one or more partial sections expressed by the code;
Among the detected partial section group, a filter unit that outputs only an advertisement video section that satisfies a predetermined condition;
A video detection apparatus comprising:   6. The video detection apparatus according to claim 5, wherein the structure analysis unit analyzes a cut point that is a break in the input video, and at least a time of a point that is neither an utterance section nor a music section among the cut points, and An image detection apparatus characterized by obtaining a time length to the next point satisfying the same condition.   6. The video detection apparatus according to claim 5, wherein the feature extraction unit includes at least one of luminance average value, variance, color average value, variance, luminance histogram, color histogram, and MFCC for each partial region. A video detection device that extracts one as a feature amount.   The video detection device according to claim 5, wherein the restart code sequence detection unit detects a restart code sequence by repeatedly detecting a partial code sequence having a predetermined number of occurrences, and the filter unit A video detection apparatus that outputs only a restart code sequence having a time length.   A video detection program for causing a computer to execute each of the processes according to any one of claims 1 to 4.

Images