JP2010186307A - Moving image content identification apparatus and moving image content identification method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable identification of moving image content even if a part on a temporal axis of reference moving image content is cut out or a telop or a logo is inserted into the reference moving image content, and to rapidly perform processing thereof. <P>SOLUTION: A moving image content identification apparatus includes: a moving image content input unit 11 for inputting moving image content; a feature amount extraction unit 12 for selecting a key frame from a plurality frames constituting the input moving image content to extract a feature amount of the selected key frame; and a database search unit 14 for, if identification moving image content has been input from the moving image content input unit 11 and the feature amount extraction unit 12 has extracted a feature amount of the key frame included in the identification moving image content, searching for a feature amount of a key frame of reference moving image content stored in a database 13 for each extracted feature amount. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

The present invention relates to a moving image content identification apparatus and a moving image content identification method for determining whether moving image content stored in a hard disk drive, other media, network storage, or the like includes a part of specific moving image content.

  With the spread of broadband in recent years and the increase in storage capacity of HDDs (Hard Disk Drives), DVDs (Digital Versatile Disks), Blu-ray discs, etc., digital content has not been approved by copyright holders or content providers. In addition, sharing and disclosing via a network has become easier, and such illegal sharing and disclosing has become a problem. Technology to automatically detect specific contents that the copyright holder has not permitted free distribution from among a plurality of digital contents by using fingerprints (features) of the digital contents for such problems. Has been proposed.

  In Patent Document 1, the feature amount of content is described using three-dimensional frequency analysis and principal component analysis. This method performs 3D frequency analysis by adding frequency analysis (FFT) in the time axis direction to the coefficient obtained by spatial frequency analysis (DCT), and then the coefficient obtained by 3D frequency analysis by principal component analysis. Feature values are extracted from. In Patent Document 2, the specific content similar to the distributed content is narrowed down using the feature amount used in Patent Document 1, and when it cannot be narrowed down, the most similar to the distributed content using the phase-only correlation method Specific content is determined, and it is determined whether or not the same content is based on a threshold value.

  In Non-Patent Document 1, even when temporal editing such as part of the video is cut out by extracting a feature amount called color layout from the entire frame of the video and sequentially matching a plurality of frames. Detection is possible.

  Further, in Non-Patent Document 2, a feature point called a corner is detected from each frame of an image, a feature amount is extracted from the periphery thereof, and each feature point is matched to perform editing such as clipping. Even if there is, illegally distributed content can be detected.

JP 2005-18675 A JP 2006-285907 A

E. Kasutani and A. Yamada, “The MPEG-7 color layout descriptor: a compact image feature description for high-speed image / video segment retrieval,” in Proc. Of ICIP, 2001, pp. 674-677. J. Law-To et al., “Video Copy Detection: A Comparative Study,” in Proc. ACM CIVR’07, pp. 371-378, 2007.

  However, in the methods disclosed in Patent Documents 1 and 2, since one feature amount is extracted from one moving image content, for example, when editing in the time axis direction such as dividing the moving image content into two is performed. There is a problem that detection is impossible. In the method disclosed in Non-Patent Document 1, since only one feature amount is extracted from the entire screen, there is a problem that detection cannot be performed if spatial editing such as inserting a telop or a logo is performed. Further, in the method disclosed in Non-Patent Document 2, dozens of feature points are extracted from one screen and all of them are matched, so that there is a problem that it takes too much time to extract and match the feature points. is there.

  The present invention has been made in view of such circumstances, and even if a part of the reference video content on the time axis is cut out or a telop or logo is inserted into the reference video content, the video content is identified. An object of the present invention is to provide a moving image content identification device and a moving image content identification method capable of performing the processing at high speed.

  (1) In order to achieve the above object, the present invention takes the following measures. That is, the moving image content identification device of the present invention is a moving image content identification device that determines whether or not the identified moving image content that is the identification target includes all or part of the reference moving image content that is the identification criterion. A video content input unit that selects a key frame from a plurality of frames constituting the input video content, and extracts a feature amount of the selected key frame; and the video content input unit When the identified moving image content is input from and the feature amount extraction unit extracts the feature amount of the key frame included in the identified moving image content, the reference stored in the database for each extracted feature amount A database search unit that searches for keyframe feature quantities of video content, and As a result of the search, based on the number of feature quantities of the reference video content corresponding to the feature quantity of the identified video content, it is determined whether or not the identified video content includes all or part of the reference video content. It is characterized by.

  In this way, the feature amount of the key frame of the identified moving image content is extracted, the feature amount of the key frame of the reference moving image content stored in the database is searched for each extracted feature amount, and the feature amount of the identified moving image content It is determined whether or not the identified video content includes all or part of the reference video content based on the number of feature quantities of the reference video content corresponding to the reference video content. Video content can be identified even when a part of the time axis is cut out or a telop or logo is inserted in the reference video content, which can reduce search time and improve search accuracy. Become.

  (2) Further, in the moving image content identification device of the present invention, the feature amount extraction unit, when the identification moving image content is input from the moving image content input unit, displays a time code of a key frame of the selected identification moving image content. The database search unit obtains a difference value between a time code in the feature amount of the identified moving image content and a time code in the feature amount of the reference moving image content corresponding to the feature amount of the identified moving image content. It is calculated for each feature amount of content, and when the number of the difference values having the same value is equal to or greater than a predetermined threshold, it is determined that the identified moving image content includes all or part of the reference moving image content. It is said.

  In this way, a difference value between the time code in the feature amount of the identified video content and the time code in the feature amount of the reference video content corresponding to the feature amount of the identified video content is calculated for each feature amount of the identified video content, If the number of the difference values having the same value is equal to or greater than a predetermined threshold value, it is determined that the identified video content includes all or part of the reference video content. Even when a part of the content on the time axis is cut out, the moving image content can be identified.

  (3) In the moving image content identification device of the present invention, the reference moving image content is input from the moving image content input unit, and the feature amount of the key frame included in the reference moving image content is extracted by the feature amount extraction unit. A feature amount storage unit for storing the feature amount of the key frame included in the extracted reference moving image content.

  As described above, when the reference video content is input from the video content input unit, and the feature amount of the key frame included in the reference video content is extracted by the feature amount extraction unit, the key frame included in the extracted reference video content is extracted. Since the feature amount is accumulated and the database is constructed, it is possible to construct a database of reference moving image content serving as an identification standard.

  (4) Further, in the moving image content gap identification device according to the present invention, the feature amount extraction unit sets key frames independently for a plurality of preset regions.

  In this way, since key frames are set independently for a plurality of preset areas, a part of the reference video content on the time axis may be cut out, or a telop or logo may be inserted into the reference video content In addition, it is possible to identify moving image content.

  (5) In the moving image content identification apparatus of the present invention, the feature amount extraction unit extracts feature amounts from a plurality of regions set in advance from the key frame.

  In this way, feature quantities are extracted from each of a plurality of preset areas from the key frame, so even if captions or logos are inserted into the key frame of the identification video content, high search accuracy is achieved. It becomes possible to do.

  (6) In the moving image content identification device according to the present invention, the feature amount storage unit creates an index for each extracted feature amount.

  As described above, since an index is created for each extracted feature amount, it is possible to increase the processing speed when performing a search using the feature amount of the identified moving image content.

  (7) Further, in the moving image content identification device of the present invention, the database search unit sets importance for the key frame and the feature amount of the identified moving image content extracted by the feature amount extraction unit. Yes.

  In this way, since the importance level is set for the key frame and the feature amount of the identified moving image content extracted by the feature amount extraction unit, the search is terminated in a certain time by performing the search from the feature amount having the higher importance level. Even in such a case, it is possible to maintain high search accuracy.

  (8) In the moving image content identification device of the present invention, the importance is set based on temporal stability of key frames and specificity of feature amounts.

  In this way, since the importance is set based on the temporal stability of the key frame and the specificity of the feature amount, the search can be terminated in a certain time, and the accuracy at that time should be maintained as high as possible. Is possible.

  (9) Further, the moving image content identification method of the present invention is a moving image content identification method for determining whether or not the identified moving image content to be identified includes all or part of the reference moving image content that is the identification criterion, The step of inputting the moving image content from the moving image content input unit, and the feature amount extraction unit select a key frame from a plurality of frames constituting the input moving image content, and extract the feature amount of the selected key frame And when the identification video content is input from the video content input unit and the feature amount extraction unit extracts the feature amount of the key frame included in the identification video content, the database search unit extracts the extracted feature amount. The key of the reference video content stored in the database for each feature amount Searching for a feature amount of a frame; and, as a result of the search, based on the number of feature amounts of the reference moving image content corresponding to the feature amount of the identified moving image content, the identified moving image content includes all of the reference moving image content or And a step of determining whether or not a part is included.

  In this way, the feature amount of the key frame of the identified moving image content is extracted, the feature amount of the key frame of the reference moving image content stored in the database is searched for each extracted feature amount, and the feature amount of the identified moving image content It is determined whether or not the identified video content includes all or part of the reference video content based on the number of feature quantities of the reference video content corresponding to the reference video content. Video content can be identified even when a part of the time axis is cut out or a telop or logo is inserted in the reference video content, which can reduce search time and improve search accuracy. Become.

  According to the present invention, the feature amount of the key frame of the identified moving image content is extracted, the feature amount of the key frame of the reference moving image content stored in the database is searched for each extracted feature amount, and Since whether or not the identified video content includes all or part of the reference video content is determined based on the number of feature quantities of the reference video content corresponding to the feature amount, all or part of the identified video content is referred to Even if a part of the video content on the time axis is cut out or a telop or logo is inserted in the reference video content, it is possible to identify the video content, thereby shortening the search time and improving the search accuracy. It becomes possible.

It is a block diagram of the moving image content identification apparatus which concerns on embodiment of this invention. Is a diagram illustrating an example of setting the rectangular region R ^i. Is a diagram illustrating an example of setting the rectangular region R ^i. Is a diagram illustrating an example of setting the rectangular region R ^i. Select one of the frames is a view showing a state of extracting features of the rectangular region R ⁱ x ⁱ (t). It is a figure which shows the method of extracting the feature-value by Color layout. It is a figure which shows the concept which selects a key frame candidate. 4 is a flowchart showing an operation of a feature amount extraction unit 12. It is a flowchart which shows the operation | movement which selects a key frame. It is a flowchart which shows a database search operation | movement.

  Embodiments of the present invention will be described below with reference to the drawings. In the present embodiment, copyright content to be detected in advance is input as reference moving image content, feature quantities are extracted, and a database is constructed. Thereafter, for the input identified moving image content, it is determined whether or not a part of the reference moving image content is included in the identified moving image content.

  FIG. 1 is a block diagram of a moving image content identification apparatus according to an embodiment of the present invention. As shown in FIG. 1, the moving image content identification device 10 includes a moving image content input unit 11, a feature amount extraction unit 12, a database 13, a database search unit 14, and a feature amount storage unit 15. These components are connected to the control bus 16 and can transmit / receive signals to / from each other.

  The moving image content identification device 10 inputs the reference moving image content and the identified moving image content from the moving image content input unit 11. The moving image content input unit 11 inputs reference moving image content provided by the copyright holder or content provider. Normally, a plurality of reference moving image contents are input, but in the present embodiment, all input reference moving image contents are connected and handled as one moving image input. The moving image content input unit 11 inputs moving image content to be identified, for example, moving image content uploaded to a moving image sharing site and moving image content stored in various storages.

The feature amount extraction unit 12 extracts feature amounts of the reference moving image content and the identification moving image content. The feature amount extraction unit 12 extracts feature amounts from the reference moving image content and the identification moving image content. The moving image content (video signal) input from the moving image content input unit 11 is MPEG-2, H.264, or the like. When compressed by a compression method such as H.264, decompression is performed. Here, a following for each rectangular region R ⁱ which has been previously determined. The arrangement and the number of R ⁱ is arbitrary.

Figure 2A~ 2C are diagrams showing an example of setting the rectangular region R ^i. FIG. 2A shows a case where the entire screen is a single rectangular area. If it is not assumed that captions or logos are inserted, this rectangular area is used. FIG. 2B shows the setting of a rectangular area when editing such as subtitles is assumed at the bottom or right of the screen. Even if editing is performed on either one, the other rectangular area is not affected. FIG. 2C shows the setting of a rectangular area when the location where editing is performed is not determined in advance.

Hereinafter, the number of regions is N _R. FIG. 3 is a diagram illustrating a state in which one of the frames is selected and the feature amount x ⁱ (t) of the rectangular region R ⁱ is extracted. As shown in FIG. 3, from the region R ⁱ (1 ≦ i ≦ N _R ) of each frame of the moving image content, the feature amount x ⁱ (t) = (x ⁱ ₁ (t), x ⁱ ₂ (t),.・ ・ X ⁱ _NDim (t)) is extracted. Note that t is the frame number and NDim is the number of dimensions of the feature quantity. The feature quantity used here is basically arbitrary. For example, Dominant color, Scalable color, Color structure, Color layout, Edge histogram, Contour shape, Motion activity, etc. defined in MPEG-7 can be used.

Here, the case where the color layout defined in MPEG-7 is used as a feature amount will be described. FIG. 4 is a diagram illustrating a method of extracting feature amounts by color layout. First, the region R ⁱ of frame t, is reduced to 8 × 8. If necessary, it is converted to the YCbCr color system. After that, discrete cosine transform (DCT) is performed, and among the obtained coefficients, 6 coefficients for the Y component and 3 coefficients for the CbCr component are selected from the low frequency in the zigzag scan order, and used as the feature amount. In this case, NDim = 12. Further, by limiting the feature quantity used in the color layout to the Y component and not using the DC coefficient, it is possible to obtain a feature quantity that does not change even when gray scale conversion or luminance shift is performed. Furthermore, by performing normalization of the histogram before performing DCT, it is possible to make the feature amount robust to gamma correction and contrast adjustment.

FIG. 6 is a flowchart showing the operation of the feature quantity extraction unit 12. Here, t is a frame number of the moving image content, and i is a number of a preset area. As shown in FIG. 6, t is set to 1 (step S1), i is set to 1 (step S2), and feature quantities are extracted (step S3). Next, 1 is added to i (step S4), and it is determined whether i is larger than the number N _{R of} regions (step S5). If i is not larger than the number of areas N _R , the process proceeds to step S3. On the other hand, if i is larger than the number N _{R of} regions in step S5, 1 is added to t (step S6), and it is determined whether t is larger than N _F (step S7). Note that N _F is the number of frames of the input moving image content. In step S7, if t is not greater than N _F, the transition to step S2. On the other hand, if t is greater than N _F, and ends.

In the present invention, after extracting the feature value x ⁱ (t) from the region R ^{i of} each frame, instead of using the feature value of all the frames, a specific frame is selected as a key frame, and the feature value of the frame is selected. Use only. That is, the feature quantity extraction unit 12 sets key frames independently for a plurality of preset areas. Then, feature amounts are extracted from a plurality of regions set in advance from the key frame.

FIG. 5 is a diagram illustrating the concept of selecting key frame candidates. The selection of a key frame, starters k _j component of the feature (x ⁱ _kj (t)) is a candidate for t to an extreme value _{(1 ≦ k j ≦ N K} ). Where N _K is the number of coefficients to be used. In order to reduce the influence of noise, a smoothing filter (Gaussian or the like) in the time direction may be applied to each x ⁱ _kj (t) before obtaining the extreme value. As a specific method for obtaining extreme values, t where x ′ ⁱ _kj (t) = 0 is obtained by the zero crossing method. Furthermore, this t is a positive integer W _j predetermined for each k _j ,
x ' ⁱ _kj (t -W _j ) <x' ⁱ _kj (t -W _j + 1) <... <x ' ⁱ _kj (t) <... <x' ⁱ _kj (t + W _j- 1) <x ' ⁱ _kj (t + W _j )
Or
x ' ⁱ _kj (t -W _j )>x' ⁱ _kj (t -W _j + 1)> ・ ・ ・> x ' ⁱ _kj (t)> ・ ・ ・>x' ⁱ _kj (t + W _j- 1)> x ' ⁱ _kj (t + W _j )
When satisfying, the frame t keyframes in the region R ^i.

FIG. 7 is a flowchart showing an operation for selecting a key frame. Here, i is the number of a preset region, and j is the number of a coefficient used for extreme value detection. As shown in FIG. 7, first, i is set to 1 (step S11), j is set to 1 (step S12), and all t where x ' ⁱ _kj (t) has an extreme value are extracted (step S12). S13). Then, 1 is added to j (step S14), j is determined whether or not greater than N _K (step S15). j is If it is not greater than N _K, a transition to step S13. On the other hand, in step S15, j is if it is greater than N _K, 1 is added to i (step S16), i is determined whether or not greater than the number N _R of the region (step S17). If i is not greater than N _R in step S17, the process proceeds to step S12. On the other hand, if i becomes larger than N _R , the process ends.

  The feature amount extraction unit 12 selects a key frame and extracts a feature amount for the identified moving image content, similarly to the reference moving image content.

The feature amount storage unit 15 indexes the feature amounts extracted by the feature amount extraction unit 12 so that the search can be performed at high speed, and stores the indexes in the database 13. The key frame selected by the feature amount extraction unit 12 and the time code of the key frame are stored in the database 13. Specifically, the feature quantity x ⁱ (t), the time code t, and the video ID are stored in the database D ⁱ _j for each k _j-th component of the feature quantity that is a reference for selecting the region i and the key frame.

  At this time, an index using a tree structure or a hash can be used so that a search (nearest neighbor search) can be performed at high speed. Examples of such methods include R-tree, ANN (Approximate Nearest Neighbor), and LSH (Locality Sensitive Hashing).

The database search unit 14 searches the database for the feature amount extracted by the feature amount extraction unit 12, and searches for partially matching content. The database search unit 14 assumes a part of the identified moving image content from a part of the reference moving image content, and estimates a common section. After that, the similarity of common section candidates is calculated, and it is determined whether they are really the same section based on a threshold value. By the feature amount extraction unit 12, the key frames selected on the basis of the k _j component of the feature in the area _{^{i, t i j (1)}} , t i j (2), ···, t i j (n ⁱ _j ). n ⁱ _j is the number of key frames selected based on the k _j-th component of the feature amount in the region i. In the database search unit 14, for all i, j, feature quantities of key frames t ⁱ _j (1), t ⁱ _j (2),..., T ⁱ _j (n ⁱ _j ) and the database D ⁱ Compare the feature values stored in _j .

At this time, the search can be terminated in a fixed time, and in order to keep the accuracy at that time as much as possible, the importance is set in the key frame for searching the database. That is, a search is performed from a key frame having a high importance level, and when the time required for the search elapses, the search is terminated there. The importance I ⁱ _j (t) is defined by the following equation.
I ⁱ _j (t) = αP ⁱ _j (t) + (1 -α) ・ Q ^ij (t)
Here, P ⁱ _j (t) is a term that evaluates robustness against temporal shift of key frames (temporal stability of key frames), and Q ⁱ _j (t) is the peculiarity of key frame features. Is a term that evaluates. Each is defined below.
P ⁱ _j (t) = (min (x ⁱ _kj (t) -x ⁱ _kj (tW _j ), x ⁱ _kj (t) -x ⁱ _kj (t + W _j ))) ^β
× (Σ _{1 ≦ j ≦ NK} (x ' ⁱ _kj (t)) ² ) ^-γ
Q ⁱ _j (t) = d _M (x ⁱ (t))
d _M (x ⁱ (t)) is a Mahalanobis distance with respect to the distribution of the feature amount obtained in advance from the reference moving image content. β and γ are parameters determined by tuning.

  When the importance of the feature amount of each key frame is calculated, a database search is performed from the feature amount of the key frame having a high importance. In the present invention, assuming that the identified moving image content is a copy of a part of any reference moving image content, the start time at which the copy was performed is obtained. Specifically, it is as follows.

Let TC be the time code of the key frame t selected based on the k _j-th component of the feature quantity in the region i. A feature quantity that is the closest to the feature quantity x ⁱ (t) of the key frame t is searched from the database D ⁱ _j . Let TC 'be the time code of the feature value that was closest. If this correspondence is correct, the identified video content has been copied from the time TC'-TC. Therefore, TC'-TC is estimated for each search, and the TC'-TC that has a high possibility is determined as a copy candidate by voting. . In order to quantize the time estimation with a fixed parameter T 1 to improve robustness and speed, we actually vote for [(TC'-TC) / T]. This voting may be performed not only from the nearest neighbor but also from the K neighbor. In that case, the calculation time increases, but the accuracy improves. Finally, the time at which more votes than the threshold Th are made is output as a detection result.

  FIG. 8 is a flowchart showing the database search operation. Here, cnt is a counter of the number of feature values that have been voted. As shown in FIG. 8, first, importance is calculated (step S21), and a search is performed with the feature quantity having the highest importance (step S22). Next, the voting is performed (step S23), and 1 is added to cnt (step S24). And it is judged whether cnt became larger than threshold value Th2 (step S25). If cnt is not greater than the threshold value Th2 in step S25, the process proceeds to step S22. On the other hand, when cnt is larger than the threshold value Th2, all times when the number of votes is Th or more are output (step S26), and the process is terminated.

DESCRIPTION OF SYMBOLS 10 Movie content identification apparatus 11 Movie content input part 12 Feature quantity extraction part 13 Database 14 Database search part 15 Feature quantity storage part 16 Control bus

Claims (9)

A video content identification device that determines whether or not the identified video content that is the identification target includes all or part of the reference video content that is the identification criterion,
A video content input unit for inputting the video content;
A feature amount extraction unit that selects a key frame from a plurality of frames constituting the input video content and extracts a feature amount of the selected key frame;
When the identified moving image content is input from the moving image content input unit and the feature amount extraction unit extracts the feature amount of the key frame included in the identified moving image content, the extracted feature amount is stored in a database. A database search unit for searching for a feature amount of a key frame of the reference video content,
As a result of the search, it is determined whether or not the identified moving image content includes all or part of the reference moving image content based on the number of feature amounts of the reference moving image content corresponding to the feature amount of the identified moving image content. A moving image content identification apparatus characterized by the above. The feature amount extraction unit acquires a time code of a key frame of the selected identification video content when the identification video content is input from the video content input unit;
The database search unit uses the difference value between the time code in the feature amount of the identified moving image content and the time code in the feature amount of the reference moving image content corresponding to the feature amount of the identified moving image content as a feature of the identified moving image content. The identification moving image content is determined to include all or a part of the reference moving image content when the number of the difference values calculated for each amount is equal to or greater than a predetermined threshold value. Item 2. The moving image content identification device according to Item 1.   When the reference moving image content is input from the moving image content input unit and the feature amount of the key frame included in the reference moving image content is extracted by the feature amount extraction unit, the reference moving image content is included in the extracted reference moving image content The moving image content identification apparatus according to claim 1, further comprising a feature amount storage unit that stores the feature amount of the key frame.   4. The moving image content identification apparatus according to claim 1, wherein the feature amount extraction unit sets key frames independently for a plurality of preset regions.   5. The moving image content identification apparatus according to claim 4, wherein the feature amount extraction unit extracts feature amounts from a plurality of regions set in advance from the key frame.   The moving image content identification apparatus according to claim 4, wherein the feature amount storage unit creates an index for each of the extracted feature amounts.   The said database search part sets the importance about the key frame and the said feature-value of the identification moving image content extracted by the said feature-value extraction part, The Claim 1 characterized by the above-mentioned. Video content identification device.   8. The moving image content identification apparatus according to claim 7, wherein the importance is set based on temporal stability of a key frame and specificity of a feature amount. A video content identification method for determining whether or not the identified video content that is the identification target includes all or part of the reference video content that is the identification criterion,
Inputting the video content from a video content input unit;
In the feature amount extraction unit, selecting a key frame from a plurality of frames constituting the input video content, and extracting the feature amount of the selected key frame;
When the identified moving image content is input from the moving image content input unit, and the feature amount extraction unit extracts the feature amount of the key frame included in the identified moving image content, the database search unit extracts each feature amount And searching for a keyframe feature quantity of the reference video content stored in the database;
As a result of the search, it is determined whether or not the identified moving image content includes all or part of the reference moving image content based on the number of feature amounts of the reference moving image content corresponding to the feature amount of the identified moving image content. A moving image content identification method comprising at least a step.

Images