JP2010015588A - Apparatus for classifying dynamic image data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus for classifying dynamic image data, which classifies compressed or uncompressed dynamic image data as a class (a shot class) with a low cost and a high resolution by using a feature as the dynamic image data and an audio feature appended with the dynamic image data if it is required. <P>SOLUTION: The apparatus for classifying the dynamic image data includes: a dynamic image data partition element 1 for partitioning the dynamic image data on an time axis in a shot unit; a feature value extraction element 11 for extracting an image feature value such as a color layout descriptor defined by MPEG-7 to be acquired from an image within the shot unit; and a similar unit detection element 12 for detecting a similar shot unit appeared in a high frequency by using the image feature value, wherein the similar unit detection element 12 detects the similar shot appeared in the high frequency by drawing a plurality of coefficient histograms for the feature values and by sequentially scrutinizing a bin having the most multiple components in a frame for representing a shot head or a shot inside. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a moving image data classification device, and in particular, provides efficient search, classification or browsing of moving image data by classifying uncompressed or compressed moving image data into a predefined class. The present invention relates to an apparatus for classifying moving image data.

  As a conventional technique related to scene classification of moving image data, for example, a method is considered in which moving image data of a television broadcast is input and classified into relatively large units such as news, sports, and commercials. In addition, a method of dividing several related video sections into logical story units is also being studied. Here, in addition to the characteristics as moving image data, there has been proposed one that uses the characteristics of audio data associated with moving image data.

  For the detection of highlight scenes as summary information, a technique has been proposed in which highlight scenes such as sports videos are extracted using compressed audio characteristics in a compressed area of compressed moving image data.

  Furthermore, in Japanese Patent Application No. 2002-285667 by the present applicant, scenes with a relatively low abstract level such as classification into dynamic / static scenes, slow scenes, camera operations such as panning and zooming, etc. Therefore, a method for classifying a scene with a relatively high level of abstraction such as a highlight scene of a sports video has been proposed.

Japanese Patent Application No. 2002-285667

  Many of the prior arts mainly analyze moving image data and accompanying audio data in the uncompressed data area, and it is necessary to perform decoding processing once on the compressed moving image data, resulting in a high processing cost. Was a problem. As for the unit of classification, classification in a relatively large unit such as each program or logical story unit is mainstream, and therefore, a more detailed classification technique is disclosed in, for example, Japanese Patent Application No. 2002-285667. Technology is required. Classification in detailed units is important and effective for browsing a specific scene in moving image data, classification in a moving image database, and the like. In the technique shown in Japanese Patent Application No. 2002-285667, scene classification at a lower level, such as dynamic / static scenes and camera operation extraction, is the mainstream. There was a problem that filtering could not be supported. For example, there is a problem that filtering such as excluding violent scenes from movie content cannot be performed.

  The present invention has been made in view of the above-described prior art, and an object of the present invention is to convert uncompressed or compressed moving image data as a moving image, and, as necessary, audio characteristics associated with the moving image. It is an object to provide a moving image data classification device that classifies various classes (shot classes) with low cost and high accuracy.

  To achieve the above object, the present invention provides an apparatus for classifying uncompressed or compressed moving image data, moving image data dividing means for dividing moving image data into shot units on a time axis, and the shot units. A feature value extracting means for extracting an image feature value such as a color arrangement descriptor defined in MPEG-7 obtained from an image in the image, and a similarity for detecting a similar shot unit having the highest appearance frequency using the image feature value The similar unit detection means comprises a histogram of a plurality of coefficients of the feature value in a shot head or a representative frame in a shot, and sequentially narrows down bins having the largest number of elements. It is characterized in that the most similar shots are detected.

  According to this feature, the most frequent similar shots can be efficiently detected in uncompressed or compressed moving image data. Further, by applying the present invention, a highlight scene can be detected with high accuracy from a television sports video, and an announcer shot can be detected with high accuracy from a television news video.

  As is apparent from the above description, according to the first to seventh aspects of the present invention, in the uncompressed or compressed moving image data, the shots are classified into various types to obtain desired ones from the moving image data. It is possible to search / browse scenes and effectively classify a large number of moving image data.

  In particular, according to the first and second aspects of the present invention, the most frequent similar shot can be efficiently detected in uncompressed or compressed moving image data.

  According to the inventions of claims 3 to 5, highlight scenes can be detected with high accuracy from television sports images, and according to the inventions of claims 6 and 7, an announcer shot can be taken from television news images. It becomes possible to detect with high accuracy.

It is a block diagram of the moving image data classification device of one embodiment of the present invention. It is a flowchart which shows operation | movement of the action class discrimination | determination part of FIG. It is a flowchart which shows operation | movement of the dramatic class discrimination | determination part of FIG. It is a flowchart which shows operation | movement of the conversation class discrimination | determination part of FIG. It is a block diagram of the moving image data classification device of 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the mode shot detection part of FIG. It is explanatory drawing of the most frequent shot detection process using a color arrangement | positioning descriptor. It is a flowchart which shows other operation | movement of a mode shot detection part. It is a block diagram of the moving image data classification device of 3rd Embodiment of this invention.

  Hereinafter, the present invention will be described in detail with reference to the drawings. First, an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram illustrating a configuration of a main part of a moving image data classification device. Here, the input moving image data will be described using an example in which the shot dividing unit 1 divides the input moving image data into shot units. However, the unit of division is related to the camera operation at the time of capturing a moving image or the content of the captured moving image. Any unit that is consistent is optional. For example, a moving image from when the start button of the camera is pressed until the stop button is pressed (for example, a zoom operation or the like may be performed during this period) can be considered as one division unit.

  First, when uncompressed or compressed moving image data and accompanying audio data or audio data multiplexed therewith are input, the shot dividing unit 1 divides the moving image data into shots. The shot division unit 1 holds the shot length Ls of each shot of the input moving image data. The moving image data of each shot divided by the shot division unit is passed to the motion information analysis unit 2.

  In parallel with this, audio data obtained by demultiplexing audio data accompanying the moving image data or audio data multiplexed with the moving image data is passed to the audio analysis unit 3.

  The motion information analysis unit 2 calculates a motion intensity value Is in the shot using the motion vector value of the predictive encoded image existing in the shot for the compressed video data. For the motion intensity Is, “Motion Intensity” that is an element of “Motion Activity Descriptor” defined in MPEG-7 can be used. The “motion intensity” element of the “motion activity descriptor” is expressed by an integer from 1 to 5 (1 is the lowest, 5 is the highest).

  On the other hand, for uncompressed video data, the motion from the previous screen is estimated using a block matching method, etc., and the obtained value is expressed as a motion vector. calculate. At this time, as the motion intensity Is as a shot, a value obtained by averaging the values of the motion intensity Ip in the target predictive encoded image within the shot, or a maximum value or an intermediate value thereof can be used. In addition, as a prediction encoded image and a motion vector to be subjected to motion information analysis, a forward prediction encoded image, a bidirectional predictive encoded image, and a forward motion vector and a backward motion vector in a bidirectional predictive encoded image Any combination of these can be used.

  The audio analysis unit 3 basically includes an audio power calculation unit 31, but can also include an audio type analysis unit 32. The audio power calculation unit 31 calculates the power Ps of the audio signal of the audio data in the input shot or the audio power Psb for each band. When calculating the audio power Psb for each band, an arbitrary bandwidth can be selected, and weighting can be performed for each band. The sum of these can be used as the audio power Ps in the shot. Therefore, Ps is expressed as follows.

  Here, lsb is a band number of the lowest band for calculating the audio power, hsb is a band number of the highest band, and w [i] is a weight for the audio power Psb [i] in the band i.

  Further, when the audio type analysis unit 32 can be used, for example, the audio data in the input shot is classified into audio types such as “silence”, “voice”, “music”, and “cheer”. As the processing of the audio type analysis unit 32, the method described in Japanese Patent Laid-Open No. 10-247093 can be used. When this audio type is determined per unit time, the most frequent class in the shot is regarded as the representative audio type Cs of the shot.

Here, a shot class handled by the classification device according to the present invention is defined.
“Action” class… In movies, shots and explosions, such as high audio volume and movement, short shot length,
"Dramatic" class ... Such as a movie, often precedes an "action" class, and some important events happen or are going to happen,
"Conversation" class ... A shot of two or more characters in a movie, etc.
“Highlight” class… In TV sports video, shots including important events such as scoring scenes,
"Announcer" class ... In the TV news video, the shot of the announcer reading the news,

  In the action class discriminating unit 4 and the dramatic class discriminating unit 5, the shot length Ls inputted from the shot dividing unit 1, the in-shot motion intensity Is obtained from the motion information analyzing unit, and the in-shot audio power Ps are inputted. The conversation class discriminating unit 6 further receives the representative audio type Cs in the shot.

  Next, functions of the action class discriminating unit 4, the dramatic class discriminating unit 5, and the conversation class discriminating unit 6 shown in FIG. 1 will be described in detail.

  The determination process in the action class determination unit 4 is performed as shown in FIG. In step S1, the shot length Ls is smaller than a certain threshold THL1 (for example, 2 seconds) (Ls <THL1), and is larger than a certain threshold THI1 (for example, 2.3) having an intra-shot motion intensity Is (Is> THI1). If the in-shot audio power Ps is larger than a certain threshold value THP1 (Ps> THP1), it is determined that the corresponding shot is the “action” class. In step S2, “action” is given as a shot class.

  The determination process in the dramatic class determination unit 5 is performed as shown in FIG. In step S3, the shot length Ls is larger than the threshold THL1 (Ls> THL1) and smaller than another threshold THL2 (for example, 5 seconds) (Ls <THL2), and the in-shot motion intensity Is is larger than the threshold THI1. Corresponding if it is small (Is <THI1) and larger than another threshold THI2 (for example 1.2) (Is> THI2) and the in-shot audio power Ps is larger than a certain threshold THP2 (Ps> THP2) It is determined that the shot to be performed is a “dramatic” class. In step S4, “dramatic” is assigned as the shot class.

  The determination process in the conversation class determination unit 6 is performed as shown in FIG. In step S5, the shot length Ls is greater than a certain threshold value THL3 (THL3> THL2, for example, 6 seconds) (Ls> THL3), and the threshold value THI3 (preferably THI3 ≧ THI2, for example, 1.5 for example) having an in-shot motion intensity Is. In some cases, it may be smaller than THI3 <THI2) (Is <THI3), in-shot audio power Ps is smaller than a certain threshold THP3 (THP3 <THP2) (Ps <THP3), and shot When the inner representative audio type Cs is “voice”, it is determined that the corresponding shot is the “conversation” class. In step S6, “conversation” is given as a shot class.

  Shots that are determined not to belong to any shot class in the action class discriminating unit 4, the dramatic class discriminating unit 5, and the conversation class discriminating unit 6 are determined to be “general-purpose” classes, Is given.

In the above embodiment, the action class determination unit 4, dramatic class determination unit 5, and the conversation class determination unit 6, the divided section length L _S, motion information I _S, and each class using the audio data P _S Although it discriminate | determined, this invention is not limited to this, You may make it discriminate | determine using at least one of the above.

  Next, a second embodiment of the present invention will be described with reference to FIG. In FIG. 5, the same or equivalent parts as in FIG. In this embodiment, the moving image data shot-divided by the shot dividing unit 1 is sent to the feature value extracting unit 11, and the image feature value of the shot is extracted. Next, the mode shot detection unit 12 detects the mode shot based on the image feature value.

  As the image feature value, for example, an image determined as a shot division point by a shot division unit, that is, image data itself of a shot start screen is held, image data of a reduced image of the image, or obtained from the image, The “Color Layout Descriptor” defined in MPEG-7 can be used. Further, not only the shot head screen but also a shot center screen, a screen representing a shot (key frame), or the like can be used for the target image.

  Here, an example using a color arrangement descriptor will be described. The color arrangement descriptor is obtained by applying 8 × 8 DCT to the luminance component and the color difference component of an image (8 × 8 pixels) obtained by reducing the original image, and has the DCT coefficient of each component as a value.

The operation of the mode shot detector 12 (mode shot detection process 1) will be described with reference to the flowchart of FIG. Here, the most frequent shot means a similar shot Sf that appears most frequently in the data. First, all the input moving image data 21 as shown in FIG. 7 is read once, and in step S11, image feature values, for example, color arrangement, are displayed from the top screen of each shot (1, 2, 3,..., N). Descriptors (a1, a2, a3,..., An) are extracted. In step S12, a certain number m is set to 1, and in step S13, the mth coefficient of the color arrangement descriptor, for example, the histogram of the mth coefficient of the luminance component of the image obtained by reducing the original image and 8 × 8 DCT. Create In the example of FIG. 7, first, a histogram is created with the first coefficients (m = 1) Y ₁ (1), Y ₁ (2), Y ₁ (3),..., Y ₁ (n).

  In step S14, a certain number n = 2, and in step S15, is the difference between the number of elements of the first mode bin and the number of elements of the second mode bin (n = 2) smaller than a predetermined criterion? A determination is made whether or not. For example, it is determined whether or not (number of elements of the first mode bin) × 0.85 <(number of elements of the second mode bin) is satisfied. Here, the number of elements of the first mode bin means a similar shot that appears most frequently in the data. Therefore, in step S15, it is determined whether or not the difference between the shot with the highest appearance frequency and the shot with the next highest appearance frequency is small.

  If this determination is affirmative, the process proceeds to step S16, where n is incremented by 1, and in step S15, the difference between the number of elements of the first mode bin and the number of elements of the (n + 1) mode bin is based on a predetermined criterion. A determination is made as to whether it is small. If this determination is affirmative, the (n + 1) most frequent bin also becomes a shot with a high appearance frequency.

  When the above process is performed and the determination in step S15 is negative, the process proceeds to step S17, and the first to (n-1) mode bins are adopted for the mode shot. Thus, the narrowing down of the most frequent shots by the first coefficient is completed. Next, in step S18, m is incremented by one. In step S19, it is determined whether or not the first to (n-1) most frequent bins have converged. That is, it is determined whether similar shots that appear most frequently in the data are sufficiently narrowed down.

If not converged, the process returns to step S13 to create a histogram with the (m + 1) th coefficient of the shots of the first to (n-1) most frequent bins, and then perform the same processing as described above. And perform processing to narrow down similar shots. This process, narrowing of the most frequent shots is performed by the second coefficient Y _2. Hereinafter, the same process is performed a third coefficient Y ₃ such options in a similar shots is determined to have sufficiently narrowed down (the determination in step S19 is affirmative), the modal shot detection process ends.

  The ordering of the first, second, third,... Coefficients Y1, Y2, Y3,... Is not limited to the ordering shown in FIG. In addition, only a luminance component, only a color difference component, or both can be used as components to be used, and coefficients that can be used in each component are also arbitrary. Further, the process of step S15 can prevent a certain shot from being leaked from similar shot detection due to a slight difference in the values of the color arrangement descriptors. In this way, similar shots are narrowed down using the color arrangement descriptor, and a shot belonging to the bin having the largest number of elements is finally determined as the most frequent shot Sf.

  Next, processing (processing 2) for further improving the accuracy of similar shot detection will be described with reference to the flowchart of FIG. In step S20 of FIG. 8, a representative value (or reference value) of the color arrangement descriptor value of the shot determined as the most frequent shot is obtained, and in step S21, the color arrangement descriptor in all shots is obtained using this value. The distance D is calculated. As the representative value, an average value or an intermediate value of each component / coefficient can be used. As a result of calculating the distance D, a shot having a sufficiently small distance below the threshold THD can be detected as the most frequent shot.

  For the calculation of the distance D, the following formula recommended in the MPEG-7 verification model can be used.

  Here, Yr [i], Cbr [i], and Crr [i] are the representative values of the i-th coefficient of the luminance Y component, the color difference Cb component, and the color difference Cr component, respectively Y [i], Cb [i], Cr [ i] is the i-th coefficient from the low frequency side of each component, and NY, NCb, and NCr are the numbers of coefficients of each component used for calculating the distance D, respectively.

  The highlight scene discriminating unit 13 shown in FIG. 5 detects a highlight scene such as a hit or a home run using, for example, a television sports video such as a baseball game. Here, the “scene” indicates a section composed of one or more “shots” that are semantically continuous.

  The highlight scene discriminating unit 13 obtains the shot number Nsf and the time Tsf between the most frequent shots Sf with respect to the most frequent shot Sf in the television sports video obtained by the processing of FIGS. 6 and 8, for example. For example, in the case of a baseball broadcast, a shot in which a pitcher throws a ball against a batter (hereinafter referred to as a throwing shot) is considered to be the most frequent shot in a baseball broadcast video. When the result of the pitch cannot be regarded as a highlight scene such as a strike, a ball, or a foul, the number of shots Nsf or the time Tsf until the next pitch shot is considered to be small or short, respectively. On the other hand, when the result of the pitch is recognized as a highlight scene such as a hit or a home run, it is considered that the shot number Nsf or the time Tsf until the next pitch shot takes a certain value or more.

  Therefore, when either or both of these are the threshold THNsf (for example, 30 shots) and THTsf (for example, 60 seconds) or more (Nsf ≧ THNsf, Tsf ≧ THTsf), the interval between these adjacent mode shots Sf It is determined that there is a highlight scene. However, in the case of baseball broadcasts, CMs may be inserted mainly when changing offense and defense, so by using the shot number Nsf between pitch shots and time Tsf together, it is possible to extract highlight scenes effectively. it can. Furthermore, it is possible to improve the accuracy of determination that the section is a highlight scene by using the fact that “cheer” is dominant with respect to the in-shot representative audio type Cs in the shot included in the section. .

  It is also possible to determine all shots existing in the corresponding section as highlight scenes, but any number before and after centering on shots where the audio type Cs is “cheer” and the audio power Ps is maximum. Can be determined as a highlight scene. As a result, for example, when the pitch shot is not detected normally or the pitch result is out, the shot number Nsf or time Tsf until the next pitch shot is not a highlight scene but the time Tsf has increased. False detection can be suppressed. A shot class “highlight” is assigned to each shot group determined as a highlight scene.

  Further, the announcer class discriminating unit 14 shown in FIG. 5 detects an announcer class from, for example, a television news video by using the most frequent shot obtained by the most frequent shot detecting unit 12. When the most frequent shot is applied to a television news video, a normal news video is followed by an announcer shot followed by a video such as a report from the site, a material video, a conference, and an explanation, and this is repeated for each news item. Since one or more announcer shots often appear for one news item, it is considered that the announcer shot is the most frequent shot in the entire news program.

However, there is a possibility that the screen used for explanation etc. is misrecognized as the most frequent shot because the background color is the same. In order to prevent this, the coefficient Y _n of the high frequency component is analyzed (for example, n> 6) for the luminance component in the color arrangement descriptor. In particular, when the explanation screen is a reduced image, the texture becomes inconspicuous and a relatively flat screen is expected. Therefore, the value of the high frequency component Y _n becomes small. On the other hand, since an announcer is reflected in the announcer shot, texture is present. Therefore, it is considered that the value does not decrease even at high frequency components. Using this property, only the announcer shot can be extracted as the most frequent shot. The shot class “announcer” is assigned to the shot determined as the announcer shot.

  By collecting and reproducing the above-mentioned “highlight” class shots and “announcer” class shots, it is possible to configure highlights of television sports videos, digests of television news videos, and the like.

  Next, FIG. 9 shows a third embodiment of the present invention. Here, the input moving image data is shot divided by the shot dividing unit 1 and subjected to the processes shown in FIGS. The process of FIG. 1 performs shot genre discrimination processing 41, that is, action class discrimination, dramatic class discrimination, and conversation class discrimination processing. On the other hand, according to the processing of FIG. 5, summary shot discrimination processing 42, that is, highlight scene discrimination and announcer class discrimination processing is performed.

  The shot class determined by the shot genre discriminating unit 41 is attached to each shot as a genre of a shot defined by, for example, a “Classification Scheme” defined in MPEG-7 in the shot genre description unit 43. Can be described as:

  Further, the summary shot description unit 44 can describe time information and the like of shots determined as highlights of sports videos and digests of news videos. As the format of the summary shot description, for example, a “hierarchical summary description scheme” defined in MPEG-7 can be used. The described information is output as an MPEG-7 description file.

  DESCRIPTION OF SYMBOLS 1 ... Shot division part, 2 ... Motion information analysis part, 3 ... Audio analysis part, 4 ... Action class discrimination | determination part, 5 ... Dramatic class discrimination | determination part, 6 ... Conversation class discrimination | determination 11, feature value extraction unit 12, mode shot detection unit 13, highlight scene determination unit 14, announcer class determination unit 31, audio power calculation unit 32 ... Audio type analysis unit, 41 ... Shot genre discrimination unit, 42 ... Summary shot discrimination unit, 43 ... Shot genre description unit, 44 ... Summary shot description unit.

Claims (7)

In an apparatus for classifying uncompressed or compressed video data,
Moving image data dividing means for dividing moving image data into shot units on the time axis;
Feature value extracting means for extracting image feature values such as color arrangement descriptors defined in MPEG-7 obtained from images in the shot unit;
Similar unit detection means for detecting a similar shot unit having the highest appearance frequency using the image feature value,
The similar unit detection means detects the most frequent similar shots by making a histogram of a plurality of coefficients of the feature value in the shot head or the representative frame in the shot and sequentially narrowing down bins having the largest number of elements. An apparatus for classifying moving image data. The moving image data classification device according to claim 1,
An apparatus for classifying moving image data, wherein coefficients of each component obtained by performing discrete cosine transform on at least one of luminance and color difference components of a reduced image are used as the plurality of coefficients of the image feature value. The moving image data classification device according to claim 1 or 2, wherein the moving image data is known in advance as a television sports video.
Equipped with highlight scene discrimination means,
The highlight scene discriminating unit highlights the section when at least one of the number of shots and the time of the section between the most frequent similar shots extracted from the similar unit detecting section exceeds a predetermined threshold. An apparatus for classifying moving image data, characterized in that a scene is determined to exist. In the moving image data classification device according to claim 3,
The highlight scene discriminating unit is configured to display a commercial in a section of a television sports video when the number of shots in a section between the most frequent similar shots extracted from the similar unit detection section is equal to or greater than a predetermined threshold. An apparatus for classifying moving image data, characterized in that it determines that a scene exists and excludes it from a candidate for a highlight scene. The moving image data classification device according to claim 3 or 4,
The highlight scene discriminating means further uses a shot having the maximum audio power whose audio type is “cheers” for the determination of the highlight scene, and the shot having the maximum audio power or the preceding and following shots including the shot. Is a highlight scene. The moving image data classification apparatus according to claim 1 or 2, wherein the moving image data is known in advance as television news video.
An announcer class discriminating means is provided,
An apparatus for classifying moving image data, wherein the announcer class discriminating means judges the most frequent similar shot extracted from the similar unit detecting means as an announcer shot of a television news video. The moving image data classification device according to claim 6,
The announcer class determining means determines that a texture as an image exists when a high frequency component of the color arrangement descriptor is higher than a predetermined frequency, and an announcer shot when the texture exists in the most similar shot A moving image data classification device characterized by the above-described determination.

Images