JP2006277232A - Classification device for dynamic image data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a classification device for non-compressed or compressed dynamic image data, capable of determining a further detailed shot type. <P>SOLUTION: This device comprises a shot classification part 5 including an explosion shot determination part 51 for determining a shot including explosion in dynamic image data; a chase shot determination part 52 for determining a shot including chase, and a close-up shot determination part 53 for determining a shot including a close-up of a face. The explosion shot determination part 51, the chase shot determination part 52, and the close-up shot determination part 53 perform the determination of explosion, chase and close-up using at least one of analysis information from a color information analysis part 2, a motion information analysis part 3 and an audio analysis part 4. An action shot determination part and a conversation shot determination part may be provided in the front stage of the explosion shot determination part 51 and the chase shot determination part 52 and in the front stage of the close-up shot determination part 53, respectively. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an apparatus for classifying moving image data by analyzing uncompressed or compressed moving image data and classifying them into predefined shot types, and in particular moving image data by classifying them into meaningful shot types. The present invention relates to an apparatus for classifying moving image data capable of efficiently searching, browsing, and estimating the contents and context of moving images.

  As a conventional technique related to shot classification of moving image data, in Japanese Patent Application Laid-Open No. 2004-260734 by the present inventors, shots are classified into “action” class, “dramatic” class, “conversation” class, “general purpose” class, and the like. An apparatus for classifying a scene type having no semantic concept such as a slow scene, a zoom scene, and a pan scene, and a scene type having a semantic concept such as a commercial scene and a highlight scene is disclosed.

In addition, Japanese Patent Application No. 2004-62866 by the present inventors discloses a device that introduces a “landscape” class as a shot type and determines a set of shots of the same class as a scene.
JP 2004-260734 A

  In the above-described existing technology, moving image data can be classified into predefined shot types, but these are relatively rough classifications, and more detailed shot types are required in terms of shot labeling. . For example, the “action” class is relatively broad, and actually there are subordinate classes such as “explosion” and “car chase”. Further, when trying to estimate the content and context of a moving image using such a shot type, it is possible to appropriately grasp the content and context of the moving image by defining a detailed shot type.

  The present invention has been made in view of the above-described prior art, and an object of the present invention is to provide an uncompressed or compressed moving image data classification device capable of determining a more detailed shot type. .

  The present invention relates to a classification apparatus for uncompressed or compressed moving image data, a shot dividing unit that performs shot division of the moving image data, a color information analysis unit that analyzes color information in the moving image data, and motion information. Using at least one of analysis information from a motion information analysis unit to analyze, an audio analysis unit to analyze audio data accompanying the moving image data, the color information analysis unit, the motion information analysis unit, and the audio analysis unit And at least one of an explosion shot determination unit for determining shots including explosion in moving image data, a chase shot determination unit for determining shots including chase, and a close-up shot determination unit including face close-up. There are features.

  Further, an action shot determination unit for determining a shot including an action from the motion information of the moving image data and the audio data is further provided, and a shot including a chase is further determined from the action, and hierarchical shot type classification is performed. There are other features in this way.

  Furthermore, a conversation shot determination unit that determines a shot including conversation from the moving image data is determined, a shot including a close-up is determined from a conversation shot obtained from the conversation shot determination unit, and a conversation shot including a close-up is determined. There is another feature in the point of determination.

  According to the present invention, uncompressed or compressed moving image data can be classified into semantic shot types such as explosion shots, chase shots, and close-up shots.

  Further, it is possible to classify hierarchical shot types by determining shots further including chase from actions or determining shots including close-up from conversation shots obtained from the conversation shot determination unit.

  Further, by using the present invention, it becomes possible to efficiently search and browse moving image data and estimate the content and context of moving images.

  Hereinafter, the present invention will be described in detail with reference to the drawings. First, an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a configuration diagram of a moving image data dividing device.

  First, uncompressed or compressed moving image data is divided into shot units by a shot division unit, and then color information is further attached from each shot to color information analysis unit 2 and motion information analysis unit 3 is further accompanied with motion information. Audio information is extracted from the audio data in the audio analysis unit 4. About these processes, the method etc. which are indicated by the above-mentioned patent documents 1 can be used.

  As a target for extracting color information, an arbitrary frame such as the first frame of the shot, the key frame representing the shot, or all the frames in the shot can be selected. As audio information, audio energy, audio type, and the like can be used, but here, it is assumed that weighted average of subband energy is used for audio data divided into subbands and encoded. To do.

  At this time, if the input moving image data is data that is compression-encoded using discrete cosine transform such as MPEG format, color information is analyzed using an image (DC image) composed only of DC components. Can do. As the color information, pixel value data in the image, an arbitrary color histogram in the image, or color arrangement information in the image can be used.

  In addition, when compression is performed using predictive coding, the target of motion information extraction is a forward predictive encoded image, a bidirectional predictive encoded image, and a forward motion vector in the bidirectional predictive encoded image. Any combination of reverse motion vectors can be used. Arbitrary values such as an average value, an intermediate value, and a maximum value in a shot can be used as the motion information for the motion information extracted for these.

  A shot classification unit 5 is arranged following the color information analysis unit 3, the motion information analysis unit 2, and the audio information analysis unit 4. The shot classification unit 5 includes at least an explosion shot determination unit 51, a chase shot determination unit 52, and a close-up shot determination unit 53. The output of the shot classification unit 5 is registered in the shot type registration unit 6.

  The explosion shot determination unit 51 receives at least color information and audio information, the chase shot determination unit 52 inputs movement information, and the close-up shot determination unit 53 receives at least color information (or audio information as necessary). The determination process is performed.

First, processing in the explosion shot determination unit 51 will be described with reference to FIG. The explosion shot determination unit 51 evaluates at least one of the following (1) and (2), and also evaluates the characteristics of audio information accompanying explosion sound.
(1) Number of pixels with pixel values corresponding to flames when color information is pixel value data
(2) Frequency of color range corresponding to flame when color information is color histogram

  Specifically, the frame FN is input to the explosion shot determination unit 51 in step S1. In step S2, the number of pixels P_flame having a color corresponding to a predetermined flame is counted for the color information of an arbitrary frame FN in the shot input to the explosion shot determination unit 51, and this value is determined in advance. If it is larger than the threshold value THP_flame, or if the frequency H_flame of the color range corresponding to the predetermined flame is larger than the predetermined threshold value THH_flame, or both are larger, the process proceeds to step S3, and the frame FN includes an explosion. It is determined that In step S4, it is determined that the shot including this frame includes an explosion.

  Here, when only the image feature is used, for example, there is a possibility that an erroneous determination due to the color of the sun or the like may occur. The accuracy can be improved by adding the condition that it is larger than the above. Furthermore, in order to cope with misjudgment due to a sudden change in brightness, etc., when frames with colors that correspond to flames appear continuously and the number of pixels fluctuates over time, these It can also be determined that the shot including the frame group includes an explosion.

  Next, processing in the chase shot determination unit 52 will be described with reference to FIG. In a chase shot such as a car chase, an object such as a car or a fighter plane moves at high speed, and a camera that shoots the object often moves regularly. Therefore, the movements included in the chase shot are relatively large and their directions are uniform. Therefore, the chase shot determination unit 52 evaluates the magnitude and direction of movement included in the shot.

  Specifically, in step S <b> 11, the shot SN is input to the chase shot determination unit 52. In step S12, with respect to the motion information in the shot SN input to the chase shot determination unit 52, the threshold THM_mag having the in-shot average MV_mag of the magnitude of the motion calculated from the motion vector and the ratio MV_dir in the most frequent direction of motion, respectively. And whether it is greater than THM_dir. When it is determined that the size is larger, the process proceeds to step S13, and it is determined that the chase is included in the shot SN. Any number of movement directions can be selected, for example, eight directions can be set.

  Since normal chase shots often appear continuously, a chase scene determination unit (not shown) can be provided after the chase shot determination unit 52, and the chase shots are determined continuously. Then, the chase scene is determined. At this time, since another shot type may be inserted between adjacent chase shots, it is not necessary to be completely adjacent, and may be separated by a predetermined number of shot intervals.

Further, the processing in the close-up shot determination unit 53 will be described with reference to FIG. The close-up shot determination unit 53 evaluates at least one of the following (3) and (4).
(3) Number of pixels with skin color pixel values corresponding to the face when color information is pixel value data
(4) Frequency of skin color range corresponding to face when color information is color histogram

  Specifically, the frame FN is input to the close-up shot determination unit 53 in step S21. In step S22, for the color information in an arbitrary frame FN in the shot input to the close-up shot determination unit 53, the number of pixels P_skin having a color corresponding to a predetermined skin color is counted, and this value is determined in advance. If the threshold value THP_skin is greater than the predetermined threshold value TTH_skin, or if the frequency H_skin of the color range corresponding to the predefined skin color is greater than the predetermined threshold value THH_skin, or if both are larger, the process proceeds to step S23, Is determined to be included. In step S24, it is determined that a face close-up is included in a shot including this frame.

  Here, when only the number of skin color pixels is used, an erroneous determination may occur due to other parts of the body. Therefore, the shape of the skin color pixel region may be further evaluated for the above determination, or the face template Thus, it can be determined with high accuracy whether or not a face is included in the image.

  FIG. 5 shows another processing example of the close-up shot determination unit 53. As in this processing flow, using the audio type obtained from the audio analysis unit, a ratio ACL within a shot of the number of seconds or frames in which the audio type is “voice” in the shot SN is a predetermined threshold THA_voice It is also possible to add a condition that it is larger than the above. As the audio type ACL acquisition means, means disclosed in Japanese Patent Laid-Open No. 10-247093 can be used.

  Specifically, in step S31, a shot SN is input, and a flag FF indicating that a face close-up is included is set to 0. In step S32, the frame FN is input. In steps S33 and S34, the same processing as in steps S22 and S23 is performed. In step S35, it is determined whether or not the frame FN is the last frame of the shot SN. In step S34, the flag FF is set to 1. If the determination in step S35 is negative, the process proceeds to step S36 and FN is incremented by one.

  When the processes of steps S32 to S36 are repeatedly performed and the determination in step S35 is affirmative, the process proceeds to step S37 to determine whether or not the flag FF = 1 and ACL> THA_voice. If this determination is affirmative, the process proceeds to step S38, and it is determined that the type of the shot SN is a close-up including a face.

  FIG. 6 is a diagram showing a second embodiment of the present invention. In this embodiment, an action shot determination unit 54 is provided before the explosion shot determination unit 51 and the chase shot determination unit 52, and a conversation shot determination unit 55 is further provided before the close-up shot determination unit 53. Other reference numerals are the same as or equivalent to those in FIG.

  Explosion shots and chase shots can be regarded as subordinate classes of the “action” class, and close-up shots can be regarded as subordinate classes of the “conversation” class. Therefore, it is possible to classify hierarchical shot types, and it is possible to reduce the processing amount required for determination by narrowing down the shots to be determined to those belonging to the main class in advance.

  For the processing of the action shot determination unit 54 and the conversation shot determination unit 55, the method or means disclosed in Patent Document 1 can be applied.

It is a block diagram which shows the structure of the moving image data classification device of one Embodiment of this invention. It is a flowchart which shows the process of an explosion shot determination part. It is a flowchart which shows the process of a chase shot determination part. It is a flowchart which shows the process of a close-up shot determination part. It is a flowchart which shows the other process of a close-up shot determination part. It is a block diagram which shows the structure of the moving image data classification device of other embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Shot division part, 2 ... Color information analysis part, 3 ... Motion information analysis part, 4 ... Audio analysis part, 5 ... Shot classification part, 6 ... Shot classification registration part , 51 ... Explosion shot determination unit, 52 ... Chase shot determination unit, 53 ... Close-up shot determination unit, 54 ... Action shot determination unit, 55 ... Conversation shot determination unit.

Claims (22)

In a classification apparatus for uncompressed or compressed moving image data, a shot dividing unit that performs shot division of the moving image data;
A color information analysis unit that analyzes color information in the moving image data, and a motion information analysis unit that analyzes motion information;
An audio analysis unit for analyzing audio data accompanying the moving image data;
An explosion shot determination unit that determines a shot including an explosion in moving image data using at least one of the analysis information from the color information analysis unit, the motion information analysis unit, and the audio analysis unit, and a chase that determines a shot including a chase An apparatus for classifying moving image data, comprising at least one of a shot determination unit and a close-up shot determination unit including a close-up of a face. The moving image data classification device according to claim 1,
The moving image data is data that has been compression-encoded using discrete cosine transform,
The apparatus for classifying moving image data, wherein the color information is analyzed on an image composed only of a direct current component relating to a luminance component and a color difference component of the moving image data. The apparatus for classifying moving image data according to claim 1 or 2,
The apparatus for classifying moving image data, wherein the color information is at least one of pixel data in an image, a color histogram in the image, and color arrangement information in the image. The moving image data classification device according to claim 1,
The apparatus for classifying moving image data, wherein the explosion shot determination unit determines based on at least one of color information in an image and a level of audio data. The apparatus for classifying moving image data according to claim 4.
The explosion shot determination unit uses pixel data in an image as color information in the image, counts the number of pixels having a color corresponding to a flame, and determines explosion, moving image data classification apparatus. The apparatus for classifying moving image data according to claim 4.
The explosion shot determination unit uses a color histogram in an image as color information in the image, measures frequency of a color range corresponding to a flame, and determines explosion, moving image data classification apparatus. The moving image data classification device according to claim 1,
The apparatus for classifying moving image data, wherein the audio analysis unit performs determination using a weighted average of subband energy in the shot. The moving image data classification device according to any one of claims 1 and 4 to 7,
The explosive shot determination unit, in an arbitrary frame in the shot, if the number of pixels corresponding to the flame or the frequency of the color range and the weighted average of the subband energy are larger than a predetermined value, respectively, An apparatus for classifying moving image data, characterized by regarding a frame including an explosion and determining a shot including the frame as a shot including an explosion. The apparatus for classifying moving image data according to claim 8.
The explosion shot determination unit further includes an explosion in the frame group when frames considered to contain an explosion appear continuously and the frequency of the number of pixels or the color range fluctuates with time. An apparatus for classifying moving image data, characterized by regarding a group of frames and determining a shot including the group of frames as a shot including an explosion. The moving image data classification device according to claim 1,
An action shot determination unit for determining a shot including an action from the motion information of the moving image data and the audio data; determining a shot including the explosion based on the action; and performing hierarchical shot type classification. An apparatus for classifying moving image data as a feature. The moving image data classification device according to claim 1,
The moving image data classification device, wherein the chase shot determining unit includes at least means for analyzing motion information in a shot. The apparatus for classifying moving image data according to claim 11,
The apparatus for classifying moving image data, wherein the movement information is at least one of a shot average of a mode of movement and a magnitude of movement in a shot. In the moving image data classification device according to claim 12,
The moving image data is data that has been compression-encoded using predictive encoding, and a shot average of the mode of motion and the magnitude of the motion in the shot is bidirectional with the forward motion predicted image in the shot. An apparatus for classifying moving image data, wherein the apparatus is calculated from motion information in at least one of the motion predicted images.
The moving image data classification device according to any one of claims 1 and 11 to 13,
The chase shot determining unit determines that the shot is a shot including a chase when the ratio of the mode of movement in the shot and the shot average of the magnitude of the movement are each greater than a predetermined value. Classification device for moving image data. The moving image data classification device according to any one of claims 11 to 14,
An apparatus for classifying moving image data, wherein a chase scene is determined by aggregating a plurality of shots determined to include a chase by the chase shot determining unit. The moving image data classification device according to claim 1,
An action shot determination unit for determining a shot including an action from the motion information and audio data of the moving image data, further determining a shot including the chase from the action, and performing hierarchical shot type classification; An apparatus for classifying moving image data. The moving image data classification device according to claim 1,
The apparatus for classifying moving image data, wherein the close-up shot determination unit is determined at least from color information in a screen. The apparatus for classifying moving image data according to claim 17,
The apparatus for classifying moving image data, wherein the close-up shot determination unit uses pixel data in an image as the color information and counts the number of pixels having a color corresponding to a skin color. The apparatus for classifying moving image data according to claim 17,
The close-up determination unit uses a color histogram in an image as the color information, and measures the frequency of a color range corresponding to skin color, and classifies moving image data. The moving image data classification device according to claim 18 or 19,
The close-up determination unit, in an arbitrary frame in a shot, if at least one of the number of pixels corresponding to the flesh color and / or the frequency of the color range is greater than a specified value, An apparatus for classifying moving image data, characterized in that a shot including the frame is determined as a shot including a close-up. The moving image data classification device according to any one of claims 17 to 20,
The close-up determination unit further uses an audio type as the audio information, and determines that the shot is a close-up shot when the ratio of the audio type in the shot is greater than a predetermined value. A classification device for moving image data. The moving image data classification device according to claim 1,
A conversation shot determining unit that determines a shot including a conversation from the moving image data is further provided, a shot including a close-up is determined from a conversation shot obtained from the conversation shot determining unit, and a conversation shot including a close-up is determined. An apparatus for classifying moving image data.

Images