JP2005252859A - Scene-splitting device for dynamic image data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a scene-splitting device for classifying non-compressed or compressed dynamic image data into various kinds of shots through the use of image characteristics or audio characteristics, to split a scene based on the classified results. <P>SOLUTION: A single or a plurality of consecutive shots, in which scenery is recorded to describe a scene after transition, are frequently inserted at a point where scenes change in the dynamic image data, such as a movie or a drama. A scenery shot determining section 4 is provided in addition to a motion information analysis section 2 and an audio analysis section 3. The scenery shot determining section 4 detects the scenery shot, by using at least motion information in a screen. A scene-splitting section 5 can detect the splitting point of the scene, based on the determination results of the determining section 4. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a scene segmentation device for moving image data, and in particular, analyzes uncompressed or compressed moving image data and classifies them into various shot types, and then presents a concept that is semantically higher than shots. The present invention relates to a scene dividing device for moving image data that performs scene division. Also, a moving image data scene dividing device capable of providing efficient search, classification and browsing of moving image data by dividing uncompressed or compressed moving image data into various scene types. About

  As conventional technology related to shot classification of moving image data, in Japanese Patent Application No. 2003-51514, which is a patent application of the present applicant, shots are classified into “action” class, “dramatic” class, “conversation” class, “general purpose” class, etc. Means for classifying, means for classifying a scene type having no semantic concept such as a slow scene, zoom scene, pan scene, etc., and means for classifying a scene type having a semantic concept such as a commercial scene or a highlight scene are disclosed. .

  As scene division methods, Japanese Patent Application Laid-Open No. 10-257436 and Japanese Patent Application Laid-Open No. 2000-56420 disclose an apparatus that integrates scenes using similarity for each divided shot. In JP-A-9-44639 and JP-A-2002-125178, a histogram of color information is used as a feature amount of a shot. In JP-A-10-257436, a representative frame of a shot is used. The similarity is obtained by fuzzy reasoning. Furthermore, Japanese Patent Laid-Open No. 2000-69420 discloses a device for scene separation based on CM detection and scene integration based on audio similarity and camera motion information.

Here, it is assumed that a shot is a set of continuous frames shot by one camera work, and a scene is a set of continuous shots that are semantically connected.
Japanese Patent Laid-Open No. 10-257436 JP 2000-56420 A JP-A-9-44639 JP 2002-125178 A JP 2000-69420 A

  The conventional technology classifies by shot unit, and when searching and browsing for shots belonging to a certain shot type, the granularity of the classification makes the search / view result enormous depending on the application. Efficiency can be reduced. In addition, when classifying by shot unit, shot types in adjacent or neighboring shots are not taken into consideration, so that temporally isolated shots may be classified as isolated shots (shot types different from the surrounding shot types). There was a drop in classification accuracy.

  In addition, since conventional scene division techniques use signal level feature values as determination elements for all colors, movements, and sounds, for example, when there are multiple color characteristics in the same scene, or different in the same scene If camera work is included or different audio features are included such as sound effects in a conversation scene, the scene may not be accurately divided.

  The present invention has been made in view of the above-described prior art, and an object of the present invention is to classify uncompressed or compressed moving image data into various shot types using image characteristics and audio characteristics, and classify the classification. An object of the present invention is to provide an apparatus for scene division based on the result. Another purpose is not to evaluate the characteristics of the signal level of each shot, but to classify the shots after classifying them into a more advanced concept (shot type), so that more accurate scene division than the conventional technology can be achieved. An object of the present invention is to provide a scene dividing device for moving image data that is made possible.

  In order to achieve the above-described object, the present invention provides a shot dividing means for performing shot division of moving image data, motion information of moving image data, and moving images in an uncompressed or compressed moving image data scene dividing device. A motion information / audio data analysis means for analyzing audio data attached to the data, and a shot in which the camera is shooting a landscape using at least one of the shot length of the moving image data, the movement information in the image, and the audio data The first feature is that it includes a landscape shot determination unit for determining and a scene division unit that regards the landscape shot as a scene division point and divides moving image data into scenes.

  In addition, a shot division unit that performs shot division of moving image data, a shot type classification unit that classifies moving image data into a plurality of semantic shot types, and a section in which the same shot type continues is regarded as a scene, and the same type A second feature is that a shot integration unit that divides moving image data into scenes by integrating the shots having the above is provided.

  According to the present invention, focusing on the fact that landscape scenes often appear at scenes and scene change points in moving image data such as movies and dramas, landscape shots are detected from the moving images, and the landscape shots are detected. Since scene division points are used, scene division of moving image data can be performed easily and accurately. For this reason, it is possible to easily and accurately search and view a desired scene from the moving image data, and to effectively classify a large number of moving image data.

  In addition, according to the present invention, since moving image data is classified into a plurality of semantic shot types, a section in which the same shot type continues is regarded as a scene, and shots having the same type are integrated. A scene composed of one shot is avoided, and the scene division of the moving image data becomes accurate. In addition, a section in which the same shot type continues due to the integration is determined as a scene, and a point where the shot type changes is set as a scene division point.

  In addition, according to the present invention, by using the signal characteristics of uncompressed or compressed moving image data, the input moving image data can be automatically and accurately classified into various shot types, and the scene is divided. It becomes possible.

  Hereinafter, the present invention will be described in detail with reference to the drawings. First, an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a configuration diagram of a moving image data scene dividing apparatus.

  First, uncompressed or compressed moving image data is divided into shot units by the shot dividing unit 1 to obtain a shot length Ls. Next, the motion information analysis unit 2 extracts motion information (motion intensity, motion direction uniformity, etc.) from the shot moving image data, and the audio analysis unit 3 extracts audio information from the accompanying audio data. For these processes, means disclosed in Japanese Patent Application No. 2003-51514 can be used. Therefore, a detailed description thereof is omitted, but a brief description is as follows.

  That is, for the compressed moving image data, the motion information analysis unit 2 calculates the motion intensity value MIs in the shot using the motion vector value of the predictive encoded image existing in the shot. For the motion intensity MIs, “Motion Intensity” that is an element of “Motion Activity Descriptor” defined in MPEG-7 can be used. The “motion intensity” element of the “motion activity descriptor” is expressed by an integer from 1 to 5 (1 is the lowest, 5 is the highest).

  On the other hand, for uncompressed moving image data, the motion from the previous screen is estimated using a block matching method or the like, and the obtained value is expressed as a motion vector. calculate. At this time, as the motion intensity MIs as a shot, a value obtained by averaging the motion intensity values in the target predictive encoded image within the shot, or a maximum value or an intermediate value thereof can be used. In addition, as a prediction encoded image and a motion vector to be subjected to motion information analysis, a forward prediction encoded image, a bidirectional predictive encoded image, and a forward motion vector and a backward motion vector in a bidirectional predictive encoded image Any combination of these can be used.

  The audio analysis unit 3 includes an audio power calculation unit, and the audio power calculation unit calculates the power Ps of the audio signal of the audio data in the input shot or the audio power Psb for each band. When calculating the audio power Psb for each band, an arbitrary bandwidth can be selected, and weighting can be performed for each band. The sum of these is the audio power Ps in the shot. The Ps is expressed as follows.

Here, Isb is a band number of the lowest band for calculating the audio power, hsb is a band number of the highest band, and w [i] is a weight for the audio power Psb [i] in the band i. The subband energy SEs can be obtained based on the audio power Ps.

  In moving image data such as movies and dramas, a single shot or a plurality of continuous shots are often inserted at scene and scene change points in order to explain the scene scene after the transition. Accordingly, a landscape shot determination unit 4 is provided in addition to the motion information analysis unit 2 and the audio analysis unit 3. The landscape shot determination unit 4 detects a landscape shot using at least the motion information in the screen. Next, by using the determination result of the landscape shot determination unit 4, the scene division unit 5 can detect scene division points. The scene division unit 5 determines scene information.

  The motion information analysis unit 2 extracts motion uniformity in addition to the motion intensity. This is an index indicating how constant the direction of movement (camera, object) in the screen is. By evaluating the uniformity of the direction of motion, it is possible to identify shots in which the camera is slowly panning / tilting the landscape. In other words, if the direction of movement in the screen can be identified by a certain evaluation scale, the object (subject) has little movement and the camera is shooting a relatively stationary scene with slow camerawork. I understand. In addition, the means disclosed in Japanese Patent Application No. 2002-285667, which is a patent application by the present inventors, can also be used for determining the pan scene.

  Next, the operation of the landscape shot determination unit 4 will be described with reference to the flowchart of FIG. The landscape shot determination unit 4 evaluates the input shot length Ls, motion intensity MIs as motion information, motion direction uniformity MDCs, subband energy SEs as audio information, and the like. Here, the motion direction unity MDCs can use the dominant direction attribute (Dominant Direction) of the motion activity descriptor (Motion Activity) defined in MPEG-7. As described above, the motion vectors existing in the screen can be classified into an arbitrary number of directions, and the direction with the highest frequency can be defined as the ratio of all the motion vectors.

   MDCs = number of motion vectors matching the most frequent direction / total number of motion vectors

  Here, the direction of the motion vector can be 8 directions (in units of 45 degrees).

  As described above, the uniformity MDCs in the direction of movement increases with shots where the camera captures a landscape, and decreases with shots in which the subject is photographed greatly or shots with fast movement of the subject. Accordingly, the shot length Ls is larger than a certain threshold value THL (for example, 1.5 seconds) (shot length Ls> THL), and the motion intensity MIs is smaller than a certain threshold value THI (for example, 1.1) (motion intensity MIs < THI), motion direction uniformity MDCs is greater than a certain threshold value THMDC (eg, 0.2) (motion direction uniformity MDCs> THMDC), and subband energy SEs is a threshold value THSE (eg, 2.2). If it is smaller than (subband energy SEs <THSE), the process proceeds to step S2 to determine that the corresponding shot is a landscape shot.

  Although it is possible to perform threshold processing on shot length, motion information, and audio information as described above, a classifier (such as a decision tree) that has been previously input as a feature vector is learned, and the input shot is classified into various shot types. Can also be classified.

  Next, a second embodiment of the present invention will be described with reference to FIG. This embodiment solves the following problems of the conventional apparatus. In other words, the conventional apparatus only determines the shot type for each shot, and does not consider the shot types of adjacent or neighboring shots. For example, a single “conversation” in a section where “action” classes are continuous. Since the class is sometimes judged, the problem is that the classification accuracy is lowered. In this embodiment, the shot type correction and shot integration are performed in consideration of the landscape shot. 3 that are the same as or equivalent to those in FIG. 1 are denoted by the same reference numerals.

  In the figure, 11 is a shot type classification processing unit, 12 is a shot type correction processing unit, and 13 is a shot integration unit.

  Here, the operation of the shot type correction processing unit 12 will be described with reference to the flowchart of FIG. The shot type correction processing unit 12 corrects misclassified shots by the following processing. Here, Type (n) represents the shot type of the nth shot.

  In step S10, i is first initialized (i = 1), and in step S11, the type Stype (i) of the i-th shot i is input. In step S12, it is determined whether or not the shot i is not a landscape shot, but Type (i) ≠ Type (i-1) and Type (i-1) = Type (i + 1). If the determination is affirmative, the process proceeds to step S13 to correct Type (i) = Type (i-1). On the other hand, if the determination in step S12 is negative, step S13 is skipped and the process proceeds to step S14. In step S14, it is determined whether or not all shots have been processed. If this determination is negative, the process proceeds to step S15, and i is incremented by one. Thereafter, the same operation as described above is performed until the determination in step S14 becomes affirmative, that is, until the moving image data ends.

  If the determination in step S14 is affirmative, i is initialized (i = 1) in step S16, the process proceeds to step S17, and the type of the i-th shot i is input again. In step S18, it is determined whether the shot i is not a landscape shot, but whether Type (i) ≠ Type (i−1) and Type (i) ≠ Type (i + 1). If this determination is affirmative, the process proceeds to step S19 to correct Type (i) = Type (i-1). On the other hand, if the determination in step S18 is negative, the process proceeds to step S20. In step S20, it is determined whether or not all shots have been processed. If this determination is negative, the process proceeds to step S21, and i is incremented by one. Next, the process returns to step S17, and the above-described processing is performed again. When the above process is repeated and the determination in step S20 becomes affirmative, the shot type correction process ends.

  Further, the shot type correction processing unit 12 may correct a shot type different from an adjacent shot type by a majority process.

  FIG. 5 shows a specific example of shots processed by the shot type classification processing unit 11, the shot type correction processing unit 12, and the shot integration unit 13 of FIG.

  FIG. 5A shows moving image data divided into scenes, and FIG. 5B shows shots divided into shots. FIG. 3C shows shots that have been shot classified by the shot type classification processing unit 11. In the illustrated example, a series of shots “conversation, action, conversation”, a series of shots “action, conversation, action”, a series of shots “action, landscape, action”, and the like are shown.

  FIG. 4D shows a shot that has been shot type corrected by the shot type correction processing unit 12. By the operation of the shot type correction processing unit 12, the series of shots “conversation, action, conversation” is corrected as “conversation, conversation, conversation”. In addition, a series of shots of “action, conversation, action” is corrected as “action, action, action”.

  That is, when the target shot is different from the preceding and following shot types and the preceding and following shot types are the same, all are corrected to the same shot type. Also, when the target shot is different from the preceding and following shot types and the preceding and following shot types are not the same, the shot type of the target shot is corrected to the previous shot type in terms of time.

  On the other hand, the series of shots “action, landscape, action” are held as they are because the landscape shot is a scene division point.

  Next, FIG. 4E shows a scene that has been shot-integrated by the shot integration unit 13. As can be seen from the figure, shots of the same type are integrated into one scene. Note that landscape shots that appear alone or in succession are assumed to be scene division points, and the scene integration processing is not performed. The start point of the landscape shot is set as a scene division point.

  With the above processing, in the corrected set of genre types, a section where the same shot type continues is determined as a scene, and a point where the shot type changes is a scene division point. Also, a scene composed of a single shot is avoided.

  Next, a third embodiment of the present invention will be described with reference to FIG. 6 that are the same as or equivalent to those in FIG. 3 are given the same reference numerals. In this embodiment, as the input to the shot type classification processing unit 11, the shot length Ls from the shot division unit 1, the audio type ACLS and the subband energy SEs from the audio analysis unit 3 are input.

  As means for acquiring the audio type ACLS, means disclosed in Japanese Patent Laid-Open No. 10-247093 can be used. The shot type sequence obtained by the shot type classification processing unit 11 is determined by the shot type series analysis unit 21 as a shot type series. The shot type series analysis unit 21 can check the probability of the shot type Type using a model that represents a time series symbol such as HMM (Hidden Markov Model). Thereby, the isolated shot type is corrected. The shot integration unit 13 performs shot integration and outputs scene information by the same processing as the process from (d) to (e) of FIG.

  The scene information obtained by the shot integration unit 13 is described together with at least time information by the scene information description unit 22, and is output as a scene description file by using a description format such as MPEG-7.

  In the present invention, the input moving image data may be movie content. In this case, the shot types to be classified are at least actions, conversations, landscapes, etc., landscape shot types are set as scene division points, and shots of actions, conversations, other shot types are corrected by the shot type correction means, It can be integrated into at least an action scene, a conversation scene, and other scenes.

It is a block diagram which shows the structure of the outline of the 1st Embodiment of this invention. It is a flowchart which shows the process of a landscape shot determination part. It is a block diagram which shows the schematic structure of the 2nd Embodiment of this invention. It is a flowchart which shows the process of a shot classification correction process part. It is explanatory drawing of the specific example of the principal part of the process of this 2nd Embodiment. It is a block diagram which shows the structure of the outline of the 3rd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Shot division part, 2 ... Motion information analysis part, 3 ... Audio analysis part, 4 ... Scenery shot determination part, 5 ... Scene division part, 11 ... Shot classification classification process , 12 ... Shot type correction processing unit, 13 ... Shot integration unit, 21 ... Shot type series analysis unit, 22 ... Scene information description unit.

Claims (10)

In a scene dividing device for uncompressed or compressed moving image data,
Shot dividing means for performing shot division of moving image data;
Motion information / audio data analyzing means for analyzing motion information of moving image data and audio data accompanying the moving image data;
Landscape shot determination means for determining a shot in which the camera is shooting a landscape using at least one of shot length of moving image data, motion information in the image, and audio data;
A scene dividing device for moving image data, comprising scene dividing means for regarding the landscape shot as a scene dividing point and dividing the moving image data into scenes. In the scene dividing device for moving image data according to claim 1,
A moving image data scene dividing apparatus using uniformity of a moving direction as the movement information. In the scene dividing apparatus for moving image data according to claim 2,
The motion direction uniformity is obtained by classifying motion vectors existing in a screen into an arbitrary number of directions, and the direction with the highest frequency is obtained as a ratio of all motion vectors. Splitting device. In a scene dividing device for uncompressed or compressed moving image data,
Shot dividing means for performing shot division of moving image data;
Shot type classification means for classifying moving image data into a plurality of semantic shot types;
A scene dividing device for moving image data, comprising shot integrating means for dividing a moving image data into scenes by considering sections having the same shot type as a scene and integrating shots having the same type. The scene dividing device for moving image data according to claim 4,
further,
Landscape shot determination means for determining a shot in which the camera is shooting a landscape using at least one of shot length of moving image data, motion information in the image, and audio data;
A scene dividing apparatus for moving image data, which regards the landscape shot as a scene dividing point and divides moving image data into scenes. In the scene dividing device for moving image data according to claim 4 or 5,
A scene of moving image data, comprising shot type correction means for correcting a shot type different from an adjacent shot type in consideration of temporal continuity of shot types obtained by the shot type classification means Splitting device. In the scene dividing device for moving image data according to claim 6,
An apparatus for dividing a scene of moving image data, wherein the shot type correction unit corrects a shot type different from an adjacent shot type by a majority process of shot types existing in a certain section. In the scene dividing device for moving image data according to claim 6,
An apparatus for dividing a scene of moving image data, wherein the shot type correction unit corrects a target shot type by collating with a series model of shot types learned in advance. In the scene dividing device for moving image data according to claim 7 or 8,
The input moving image data is movie content, and the shot types to be classified are at least action, conversation, landscape, and others, the landscape shot type is a scene dividing point, and the shots of action, conversation, and other shot types are shot. A scene dividing apparatus for moving image data, wherein the scene dividing device is corrected by type correction means and integrated into at least an action scene, a conversation scene, and other scenes. In the scene dividing device for moving image data according to any one of claims 1 to 9,
A scene division apparatus for moving image data, comprising scene information description means for describing information divided into scenes together with at least time information.

Images