JP2007072789A - Image structuring method, device, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image structuring method and the like capable of dividing a personal image with blur due to high compression or low quality into six types of segments by means of camerawork and a combination of presence of a motion object inside an actual environment at a low calculation cost. <P>SOLUTION: In this image structuring method, an image to be analyzed is read, a motion vector between frames of the input image is calculated, a characteristic quantity determining a segment type from the calculated motion vector is calculated, and using the characteristic quantity, to identify the segment type of a frame unit. Based on the segment type of the frame unit, the input image is divided, and the divided image and the segment information are stored in a storage means. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to video structuring technology and indexing technology for improving manageability of video content, and in particular, unedited video material content (hereinafter, referred to as “photographed content”) taken by a photographer using a mobile phone with a camera, a digital camera, a digital video, The present invention relates to a video structuring method, apparatus, and program for dividing a video taken by an individual (referred to as personal video) using motion information.

  The conventional technique for dividing video into segments is to divide scenes that are similar to the video production process, that is, the method of dividing using editing information, and the video content such as color and movement into the same segment. It is roughly divided into how to do.

  As a typical technique for dividing according to the video editor's intention, in the video content, detection of cut of the edited video, detection of the section where the telop is displayed, detection of the section where the camera work has occurred, music generation This is a method of detecting a section in which a voice is generated and dividing a section in which each event occurs as a segment. Since this method uses an event in editing, it is particularly a method that functions effectively for edited video content (see, for example, Patent Document 1).

  As a typical conventional technique for classifying scenes with similar video content as the same segment, there is a method of dividing a video by detecting camera work using an MPEG motion compensation vector (for example, Non-Patent Document 1).

Further, there is a method of dividing an image into a steady state and a transitional state by paying attention to the amount of change in the color information of the image (for example, see Non-Patent Document 2).
JP-A-11-224266 Kentaro Dobashi, Ryoyuki Kodate, Ryuji Nishitoba, Hideyoshi Tominaga, “Examination of Camerawork Detection Using MPEG2 Motion Vector”, IEICE, Image Coding Symposium PCSJ2000 (2000) Tsutsumi Fujio, “Video Segmentation Method for Wearable Visual Memory Aid”, WISS2001 pp.155-160 (2001)

  However, in the above prior art, when an unedited video material content photographed by a photographer using a camera-equipped mobile phone, digital camera, digital video, etc. is divided into segments, there is no cut point or telop in the first place. The video cannot be divided into segments using.

  Also, in the case of a method of dividing scenes with similar video content into the same segment using camera work and color information, it is possible to divide by the movement of camera work etc. The most important subject movement is not considered, and the classification based on the subject movement or the like cannot be performed.

  Further, even when attention is paid to the movement of the subject from the video, the subject must be extracted from the background, and there is a problem that the cost of the calculation amount is generally increased. In addition, it is difficult to separate the background and the subject from an image with camera shake or unstable camerawork as if the photographer photographed without using a tripod or the like.

  The present invention has been made in view of the above points, and is intended for personal images that have high compression, low quality, and camera shake. An object of the present invention is to provide a video structuring method, apparatus, and program that can be divided into six types of segments in combination.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) defines a segment type based on camera work and presence / absence of a moving object in a video, divides video content into segments, and labels each segment with the segment type Because
A video reading step (step 100) of reading the video to be analyzed and temporarily storing it in the storage means;
A motion vector calculating step (step 101) of reading an input video from the storage means and calculating a motion vector between frames of the input video;
A feature amount calculating step (step 102) for calculating a feature amount for determining a segment type from the calculated motion vector;
A segment type identification step for identifying a segment type in units of frames using the feature amount (step 103);
Based on the segment type in units of frames, the input video is divided (step 104), and the divided video and segment information are stored in the storage means (step 105).

The present invention (Claim 2) is the video structuring method of Claim 1,
In the feature amount calculating step (step 102),
The frame image of the input video is divided into predetermined areas, and a feature amount is calculated for each divided area.

The present invention (Claim 3) is the image structuring method of Claim 2,
In the feature amount calculating step (step 102),
As a feature quantity, any one or more of an average, a variance (standard deviation), an average change amount, and a variance (standard deviation) change amount in magnitude and direction of a motion vector is used and identified.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 4) defines a segment type based on camera work and the presence or absence of a moving object in a video, divides video content into segments, and labels each segment with a segment type. Because
Video temporary storage means 2 for temporarily storing the read video data;
Video / segment information storage means 7 for storing the divided video and segment information;
A video reading means 1 for reading a video to be analyzed and temporarily storing it in the video temporary storage means 2;
A motion vector calculation means 3 for reading an input video from the video temporary storage means 2 and calculating a motion vector between frames of the input video;
Feature quantity calculating means 4 for calculating a feature quantity for determining a segment type from the calculated motion vector;
Segment type identifying means 5 for identifying the segment type in units of frames using the feature amount;
And a video dividing unit 6 that divides the input video based on the segment type in units of frames and stores the divided video and segment information in the video / segment information storage unit 7.

Further, the present invention (Claim 5) is characterized in that in the feature amount calculation means 4 of the video structuring apparatus according to Claim 4,
Means for dividing the frame image of the input video into predetermined regions and calculating a feature amount for each of the divided regions.

Further, the present invention (Claim 6) is characterized in that the feature amount calculation means 4 of the video structuring apparatus according to claim 5
Includes a means for identifying and using as a feature quantity any one or more of average, variance (standard deviation), average change amount, and variance (standard deviation) change in magnitude and direction of the motion vector. .

The present invention (Claim 7) provides a computer,
A program for functioning as the video structuring apparatus according to claim 4.

  As described above, according to the present invention, for personal video that is generally complicated to manage, the motion information that is the most important information regarding the video is divided by the presence or absence of camerawork and moving objects. Can be indexed. As a result, the practical effects such as selection of representative images, more efficient application of metadata in the video, and improvement of the video listing are enormous.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows the configuration of the video structuring apparatus in one embodiment of the present invention.

  The video structuring apparatus shown in FIG. 1 includes a video reading unit 1, a video temporary storage unit 2, a motion vector calculation unit 3, a feature amount calculation unit 4, a sub-shot type identification unit 5, a video division unit 6, and video / segment information storage. And a learning data storage unit 8.

  The video reading unit 1 reads the one-shot personal video to be analyzed and stores it in the video temporary storage unit 2. Here, it is assumed that the personal video to be analyzed is already managed for each shot and stored in storage means such as a disk device. Note that a shot refers to a series of images taken from turning on and off the recording button of the camera. The video input to the video structuring apparatus is performed in units of shots. In particular, such video is called “one-shot video”. Depending on the system, the segment of the divided video is specifically defined as “sub-shot”, and the sub-shot type is defined as “sub-shot type”.

  The motion vector calculation unit 3 sequentially reads the video stored in the video temporary storage unit 2 in units of frames and calculates the motion vector of the frame.

  The feature amount calculation unit 4 calculates a feature amount for determining the sub-shot type from the motion vector calculated by the motion vector calculation unit 3. Here, in order to incorporate the elements of local motion and global motion into the feature amount, the frame image is divided into predetermined regions, divided into regions, and feature amounts are calculated for each region. Since the feature amount calculation unit 4 calculates the feature amount for each predetermined region without extracting the moving object from the background, the calculation amount cost can be suppressed.

  The sub-shot type identification unit 5 refers to the learning data stored in the learning data storage unit 8 and identifies the segment type for each frame using the feature amount calculated by the feature amount calculation unit 4.

  The learning data storage unit 8 holds learning data regarding all types of subjects and camera shake. This makes it possible to robustly identify any subject type or camera shake.

  The video dividing unit 6 divides the video using the segment type in units of frames and stores the video in the video / segment information storage unit 7.

  FIG. 4 shows an example in which the one-shot video in one embodiment of the present invention is divided into sub-shots.

  The example shown in the figure is an image in which a stopped subject starts to move, follow-up shooting is performed, and the subject stops. In the case of this video, four sub-shots are included.

  With the above configuration, when receiving a one-shot video as input, the video structuring apparatus automatically divides the video content into sub-shot types based on the photographer's camera work and the presence or absence of moving objects in the video.

  Next, the operation in the above configuration will be described in detail.

  First, a sub-shot of personal video is defined. FIG. 4 is a diagram showing the sub-shot types in the embodiment of the present invention.

  In the present embodiment, as shown in FIG. 4, the sub-shot has six types of ω0 to ω5. The vertical axis represents the type of camera work. There are three types of camera work: when there is no camera work, when there is camera work, and when zooming. When there is camera work, it includes pan, tilt, dolly, boom, and track. The vertical axis is divided into two categories when there is no moving object in the real world and when there is no moving object. Below are descriptions and examples of each category.

ω0: No camera work, no animal body;
(Example) When you are just shooting the object of interest, or when you are waiting to move;
ω1: No camera work, animal body;
(Example) When the object of interest moves within the frame, etc.
ω2: Camera work, no animal body;
(Example) Panning or tilting to take pictures of scenery, buildings, or the atmosphere of the place;
ω3: Camera work, no animal body;
(Example) When moving the camera according to the movement of the object of interest, etc .;
ω4: Camera work, moving body (no follow)
(Example) When panning or tilting to capture scenery, buildings, or the atmosphere of the place, including unanimated animals.
ω5: zoom section;
(Example) When there is an object of interest and you zoom in on that object, or when you zoom out to fit the entire scene in a frame, etc.
In the case of zooming, since the action itself includes the intention of a large photographer, it is not classified in detail by the presence or absence of a moving object.

  FIG. 6 is a flowchart of the operation in one embodiment of the present invention.

  Step 201) The video reading unit 1 reads the input one-shot video to be analyzed and accumulates it in the video temporary storage unit 2.

  Step 202) The motion vector calculation unit 3 sequentially reads out the video from the video temporary storage unit 2 in units of frames, and calculates the motion vector of the frame. In this embodiment, the motion vector is calculated by sequentially calculating the correspondence between the feature point frames. As another method, there is a method of calculating by block matching.

A specific method for associating feature points is shown below. Assume that the total number of frames of the video is N, the i-th frame image is Img (i), and the i + 1-th frame image is Img (i + 1). However, i = 1, 2,..., N−1. M feature points P _ij = (px _ij , py _ij ) (i = 1, 2,..., N−1, j = 1, 2,..., M) are detected from Img (i). In the feature point detection, points that have a large change in the density distribution of the image, such as corners and contours of an object, are extracted. For example, a Harris operator that is a conventional technique may be used. Detection of corresponding points Q _ij = (qx _ij , qy _ij ) (i = 1, 2,..., N−1, j = 1, 2,..., M) in Img (i + 1) _This can be realized by referring to the concentration distribution in the vicinity of _ij and obtaining a position having a high correlation with the concentration distribution in Img (i + 1).

Here, from the set of calculated natural feature points, a motion vector V _ij (i = 1, 2,..., N−1) between Img (i) and Img (i + 1) from the following formula (1). j = 1, 2,..., M).

V _ij = (vx _ij , vy _ij ) = (qx _ij −px _ij , qy _ij −py _ij ) (1)
(Where i = 1, 2,..., N−1, j = 1, 2,..., M)
The calculated motion vector V _ij and the start point P _ij of the motion vector are output. It is assumed that i = 1, 2,..., N−1, j = 1, 2,..., M, and i and j correspond to V _ij and P _ij . Finally, V _ij and P _ij are output to the memory in the motion vector calculation unit 3.

Step 203) The feature amount calculation unit 4 uses V _ij and P _ij (i = 1, 2,..., N−1, j = 1, 2,... M) read from the memory of the motion vector calculation unit 3. As input. The feature amount calculation unit 4 calculates a feature amount for identifying the segment type in units of frames from the motion vector.

In the present embodiment, as the feature amount,
・ Average motion vector size,
・ Standard deviation of the magnitude of the motion vector,
Standard deviation of the direction of the motion vector,
• Four of the average amount of change in the direction of the motion vector are adopted.

  When calculating the feature amount, as a psychology when a human shoots, a characteristic that a subject that is an object of interest is shot so as to be positioned near the center of the frame is used. Specifically, as shown in FIG. 7, the feature amount is calculated by dividing the inside of the frame into the inside and outside, thereby incorporating the elements of local motion and global motion into the feature amount. Hereinafter, the inner part of the frame is referred to as “inner area” and the outer part of the frame is referred to as “external area”. The definition of the area is arbitrary, but in this embodiment, the rectangle is similar in shape to the frame image, the size is 50% of the frame image, and the position is set so that the center of the frame image coincides with the center of the rectangle. The inside of the rectangle is defined as “inner area” and the outside is defined as “outer area”. Here, when the sub-shot identification is performed by changing the size of the rectangle experimentally using the learning data, a ratio at which a result with the highest identification rate can be obtained may be employed.

In order to determine whether the motion vector is an internal region or an external region of the frame, it is determined whether P _ij that is the start point of the motion vector is in the internal region or the external region of the frame.

Here, the feature quantity to be calculated is defined. The number of feature points included in the area to be handled is assumed to be K (k <M). The motion vector of the kth feature point of the feature point included in the area handled by the i-th frame image is
V _ik = (vx _ik , vy _ik ) (i = 1, 2,..., N−1, k = 1, 2,... K)
Formula (1)
Suppose that

It is.

The above four feature quantities are defined. The number of feature values in the inner region of the frame is Kinner, the number of feature values in the outer region is K _outer , and the value replaced with K in the above equations (2) to (5) is defined as the feature amount. As a result, a total of 8 dimensions of feature quantities are obtained for each frame, 4 dimensions for the internal area, 4 dimensions for the external area. With respect to the feature amount, the feature amount is calculated for all frames of the shot. After calculating the feature values of all the frames, in order to adjust the feature value scale, the average value is used for each feature value, and the normalization processing is performed so that the variance is equal at the center. Output as a quantity. The output feature quantity is T _i (i = 1, 2,..., N−1) and is output to a memory (not shown) in the feature quantity calculation unit 4.

  Step 204) Let N be the total number of frames.

  Step 205) The loop counter i is initialized (i = 1).

Hereinafter, the sub-shot type identification unit 5 receives the feature amount T _i calculated by the feature amount calculation unit 4 and determines the sub-shot type for each frame. As a determination method, there is also a method using a threshold value. However, in the case of home video, it is difficult to identify with a certain threshold process because there are camera shakes and subjects that cannot be predicted in the captured video. Therefore, in this embodiment, a method using learning data is adopted. In the present embodiment, the identification method is based on the simplest nearest neighbor determination rule. Another method is discriminant analysis.

  The learning data collects video classified into each sub-shot manually, and the same feature amount calculation processing by the motion vector calculation unit 3 and the feature amount calculation unit 4 described above in step 203 and step 204 is performed in advance. The feature amount is determined. This learning data is held in the learning data storage unit 8 in advance.

  Step 206) The sub-shot type identification unit 5 reads the learning data from the learning data storage unit 8, and maps the learning data group to the 8-dimensional identification space.

  Step 207) The sub-shot type identification unit 5 reads the feature amount of the i-th frame from a memory (not shown) in the feature amount calculation unit 4.

  Step 208) The sub-shot type identification unit 5 calculates learning data having the closest Euclidean distance to the identification target mapped in the identification space.

  Step 209) The sub-shot type identification unit 5 sets the sub-shot type to which the learning data with the shortest Euclidean distance belongs as the identification target sub-shot type. At this time, in order to prevent erroneous determination, a method of extracting the top a learning data points (a is predetermined) having the closest Euclidean distance and determining the type by majority decision is also conceivable. In addition, when the Euclidean distance is larger than a predetermined threshold ε, it can be determined that the reliability is low, and therefore it is conceivable to perform a rejection process.

  Step 210) It is determined whether all the frames have been identified. If there is an identification target to be determined, the process proceeds to Step 211, and if all the frames have been identified, the process proceeds to Step 212. At this time, the sub-shot type Ωi (i = 1, 2,..., N) of each frame is output. However, Ωi is any one of ω0, ω1, ω2, ω3, ω4, and ω5.

  Step 211) Set i = i + 1 and proceed to Step 207.

  Step 212) The video dividing unit 6 receives the sub-shot type Ωi (i = 1, 2,..., N) of each frame as an input, and divides the input one-shot video into sub-shots. FIG. 8 shows an example of sub-shot division in the video division unit according to the embodiment of the present invention. In each frame, as shown in FIG. 8, noise is generated in some places, and there is a possibility that a large number of sub-shots are made. Therefore, the frames are divided in chronological order for each predetermined number of frames b. This section is called a sub-shot minimum section. FIG. 8 shows an example in which b = 8. The sub-shot type of the minimum sub-shot section is determined by majority using the b frame sub-shot types included in the minimum sub-shot section. Here, noise refers to a frame in which the sub-shot type between frames is determined to be ω 0 in the interval determined by majority vote with ω 1 in the minimum sub-shot interval. Similarly, in the sub-shot minimum section, in the section determined to be ω2, a frame in which the sub-shot type between frames is determined to be ω3 becomes noise.

  If it is not decided by majority decision, the identification result of the sub-shot minimum interval before and after is adopted, the priority order is assigned to the sub-shot type in advance, the majority decision is made by merging with the minimum sub-shot interval before and after, etc. , Rules can be set in advance. In the present embodiment, when the majority decision is not made, the identification result of the previous segment minimum section is adopted. If the first sub-shot minimum section of the one-shot video is not determined by majority, the sub-shot type is ω0. The same processing is performed in all the sub-shot minimum sections, and the result Ω′j is output to a memory (not shown) in the video dividing unit 6. Here, if the number of sub-shot minimum sections is U, j = 1, 2,...

  Step 213) The video dividing unit 6 obtains the sub-shot type Ω′j of the sub-shot minimum section from the memory, merges if the preceding and succeeding types are the same, and performs the merging if the opposite type is switched. And the frame numbers of the in and out points of the sub-shot are recorded. The video / segment information is stored in the video / segment information storage unit 7 together with the information on the in and out points of the sub-shot and the video. FIG. 9 shows an example of video segment information according to an embodiment of the present invention.

  Through the processing described above, calculation costs can be kept low for personal images with high compression, low quality, and camera shake, and roughly divided into 6 types of segments based on the combination of camera work and the presence or absence of moving objects in the real environment. It becomes possible to divide into.

  In addition, it is also possible to construct | assemble said operation | movement as a program, install it in the computer utilized as an image | video structuring apparatus, to execute, or to distribute | circulate through a network.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to a video structuring technique and an indexing technique.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the image | video structuring apparatus in one embodiment of this invention. It is an example which divided | segmented the one-shot image | video in one embodiment of this invention into the subshot. It is a figure which shows the subshot classification in one embodiment of this invention. It is a flowchart of the operation | movement in one embodiment of this invention. It is an example of the internal area | region and external area | region in one embodiment of this invention. It is an example of sub-shot division in one embodiment of the present invention. It is an example of the video segment information in one embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image | video reading means, image | video reading part 2 Image | video temporary storage means, image | video temporary storage part 3 Motion vector calculation means, Motion vector calculation part 4 Feature value calculation means, Feature value calculation part 5 Segment type identification means, Sub-shot type identification part 6 Video division means, video division section 7 Video / segment information storage means, video / segment information storage section 8 Learning data storage section

Claims (7)

A video structuring method that defines segment types based on camerawork and presence or absence of moving objects in video, divides video content into segments, and labels each segment with a segment type,
A video reading step of reading the video to be analyzed and temporarily storing it in the storage means;
A motion vector calculating step of reading an input video from the storage means and calculating a motion vector between frames of the input video;
A feature amount calculating step for calculating a feature amount for determining a segment type from the calculated motion vector;
A segment type identification step for identifying a segment type in units of frames using the feature amount;
A video dividing step of dividing the input video based on a segment type in a frame unit and storing the divided video and segment information in a storage means;
A video structuring method characterized by: In the feature amount calculating step,
Dividing the input frame image of the video into predetermined regions and calculating a feature amount for each of the divided regions;
The image structuring method according to claim 1. In the feature amount calculating step,
As the feature quantity, the size and direction of the motion vector are averaged, variance (standard deviation), average change amount, and change amount of variance (standard deviation) are used in combination and identified.
The video structuring method according to claim 2. A video structuring device that defines segment types based on camerawork and the presence or absence of moving objects in a video, divides video content into segments, and labels the segment types for each segment,
Video temporary storage means for temporarily storing the read video data;
Video / segment information storage means for storing the divided video and segment information;
Video reading means for reading the video to be analyzed and temporarily storing it in the video temporary storage means;
A motion vector calculating means for reading an input video from the video temporary storage means and calculating a motion vector between frames of the input video;
Feature quantity calculating means for calculating a feature quantity for determining a segment type from the calculated motion vector;
Segment type identifying means for identifying a segment type in units of frames using the feature amount;
Video dividing means for dividing the input video based on a segment type in units of frames, and storing the divided video and segment information in the video / segment information storage means;
A video structuring apparatus comprising: The feature amount calculating means includes:
Means for dividing the input frame image of the video into predetermined regions and calculating a feature value for each of the divided regions;
The image structuring apparatus according to claim 4. The feature amount calculating means includes:
Means for using and identifying one or more of the average and variance (standard deviation), the average change amount, and the variance (standard deviation) change amount of the magnitude and direction of the motion vector as the feature amount including,
The image structuring apparatus according to claim 5. Computer
7. A video structuring program which functions as the video structuring apparatus according to claim 4.