JP2003281548A - Video editing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video editing method and program for obtaining suitable and high-precision tracking result even if deformation of an object from a reference image can't be approximated by one set of affin deformation. <P>SOLUTION: In this video editing method using the result of tracing an object in video data, at least one still-picture representing the object is obtained as a reference image from the video data. According to the operation from a user, a dot or an area is specified on the reference image. According to the specification result, the object is divided into a plurality of small objects. Each of the plurality of divided small objects is traced. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video editing method and program suitable for video editing such as tracking and extracting an object in a video.

[0002]

2. Description of the Related Art In the field of image processing, there is a technique of focusing on a specific subject in a video and tracking the object. In this technology, the related information (for example, name, price, etc.) is given to the object in the image, and the anchor area is created when the user creates the interactive image by presenting the related information when the user points the object. It is used to specify an image or create a video for cutting out a tracked object from a background and synthesizing it with another background image for processing, or creating a web page content.

MPEG-4, which is an international standard for moving image compression
In Reference 1 “MPEG-4 standardized method”.
s for the compression of arbitrarily shaped video
objects '' N. Brady (IEEE Transactions on Circuits an
d Systems for Video Technology, vol. 9, no. 8, pp.
1170-1189, December 1999), a function of object coding for coding data for each object is adopted. In this object encoding, a tracking cutout technique of an object in a video is used to create object data before encoding.

As one of the concrete algorithms,
Japanese Patent Application Laid-Open No. 2000-132691 "Image region tracking method and apparatus" (Reference 2 "High-speed object tracking method using deformation estimation by robust statistics" Kaneko, Hori (Proceedings of the 5th Image Sensing Symposium, SII ' 99, C-18, p
p.129-134, 1999)). This method will be described with reference to FIG.

That is, a finite number of feature points effective for tracking are selected from the object area designated by the user's manual input in the reference image, as shown in FIG. Specifically, feature point candidates are arranged in a grid at predetermined intervals in the object region, a block of a predetermined size (for example, 8 × 8 pixels) is defined around the feature point candidates, and such feature point candidates are set. A feature point candidate that is determined to be unlikely to cause a block matching error (tracking error) when performing block matching in the surrounding area around the center and having low similarity obtained from this block matching, that is, when performing tracking with the processed image. Are selected in order from the highest reliability.

Subsequently, as shown in FIG. 10B, block matching is performed between the block centering on the feature point arranged in the reference image and the processed image to estimate the movement vector. In general, since these movement vectors include outliers, an affine parameter representing the deformation of the object is estimated by robust estimation, and FIG.
Tracking of an object in a processed image can be realized as in (c).

[0007]

However, in the conventional object tracking method using the affine deformation estimation, since the deformation is estimated by a set of affine parameters with respect to the movement of the object, the object from the reference image is estimated. If the deformation of 1 cannot be approximated by a set of affine deformations, an appropriate and highly accurate tracking result may not be obtained.

The present invention has been made in consideration of such circumstances, and it is possible to obtain an appropriate and highly accurate tracking result even when the deformation of an object from a reference image cannot be approximated by a set of affine deformations. An object of the present invention is to provide a video editing method and program that can be performed.

[0009]

In order to achieve the above object, the video editing method and program of the present invention are configured as follows.

A video editing method according to the present invention is a video editing method which involves tracking an object in video data, wherein at least one still image including the object is acquired from the video data as a reference image, and from a user. By dividing the object into a plurality of small objects by designating points or regions on the reference image in accordance with the operation, and connecting information between the plurality of small objects based on the division result in the dividing step. And the object tracking step of tracking each of the divided small objects.

A video editing program according to the present invention is a video editing program accompanied by tracking of an object in video data, and has a function of acquiring at least one still image containing the object from the video data as a reference image. And a function of dividing the object into a plurality of small objects by designating a point or a region on the reference image according to an operation from a user, and connection information between the plurality of small objects by the dividing function. It is a video editing program for causing a computer to realize a function of calculating based on a division result and an object tracking function of tracking each of the plurality of divided small objects.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 is a flowchart showing the flow of a video editing process according to the first embodiment of the present invention. The video editing process is realized as a program that runs on a computer (not shown), for example. As the computer, for example, a general-purpose personal computer (PC) or a workstation (W for image processing)
S), which has a basic hardware configuration such as a CPU, a memory, an external storage device, a graphic board (image processing unit), a display, and input / output devices such as a keyboard and a mouse. The computer program according to the present embodiment is an operating system (O) operating on the computer.
It can be realized as an application program corresponding to S).

In the video editing process according to the present embodiment, video data (moving image) of an object is input, object information is extracted, which will be described in detail later, and then object tracking is executed and the tracking result is output. As shown in FIG. 1, it is composed of a reference image storing step S101, an object information extracting step S102, and an object tracking step S103.

First, in the reference image storing step S101, a still image at any time in the video data is obtained as a reference image. For example, the reference image may be a frame in which the object to be tracked first appears in the video, or a series of frames representing the object at an arbitrary time point designated by the user. The reference image is temporarily stored in the memory in the computer and the external storage device.

Next, in the object information extracting step S102, information on the position and structure of the object in the reference image is extracted. The information on the position and structure of the object is information indicating in what form the object is in the reference image and at which position in the reference image, and is generally called a “mask image”. The mask image is, for example, a so-called binary image, which is an image composed of pixel values of only “0” or “1”.

This object information extracting step S102 is performed by the user with respect to the reference image in the object tracking method using the deformation estimation by the robust statistics described in the above-mentioned Japanese Patent Application Laid-Open No. 2000-132691 "Image area tracking method and apparatus". This corresponds to a portion in which the object area is designated by manual input. Further, as described above, since the tracking by this conventional method is performed as a modification represented by a set of affine parameters, it is possible to track the face of a car or a person, but the face and the body move separately. It was somewhat difficult to track a subject consisting of multiple objects such as a person's upper body with high accuracy.

Therefore, the object information extracting step S of the present embodiment
In 102, not only can the position and structure of the entire object be extracted as attribute information about the object, but this object can also be divided into a plurality of small objects. When such division is performed, each small object can be divided. It is configured to extract the (labeled) position and structure information for each. Incidentally, it is preferable that the connection between the plurality of small objects, that is, the connection information between the small objects is extracted and stored if necessary. The eighth mentioned later
In the embodiment, the mask image is highly accurate based on the stored connection information.

As shown in FIG. 2, the object information extracting step S102 includes an object background separating step S201 and an object dividing step S2.
It is roughly divided into 02. In the object background separation step S201,
The object in the reference image and the background are separated, and information on the position / structure of the object is extracted. This object background separation step S201
For example, a method of moving a pointer composed of a plurality of pixels on the screen by a user's operation, making the area to which the pointer has moved an object area, and extracting the shape and position thereof (Japanese Patent Laid-Open No. 2001-2001).
No. 14477), a region segmentation method based on color information of an object, a contour line tracking method for tracking the contour line of an object in pixel units based on edge information in a reference image, and a snake method using a dynamic contour line. Alternatively, when there is a background image in which an object does not exist, it can be realized by using a known object area designating method such as a background subtraction method.

The next object dividing step S202 is a step of dividing the object into a plurality of small objects by moving the pointer in the image in accordance with the user's operation on the mouse or the like. An example of the object dividing step S202 will be described in detail with reference to FIG. First, FIG. 3A shows one initial reference image in which a person's upper body S, which is an object, is drawn on the background BK0. On the other hand, FIG. 3B shows the object background separation step S20.
1 is the reference image in which the object S is separated from the background. The separated background portion BK1 is filled with a predetermined image pattern (check pattern, etc.). It should be noted that in the figure, for the sake of explanation, it is filled with a blank.

Here, FIG. 4 is a diagram for explaining the dividing pointer applied to the object dividing step S202 of the present embodiment. That is, P1 shown in FIG. 4A is a pointer that moves in the screen according to the user's operation, and erases the pixel data of the object in the passing locus (makes it into the background pixel data) the object. Configure a split tool for splitting. The user moves the division tool P1 in the image as shown in FIG. 4B to divide the object into a plurality of small objects. Here, the upper body of the person is divided into two small objects, a face portion and a body portion. By using this division tool P1, as shown in FIG. 4C, the position / structure information of the face portion and the position / structure information of the body portion are obtained and held. Further, as the connection information between the two small objects including the face part and the body part, the division tool P1 holds the coordinates of the points entering and exiting the object, respectively. In the example of FIG. 4, the object is divided into two small objects by using the division tool P1 and the position / structure information about each is extracted, but it may be divided into three or more small objects. Of course.

Next, the object tracking step S103 is a step of applying the above-described known object tracking method to each small object divided in the reference image by using the extracted position / structure information. Note that the object tracking method is not limited to only the object tracking method using the deformation estimation by the robust statistics described in the above-mentioned Japanese Patent Application Laid-Open No. 2000-132691 "Image region tracking method and apparatus".

More specifically, this object tracking step S10
3 processes the object tracking for the first small object (for example, face) by the first affine deformation according to the flow of FIG. 11, and the object tracking for the second small object (for example, the body) is performed in the flow of FIG. 11. According to the second affine transformation.

FIG. 11 is a flowchart showing details of the object tracking step S103.

This shows a procedure for implementing an object tracking algorithm using affine deformation estimation based on robust statistics.

(Step S1) A mask area corresponding to an input image (reference frame image) as shown in FIG. 12A is input as shown in FIG. 12B.

(Step S2) As shown in FIG.
The feature points P1 to P9 are selected in the reference frame.

(Step S3) A variable i indicating a frame number is initialized and set to i = (initial frame number + 1).

(Step S4) As shown in FIG.
Using the luminance information of the initial frame and the i-th frame, affine deformation parameters are obtained by block matching of blocks including each feature point. Furthermore, a set of affine deformation parameters is obtained by robust estimation.

(Step S5) As shown in FIG.
Using the affine deformation parameter obtained in step S4, the mask in the manually input initial frame is deformed to estimate the mask in the i-th frame. Thereby, the mask area in the i-th frame image as shown in FIG. 16A can be obtained as shown in FIG. 16B.

(Step S6) If the frame number i reaches the end frame, the process is ended. If not reached, 1 is added to i (step S7), and the process returns to step S4.

Therefore, even when the deformation of the object from the object in the reference image is represented by a plurality of affine deformations, the object in the image can be appropriately and accurately tracked. When the connection information between the small objects is held, it is preferable to perform the tracking process so that the connection information is maintained.

(Second Embodiment) In this embodiment, a small object selecting tool is used in place of the dividing tool P1 in the object dividing step S202 described in the first embodiment. Similarly, it is configured to extract the position / structure information of the small object. Other steps are the same as those in the first embodiment. FIG. 5 is a diagram for explaining the small object selection tool of this embodiment. T1 to T3 shown in FIG. 5 are components of a selection tool for selecting a small object from the object area. When the user wants to select a face portion as a small object from the upper body of the person in FIG.
First, as shown in FIG. 5A, the selection start point T1 is placed outside the small object.
Is specified. Subsequently, the user moves the selection tool T2 to display a variable rectangle (may be an ellipse) T3 surrounded by the selection start point T1 and the selection tool T2 as a selection range on the screen as shown in FIG. 5B. . Then, as shown in FIG. 5C, the user operates the selection tool T2 so that the selection area includes the face portion. In addition,
The body part may be selected as a region other than the face part in the object region in which the object is separated from the background in the object background separation step S201. Thereby, the position / structure information of the face portion and the position / structure information of the body portion can be extracted. Further, as the connection information between the two small objects including the face part and the body part, the intersection of the selection range and the object region may be held. For example, in FIG. 5C, the selection range corresponding to T3 intersects at two points on both ends of the body part (neck).

As described above, according to the present embodiment, the position / structure information of the object can be accurately extracted as in the first embodiment.

(Third Embodiment) In this embodiment, in the object dividing step S202 described in the first embodiment, a connection point designating tool is used in place of the dividing tool P1. Similarly, it is configured to divide into small objects. Other steps are the same as those in the first embodiment. FIG. 6 is a diagram for explaining the connection point designating tool of this embodiment. U1 to U4 shown in FIG.
Is a component of a designation tool for designating a connection point in the object region in which the object is separated from the background in the object background separation step S201. When the user wants to divide the face part and the body part into small objects from the upper body of the person in FIG. 6A, respectively, first, as shown in FIG. 6A, at the position that is the boundary between the face part and the body part. The designated tool U1 is moved and designated as a connection point. Then, as shown in FIG. 6B, the position U2 of the background area having the shortest distance from the designated point and the position U3 of the background area having the shortest distance in the opposite direction are automatically derived. These U2-U1
-Connect with a dividing line (preferably a free curve) U4 to connect the three points of U3. Thus, as shown in FIG. 6C, it is possible to obtain two small objects including a face part and a body part. Therefore, the position / structure information of the face part and the position / structure information of the body part can be extracted respectively. Further, as the connection information between the two small objects including the face part and the body part, the connection point U1 or the coordinates of the division positions U2 and U3 may be held.

As described above, according to the present embodiment, the position / structure information of the object can be accurately extracted as in the first embodiment.

(Fourth Embodiment) In this embodiment, as in the first to third embodiments described above in the object information extracting step S102, the object area is separated from the background by the object background separating step S201 as one object. Instead of separating and then dividing this object region into smaller objects,
The position of an object composed of a plurality of small objects by the processing on the reference image stored in the reference image storing step S101.
This is to extract structural information. For example, in FIG.
Let (a) be the reference image stored in the reference image storing step S101. First, the user has described the object background separating step S201 in the first embodiment in Japanese Patent Laid-Open No. 2001-
Only the face part is separated from the reference image by using the method described in Japanese Patent No. 14477. That is, with respect to the reference image of FIG. 7A, a pointer made up of a plurality of pixels is moved on the screen by a user operation, and the area moved by the pointer is designated as an object area by the face portion R1.
Only these are separated as shown in FIG. Then, in FIG.
Similarly, with respect to the reference image of (a), the pointer is moved within the screen, and only the body part R2 is separated as shown in FIG. 7C by the method of designating the area moved by the pointer as the object area. In this way, by specifying the object area and background area independently for multiple areas,
It is possible to extract the position / structure information about a plurality of small objects. It should be noted that, as the connection information, the end points of the overlapping area, the center of gravity, and the like may be held.

As described above, according to the present embodiment, the position / structure information of the object can be accurately extracted as in the first embodiment.

(Fifth Embodiment) Regarding position / structure information (reference frame and reference mask) as object extraction information in the tracking processing of the object tracking step S103, the first embodiment is always stored in the reference image storing step S101. The same reference frame and reference mask, and the object information extraction step S1
The information extracted by 02 is used to track each small object in the processed image. In the present embodiment, the reference frame and the reference mask are updated (renewed) with the object tracking result of the past processed image which has already been subjected to the object tracking processing. That is, the object in the processed image is tracked based on the position / structure information of the object estimated by the tracking process.

As a result, since the position / structure information of the object, which is more likely to be smaller than the deformation from the reference image, is used for the object of the processed image, it is possible to perform the tracking process with higher accuracy. Become.

(Sixth Embodiment) This embodiment is the object tracking step S103 described in the first to fifth embodiments.
As post-processing, Japanese Patent Laid-Open No. 2000-82145 "Object Extraction Device" (reference document "High-precision extraction of subject contour line using LIFS" Ida, Sanbonsugi, Watanabe (IEICE Transactions, D- II, vol. J82-D-II, no. 8, pp. 1282-1
289, 1999)). Thereby, the tracking accuracy of the object can be further improved.

In the tracking process described so far, since the deformation of the object is expressed only by the affine deformation, an error may occur. For example, H1 in FIG. 8A is obtained by applying the estimated affine transformation to the shape of the upper body of a person (mask image) of a certain reference image. What is the shape of the upper body of a person in the processed image? The contour line is slightly misaligned. in this case,
Japanese Patent Laid-Open No. 2000-82145 "Object Extraction Device"
8B by performing the contour correction processing described in FIG.
As shown in, the shape can be tracked more accurately.

(Seventh Embodiment) This embodiment is the same as the sixth embodiment.
The present invention relates to a combination of the embodiment and the fifth embodiment.
That is, in the present embodiment, the position / structure information of the object accurately tracked by performing the contour correction processing described in the sixth embodiment is used as the object extraction information as described in the fifth embodiment, and the processed image is obtained. Track objects in.

As a result, for the object of the processed image,
It is possible to use the position / structure information of the object represented by a closer deformation, and since the position / structure information is also the information whose accuracy has been improved by the contour correction processing, the object of the processed image can be more accurately processed. It becomes possible to perform a tracking process.

(Eighth Embodiment) The present embodiment is an embodiment for explaining in detail the tracking process which maintains the connection information described in the first embodiment.

As shown in FIG. 9A, the object dividing step S
Assume that the upper half of the person is divided into a face portion and a body portion by 202. Here, the coordinates of the two connection points C1 and C2 on the right side of the upper body of the person and the coordinates of the two connection points Q1 and Q2 on the left side of the upper body of the person are stored together. According to the first embodiment, the face part and the body part are independently subjected to the tracking process for each object. At this time, the connection points C1 and C2
And Q1 and Q2 also move independently.

After the object tracking step S103, in the present embodiment, as shown in FIG. 9B, the connection point C on the right side of the face portion is
1 and the corresponding connection point C2 in the body part are connected to each other by a connection line W2
To connect. Further, the connection point Q1 on the left side of the face portion and the corresponding connection point Q2 in the body portion are connected by the connection line W1. Further, an area surrounded by the connecting lines W1 and W2, the boundary line of the face part and the boundary line of the body part is defined as an object part. As a result, it becomes possible to implement a tracking process that maintains the connection information. FIG. 9C shows a mask image showing the position / structure information of the upper body region of the person obtained as a result. According to such an eighth embodiment, it is possible to improve the accuracy when cutting out (extracting) an object in an image.

The present invention is not limited to the above-described embodiment, but can be variously modified and carried out.

[0049]

As described above, according to the video editing method of the present invention, by extracting the position / structure information of a plurality of small objects in a reference image, a plurality of affine deformations can be combined accordingly. Makes it possible to track an object. Therefore, it is possible to accurately track the object in the image even for the object whose deformation in the reference image can be approximated by a combination of a plurality of affine deformations.

[Brief description of drawings]

FIG. 1 is a flowchart showing a flow of video editing processing according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing details of the object information extraction process in FIG.

FIG. 3 is a diagram for explaining in detail an example of an object dividing step according to the first embodiment.

FIG. 4 is a diagram for explaining a division pointer according to the first embodiment.

FIG. 5 is a diagram for explaining a small object selection tool according to the second embodiment.

FIG. 6 is a diagram for explaining a connection point designation tool according to a third embodiment.

FIG. 7 is a diagram for explaining extraction of position / structure information of an object based on a reference image according to the fourth embodiment.

FIG. 8 is a diagram for explaining the sixth embodiment,
Diagram showing the deviation of the contour line based only on affine deformation

FIG. 9 is a diagram for explaining a tracking process that keeps connection information according to the eighth embodiment.

FIG. 10 is a diagram for explaining a high-speed object tracking method using deformation estimation based on robust statistics.

FIG. 11 is a flowchart showing the processing steps of the object tracking method.

FIG. 12 is a diagram for explaining the processing steps of the object tracking method, showing a reference frame and its mask image.

FIG. 13 is a diagram for explaining the processing steps of the object tracking method, showing the selection of feature points in a reference frame.

FIG. 14 is a diagram for explaining the processing steps of the object tracking method, showing the destination of the feature point.

FIG. 15 is a view for explaining the processing steps of the object tracking method, showing the deformation estimation of the mask image of the i-th frame.

FIG. 16 is a diagram showing a reference image of the i-th frame and its mask image.

[Explanation of symbols]

S101 ... Reference image storing step S102 ... Object information extraction step S103 ... Object tracking image S201 ... Object background separation process S202 ... Object dividing step

Claims (16)

[Claims] 1. A video editing method involving tracking of an object in video data, the step of acquiring at least one still image including the object from the video data as a reference image, and the reference according to an operation from a user. Dividing the object into a plurality of small objects by designating points or regions on the image; calculating connection information between the plurality of small objects based on the division result in the dividing step; An object tracking step of tracking each of the plurality of small objects that have been created. 2. The video editing method according to claim 1, wherein the object tracking step uses affine deformation estimation based on robust statistics. 3. The dividing step comprises a step of dividing the object by erasing pixel data of the object in a passing locus of a pointer moving on the reference image. The video editing method according to any one of items. 4. The dividing step comprises the step of designating an area on the reference image and dividing the object into a plurality of small objects divided by the area.
Or the video editing method according to any one of 2. 5. The dividing step includes a step of designating a connection point between the small objects and a step of generating a dividing line including the designated connection point based on the reference image. The video editing method according to claim 1. 6. The method according to claim 1, further comprising a step of updating a reference image used for the current object tracking by using a previous object tracking result obtained by the object tracking step. Video editing method described. 7. The video editing method according to claim 1, further comprising a step of performing contour line correction on the object tracking result. 8. A step of holding connection information between the plurality of small objects before and after the object tracking step, and a step of updating the object tracking result based on the connection information after the object tracking step, The video editing method according to any one of claims 1 to 7, further comprising: 9. A video editing program accompanied by tracking of an object in video data, comprising: a function of acquiring at least one still image including the object from the video data as a reference image; A function of dividing the object into a plurality of small objects by designating points or regions on the reference image, and a function of calculating connection information between the plurality of small objects based on a division result by the dividing function. An image editing program for causing a computer to realize an object tracking function of tracking each of the plurality of divided small objects. 10. The video editing program according to claim 9, wherein the object tracking function uses affine deformation estimation based on robust statistics. 11. The dividing function comprises a function of dividing an object by erasing pixel data of the object in a passing locus of a pointer that moves on the reference image. The video editing program according to any one of items. 12. The dividing function comprises a function of designating an area on the reference image and dividing the object into a plurality of small objects divided by the area. The video editing program according to any one of items. 13. The dividing function comprises a function of designating a connection point between small objects and a function of generating a dividing line including the designated connection point based on the reference image. The video editing program according to claim 9. 14. The method according to claim 9, further comprising a function of updating a reference image used for the current object tracking by using a previous object tracking result by the object tracking function.
3. The video editing program according to any one of 3 above. 15. The method according to claim 9, further comprising a function of performing contour line correction on the object tracking result.
4. The video editing program according to any one of 4 above. 16. A function of holding connection information between the plurality of small objects before and after the object tracking, and a function of updating the object tracking result based on the connection information after the object tracking. The video editing program according to any one of claims 9 to 16, characterized by: