JPH04320589A - Method for extracting moving object in moving image - Google Patents

Abstract

PURPOSE:To provide a moving object extracting method capable of preventing an object other than a moving object having low resistance against noise from being extracted from a moving image. CONSTITUTION:The moving object extracting method is characterized by finding out the speed vectors of respective small areas in a frame image, forming a frequency graph indicating the size and directions of the speed vectors, removing previously set patterns represented by components having large frequency or the like from said graph, and extracting a small area corresponding to a component of a left speed vector in the original frame image as a moving object area.

Description

[Detailed description of the invention]

0001

[Industrial Application Field] The present invention enables humans to edit and edit moving images.
This is a technology that automatically cuts out a moving object, which is a feature in a moving image, to be used as a clue when performing a search, and is related to a field that deals with video in general.

0002

2. Description of the Related Art Conventional techniques in this field can be roughly divided into two types: a technique for extracting objects from still images and a technique for extracting objects using inter-frame differences in moving images. In object extraction technology from still images, primitive element groups are obtained using basic image processing techniques such as binarization processing and edge extraction processing, and then element groups are extracted using a dictionary of interpretations prepared in advance. A process was used to extract objects from

0003

However, this technique has the disadvantage that it is susceptible to noise in the original image during the processing up to obtaining the element group, and this has an adverse effect on the comparison with the dictionary. Another problem is that the interpretation dictionary must be tuned up in advance in order to avoid erroneously extracting objects in the background image.

On the other hand, in the method of directly performing difference processing between frames on a moving image and extracting a moving object based on the obtained difference information, the entire frame image can be seen at a glance by camera operations such as panning and zooming. If the object appears to be moving, there is a problem in that the entire frame is extracted as a moving object. Furthermore, since the difference is taken pixel by pixel even when no camera operation is performed, the method is vulnerable to noise from frame to frame, and has the disadvantage that noise regions are extracted as regions of moving objects.

SUMMARY OF THE INVENTION An object of the present invention is to provide a method for extracting a moving object from a moving image, which solves the problem of the prior art in that objects other than moving objects that are susceptible to noise are extracted.

[0006]

[Means for Solving the Problems] A moving object extraction method in a moving image according to the present invention obtains a velocity vector for each small region in a frame image, and then calculates a frequency graph of the magnitude and direction of this velocity vector. This method removes preset patterns that are represented by components with high frequency, and then extracts a small area in the original frame image that corresponds to the remaining velocity vector component as a moving object area. .

[0007]

[Operation] In the present invention, since the velocity vector distribution for each small region in a frame having a certain size is used, it is robust against fine noise components in each frame. , by removing components of the velocity vector pattern caused by camera operation, image regions other than moving objects, which were extracted as moving regions in conventional techniques, can be extracted when the entire image in the frame is changing. disappears.

[0008]

[Embodiment] An embodiment of the present invention will be explained using FIG. FIG. 1 is an original frame image. FIG. 2 is a distribution diagram of velocity vectors created from the original frame image. Here, each small area is set to be large for the purpose of schematic illustration. Furthermore, here, it is assumed that the camera operation involves panning diagonally downward to the left, and an image in which a moving object is moving in the opposite direction.

FIG. 3 is a frequency graph (histogram) showing the frequency of the magnitude and direction of each velocity vector in the velocity vector distribution diagram. Here, in the background image, the component of the velocity vector for the region of the part that is moving due to the panning operation of the camera appears as the component with the highest frequency. Furthermore, the component with the second highest frequency is a component of the velocity vector corresponding to the region of the moving object. When each part of a moving object has complicated movements, the second peak P2 will be higher than the first peak P1, as shown in Fig. 3.
does not appear significantly, but this does not pose a problem in the present invention. Further, when the camera pans while chasing a moving object, the velocity vector of the background region is large and the velocity vector of the moving object is small, but in terms of frequency, the component of the background region increases.

[0010] By recognizing that it is a panning operation using a camera operation recognition method that has already been announced (for example, Japanese Patent Application No. 2-268521, ``Method for detecting the imaging state of an image pickup device''), the following processing can be performed. Become. FIG. 4 is a histogram after removing the above-mentioned most frequent velocity vector component. By removing the most frequent velocity vector component, areas in the image that are not moving objects, such as background areas, are removed. FIG. 5 is a diagram showing only small regions in the velocity vector distribution corresponding to the components in FIG. 4. FIG. 6 shows a small area in the original frame image corresponding to the velocity vector component shown in FIG. That is, (b
, 3), (b, 4), (c, 2), (c, 3), (c,
The small region 4) is cut out as the region of the moving object. The flow of the above processing is shown in FIG. In FIG. 7, (1) to (8) indicate each step, A represents the velocity vector of each small region, B represents the component with the highest frequency, and C represents the small region having the component with the highest frequency. Further, if each camera operation is recognized, by performing the same processing as described above, only the moving object in the frame image in which the camera operation such as fixation or tilting is performed can be extracted. That is, in the conventional technology, an area determined to be a moving object area can be removed by camera operation.

[0011] Processing becomes somewhat complicated even for frame images in which other camera operations such as zooming, booming, tracking, and dolly are performed, but it can be handled. Other embodiments are shown in FIGS. 8-10. This shows the following pattern that must be determined in advance when determining the component B with the highest frequency in step (5) in FIG. Figure 8 is a pattern for zooming, Figure 9 is a pattern for booming and trucking, and Figure 10 is a pattern for dolly. By extracting , it is possible to also extract a moving object in the frame image in which the camera operation described above is being performed.

FIG. 11 is a diagram showing the relationship between the type of camera operation used in the present invention, the position of the convergence point or divergence point, the magnitude of the vector, and the direction of the vector. The imaging state detection method of the image capturing device is driven by optical flow. Optical flow is an apparent velocity vector distribution on a screen caused by relative motion between an image capturing device and a subject.

Camera operations can be roughly divided into fixed shots, camera direction, camera position, camera height changes, and lens screen changes.

The definition of the image input operation from the optical flow is based on the relative magnitude and direction of each velocity vector in the velocity vector distribution of the optical flow, the convergence point position of all and some of the velocity vectors in the velocity vector distribution, or This is done by calculating and analyzing the divergence point position. The camera operation will be explained below.

○Fix Fix operation refers to a camera operation that causes the camera to stand still relative to the subject.

○Panning Panning operation refers to a camera operation in which the camera head is directed in the left/right (horizontal) direction.

○Tilt Tilt operation refers to a camera operation in which the camera head is directed up and down (vertically).

○Dolly Dolly operation refers to a camera operation in which the camera is moved in the direction of the subject (back and forth direction).

Track Track operation refers to a camera operation in which the camera is moved left and right (laterally) with respect to the subject.

Booming Booming operation refers to a camera operation in which the camera pedestal is used to raise and lower the camera height to change the camera angle.

[0021] Zoom Zoom operation refers to a camera operation that continuously changes the size of the object by changing the angle of view of the zoom lens without changing the relative positions of the camera and the object.

The relationship between the relative magnitude and direction of each velocity vector of the optical flow described above, the convergence point position or divergence point position of all or some of the velocity vectors in the velocity vector distribution, and the operation of the image input device is shown below. The example shown in FIG. 11 is that there is no overlap in the magnitude, direction, convergence point or divergence point position of the velocity vectors in any operation.

[0023]

As explained above, the present invention obtains a velocity vector for each small area in a frame image, creates a frequency graph of the magnitude and direction of this velocity vector, and calculates pattern components corresponding to individual camera operations. The small area in the original frame image corresponding to the remaining velocity vector component is cut out as the moving object area, so the area of the moving object in the video image appears to be moving due to camera operation etc. It is possible to extract background images that look similar without erroneously extracting them.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing an original frame image for explaining processing in an embodiment of the present invention.

FIG. 2 is a velocity vector distribution diagram for explaining processing in an embodiment of the present invention.

FIG. 3 is a histogram for explaining processing in an embodiment of the present invention.

FIG. 4 is a histogram after removing the highest frequency for explaining the processing in the embodiment of the present invention.

FIG. 5 is a diagram showing a corresponding area of a velocity vector distribution diagram for explaining processing in an embodiment of the present invention.

FIG. 6 is a diagram showing a corresponding area in an original frame image for explaining processing in an embodiment of the present invention.

FIG. 7 is a diagram showing the flow of processing performed in an embodiment of the present invention.

FIG. 8 is a diagram showing a histogram pattern given in advance in another embodiment of the present invention.

FIG. 9 is a diagram showing a histogram pattern given in advance in still another embodiment of the present invention.

FIG. 10 is a diagram showing a histogram pattern given in advance in still another embodiment of the present invention.

FIG. 11 is a diagram showing the relationship between the type of camera operation used in the present invention, the position of a convergence point or a divergence point, the magnitude of a vector, and the direction of a vector.

[Explanation of symbols] P1 1st peak P2 Second peak

Claims (1)

[Claims] Claim 1: A method for extracting a moving object from a moving image composed of a series of temporal frames, comprising:
Find the velocity vector for each small area in the frame image, create a frequency graph of the magnitude and direction of this velocity vector, remove pattern components corresponding to individual camera operations, and remove the original frame corresponding to the remaining velocity vector components. A method for extracting a moving object from a moving image, characterized by cutting out a small region in the image as a moving object region.