JP2000270337A - Device and method for motion detection - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect a motion vector of a dynamic picture object even when the dynamic picture object is moved in a moving picture so as to be overlapped with a still object resulting in deforming the dynamic picture object. SOLUTION: A forward motion vector is detected by matching a dynamic picture object included in a 1st picture with a dynamic picture object included in a 2nd picture so as to obtain a forward matching distance (ST41). Furthermore, a reverse motion vector is detected by matching the dynamic picture object included in the 1st picture with a dynamic picture object included in a 3rd picture so as to obtain a reverse matching distance (ST42). On the basis of a comparison result between the forward matching distance and the reverse matching distance (ST43), the forward motion vector or the reverse motion vector is detected as a motion vector of the dynamic picture object included in the 1st picture (ST44, 45).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion detecting apparatus and method for detecting a motion vector of a moving image object included in a moving image.

[0002]

2. Description of the Related Art Due to the remarkable progress in the processing capability of computer computers in recent years, the object of image processing has been extended to moving images. Also in the field of the Internet, the appearance of a moving image browser, a moving image search system for searching for moving images, and the like have been demanded. In the moving picture browser and the moving picture search system, it is necessary to detect a motion vector of a moving picture with high accuracy and perform image processing.

When a moving image includes a moving image object having motion, an occlusion occurs in which the outline of the moving image object is deformed when the moving image object overlaps with a still object having motion and no motion. . When the motion vector of the still object A is “0”, the moving image object B has the motion vector B, and the moving image object B is set as the background of the still object A, as shown in FIG. It can be seen that this has occurred. According to FIG. 20,
It can be seen that the overlap ratio between the still object A and the moving image object B changes between the frames t and t + 1.

[0004] Here, assuming that there is no deformation of the still object A and the moving image object B itself between the frame t and the frame t + 1, the frame t indicates the still object A by being divided into a plurality of regions. Area A
t and an area Bt indicating the moving image object B. In addition, the frame t + 1 is divided into regions to form an image including a region At + 1 indicating the still object A and a region Bt + 1 indicating the moving image object B.
Then, the area B indicating the background moving image object B with respect to the stationary object A is eroded by the area A indicating the stationary object A which is the foreground, and forms a pseudo contour including the eroded portion.

[0005] Then, from time t to time t + 1,
Even if the foreground area At becomes the area At + 1, the shape of the area A does not change. However, if the background area Bt becomes the area Bt + 1, the area Bt + 1 has a different shape from the area Bt due to occlusion.

[0006] Occlusion can be detected by detecting the presence or absence of the deformation from the area Bt to the area Bt + 1.

[0007]

[0006] Further, by performing matching using an area as a template, a frame t to a frame t
When detecting a motion vector to +1, the motion vector from the region At to the region At + 1 has a large similarity because there is no deformation between frames, that is, the minimum matching distance becomes small, and an accurate motion vector is detected. Can be.

On the other hand, the motion vector from the area Bt to the area Bt + 1 has a small similarity due to deformation due to occlusion between frames, that is, the minimum matching distance often does not become small. That is, the motion vector from the region Bt to the region Bt + 1 is often not accurately detected because the pseudo contour is deformed by changing from the region Bt to the region Bt + 1.

In view of the above, the present invention has been proposed in view of the above-described circumstances, and the moving image object is deformed when the moving object overlaps with the still object in the moving image. It is another object of the present invention to provide a motion detection device and method capable of accurately detecting a motion vector of a moving image object.

[0010]

A motion detecting apparatus according to the present invention for solving the above-mentioned problems comprises a first image constituting a moving image, and a second image which is temporally located behind the first image. In a motion detection device for detecting the motion of a moving image object included in a third image positioned temporally before the image and the first image, a moving image object included in the first image is converted to a second image. A forward motion detecting means for obtaining a forward matching distance by matching a moving object included in the first image and a moving object included in the third image; Comparing the forward direction matching distance and the backward direction matching distance with the backward direction detecting unit that calculates the backward direction matching distance by matching the image object and detects the backward direction motion vector Motion vector detecting means for detecting the forward motion vector or the backward motion vector as a motion vector of a moving image object included in the first image based on the comparison result. It is.

[0011] Such a motion estimating device converts a motion vector of a moving image object included in a first image into a forward motion vector or a backward motion vector according to a forward matching distance and a backward matching distance. You can choose.

Further, the motion detection method according to the present invention is characterized in that a first image constituting a moving image, a second image temporally behind the first image and a first image are temporally shifted. In the motion detection method for detecting the motion of the moving image object included in the third image located before, the moving image object included in the first image is matched with the moving image object included in the second image. A forward matching distance is obtained, a forward motion vector is detected,
The moving image object included in the first image is matched with the moving image object included in the third image to obtain a backward matching distance, a backward motion vector is detected, and a forward matching distance and a backward matching distance are calculated. , Based on the comparison result of the comparison, the forward motion vector or the backward motion vector is detected as the motion vector of the moving image object included in the first image.

According to such a motion detection method, a motion vector of a moving image object included in a first image is converted into a forward motion vector or a backward motion vector according to a forward matching distance and a backward matching distance. You can choose.

[0014]

Embodiments of the present invention will be described below in detail with reference to the drawings.

The present invention is applied to, for example, an image processing apparatus 1 configured as shown in FIG. This image processing device 1
Is a recording medium 2 storing a motion detection program for detecting a motion vector of a moving image object included in a moving image described below, and a CPU 3 which performs processing according to the motion detection program stored in the recording medium 2.
And

The CPU 3 temporarily stores the moving image data in the image memory 5, for example, in response to the input of the moving image data from the outside via the interface (I / F) circuit 4, and stores the moving image data in the recording medium 2. By performing a process of reading and executing the motion detection program, a process of detecting a motion vector of a moving image object included in a plurality of frames included in the input moving image data is performed.

In the image processing apparatus 1,
By displaying the input moving image data, the motion vector of the moving image object, and the like on the display monitor 6, for example, the processing result is presented to the user.

When performing processing for detecting a motion vector for a moving image object included in a plurality of frames, the CPU 3 activates a motion detecting program stored in the recording medium 2 and simultaneously executes a moving image to be processed. The frames constituting the data are read from the image memory 5. At this time, when the CPU 3 performs the process of detecting the motion vector for the frame t, for example, FIG.
As shown in (1), a process of reading a frame t-1, frame t, and frame t + 1 constituting a moving image including a still object A having no motion and a moving image object B having motion is performed. The moving image object B shown in FIG. 2 is the background of the still object A, and the moving image including the moving object B is added to the still object A as time t-1 to time t and time t + 1. Occlusion occurs in which the amount of overlap increases and the shape of the moving image object B changes.

As shown in FIG. 3, the moving image data read from the image memory 5 includes a plurality of frames arranged in a time series such as a frame t-1, a frame t, and a frame t + 1. The area division processing is performed by the CPU 3 every time. Thereby, each frame is divided into a plurality of regions according to the pixel values. The details of the processing performed by the CPU 3 in the above-described area division processing will be described later.

In the next step ST2, in the moving image data subjected to the area division processing, a motion vector is detected for the frame t using the frame t-1 and the frame t, and a motion vector is detected using the frame t and the frame t + 1. Thus, the motion vector for the frame t is detected by the CPU 3. In the following description, the frame t-
The motion vector for frame t detected using frame 1 and frame t is referred to as a “forward motion vector”, and the motion vector for frame t detected using frame t and frame t + 1 is referred to as “reverse motion vector”. Call.

In the next step ST3, step S
Using the forward motion vector and the backward motion vector detected at T2, the CPU 3 detects a motion vector for the frame t. This step S
The details of the processing performed by the CPU 3 at T3 will be described later.

Next, the area dividing process in step ST1 will be described. The CPU 3 performs, for example, a process according to a flowchart shown in FIG. 4 to perform a region division process on each of the frames t-1, t, and t + 1 read into the image memory 5. In the following description of the area division processing, the frame t-1, the frame t, and the frame t
+1 is simply called a "frame".

When performing the area dividing process, the CPU
3 divides a frame into a plurality of areas by executing, for example, an area dividing program stored in the recording medium 2.
At this time, the CPU 3 compares the arbitrarily selected reference region with a reference region included in a search range set around the reference region, and merges the reference region and the reference region according to the comparison result. Thus, an area is generated, and the input frame is divided into a plurality of areas. Thereby, the CPU 3
For example, a plurality of frames included in a frame are divided by classifying the frame into a frame indicating a portion having uniform pixel values, a region indicating a portion (texture) in which pixel values change periodically, and dividing the frame. Creates an area representing the object.

According to the flowchart shown in FIG. 4, first, in step S11, the CPU 3 sets the number of pixels constituting the search range S and shows the relationship between the reference area and the reference area in the pixel pattern. Initial setting is performed so that the pixel threshold value T is close to zero. Also, C
PU3 sets a minimum area value MSA indicating a minimum area when the area is set as a region. This initial setting may be performed based on, for example, an instruction from the user, or may be performed according to a region division program.

In the next step ST12, the CPU 3
Performs a process of generating a plurality of regions constituting the input frame using the pixel threshold value T, the search range S, and the minimum area value MSA set in step ST11 described above. The details of the process in step ST12 will be described later.

In step ST13, the CPU 3
By performing the process shown in step ST12, an area is generated for the entire frame, and it is determined whether a frame divided into a plurality of areas has been generated. Then, when CPU 3 determines that an area has been generated for the entire frame, it ends the process, and when it determines that an area has not been generated for the entire frame, it proceeds to step ST14.

In step ST14, the CPU 3
A process for changing the pixel threshold value T and the search range S and a process for returning to step ST12 are performed. That is, the CPU 3
Generates an area in a part of the frame that matches the pixel threshold T and the search range S by performing processing on the entire frame with the set pixel threshold T and the search range S, and generates the pixel newly set in step ST14. Threshold T
Further, by generating an area in the search range S, a frame is generated as a result of which the entire area is divided by the area. Further, in this step ST14, the CPU 3 changes the above-described pixel threshold T and the search range S,
A process for changing the minimum area value MSA may be performed.

Next, the processing shown in step ST12 will be described with reference to the flowchart shown in FIG.

First, in step ST21, the CPU
3, initialization is performed such that the search range S is, for example, 3 × 3 pixels shown in FIG. 6 and the pixel threshold value T indicating the pixel pattern relationship between the reference region and the reference region is a positive value close to zero. . Further, the CPU 3 sets a minimum area value MSA indicating a minimum area as a region. This initial setting may be performed based on, for example, an instruction from a user, or may be performed according to a region division program.

In the next step ST22, the CPU 3
Performs a process of designating a region including a plurality of pixels included in a frame as the reference region P. Where CP
U3 designates the i = 1st reference region from the reference region P which is composed of i = 1, 2,..., N constituting the frame. The CPU 3 sets the initial value of the area to A = P.
That is, the CPU 3 performs a process of making the area of the area A the same as the area of the reference area P. The CPU 3 sets the search range S around the reference region P and having a larger number of pixels than the reference region P, and sets the search range S around the reference region P in step S23 as the reference region. A process for designating the included minute area Q is performed. The minute area Q is composed of j = 1, 2,.
The search area P is composed of the individual pieces.

In the next step ST24, the CPU 3
Is, as shown in the following equation 1, the pixel value of the reference area P designated in step ST22 and the pixel value of step S22.
A process of comparing the absolute value of the difference between the pixel value of the minute area Q designated in T23 and the pixel threshold T set in step ST21 described above is performed. | P−Q | <T (Equation 1) Here, Expression 1 indicates that the relationship between the minute area Q and the reference area P is within a range that allows the pixel threshold T to be substantially the same pixel pattern. Is shown. Then, the CPU 3
When it is determined that the absolute value of the difference is smaller than the pixel threshold T, the process proceeds to step ST25. When it is determined that the absolute value of the difference is not smaller than the pixel threshold T, the process proceeds to step ST26.

In step ST25, the CPU 3
A process of merging the minute area Q with the reference area P is performed. As described above, the CPU 3 determines that the absolute value of the difference between the pixel value of the reference area P and the pixel value of the minute area Q is equal to or less than the pixel threshold T in the above-described step ST24, that is, the reference area P and the minute area Q Is smaller than or equal to the pixel threshold T, it is determined that the reference area P and the minute area Q have substantially the same pixel pattern, and the area of the area is increased by merging the reference area P and the minute area Q. Perform processing.

In the next step ST26, the next minute area is designated by incrementing the minute area Q designated in the above-mentioned step ST23, and step S23 is executed.
Proceed to T27.

In the next step ST27, whether or not the processing from the above-mentioned steps ST24 to ST26 has been performed over the entire search range S (j = m) using the minute area Q specified in the above-mentioned step ST23 Determine whether or not. When the CPU 3 determines that the processing from step ST24 to step ST26 has been performed over the entire search range S, the process proceeds to step ST28, and performs the processing from step ST24 to step ST26 over the entire search range S. When it is determined that the search has not been performed, the process returns to step ST24, and steps ST24 to ST2 are performed until the processing is performed over the entire search range S.
The processing up to 6 is repeated.

In step ST28, the CPU 3
The area of the region generated by repeating steps ST24 to ST27 described above and step ST21
Compare the size with the minimum area value MSA set in the above. When the area of the region is smaller than the minimum area value MSA, the CPU 3 proceeds to step ST29, and when the area of the region is not smaller than the minimum area value MSA, the CPU 3 proceeds to step ST30.

In step ST29, the CPU 3
At step ST26, the area generated by performing the processing from step ST24 to step ST27 is not recognized as an area constituting a frame, and the process proceeds to step ST31.

At step ST30, the CPU 3
In step ST28, a process of recognizing a region generated by performing the processes in steps ST24 to ST27 as a region forming a frame is performed, and the process proceeds to step ST31.

In the next step ST31, the CPU 3
Performs a process of proceeding from the reference region targeted for the above-described process to the next reference region.

In the next step ST32, the CPU 3
Determines whether or not the process of generating an area for the entire frame using the pixel threshold T and the search range S set in the above-described step ST21 in accordance with the advance of the processing target in the above-mentioned step ST31 to decide. Then, when CPU 3 determines that the processing for generating the area has been performed for the entire frame, the processing ends, and when it determines that the processing for generating the area has not been performed for the entire frame, the CPU 3 returns to step ST23 and returns to steps ST23 to ST23. The input image is divided into a plurality of regions by repeatedly performing the process shown in step ST31.

As shown in FIG. 6, the CPU 3 performing such processing in step ST12 sets the search range S to 3 × 3 pixels in step ST11 or step ST14 for the portion 10 having uniform pixel values in the frame. 5 is performed, and the process shown in the flowchart shown in FIG. 5 is performed on the portion having a textured pixel value in step ST14 by setting the search range S to 5 × 5 pixels in step ST14. Perform division processing. Thereby, the CPU 3
Steps ST24 to S using the pixel threshold T
By performing the process of T27, the process of merging the minute region Q within the search range S indicated by the star in FIG. 6 and the reference region P indicated by the circle is performed, and even if it is a texture region, a single region is obtained. Set. That is, by performing the above-described region division processing, the region can be divided according to the pattern of the object included in the frame, and for example, the region constituting the object included in the frame is set.

More specifically, as shown in FIG. 2, the CPU 3 divides each frame including the still object A and the moving image object B into a frame including the region A and the region B.

Next, processing for determining a motion vector for a frame t, which corresponds to the above-described steps ST2 and ST3, will be described with reference to the flowchart of FIG. When the processing for determining the motion vector shown in FIG. 7 is performed, the processing described below is performed by reading the frame t-1, the frame t, and the frame t + 1 into the image memory 5 and reading and executing the motion detection program. .

According to the flowchart shown in FIG.
First, in step ST41, the CPU 3 performs matching with the region constituting the moving image object B included in the frame t-1 using the region constituting the moving image object B included in the frame t as a template. At this time, the CPU 3 determines the position where the overlapping portion is the largest in the vicinity of the region constituting the moving image object B included in the frame t-1 as the matching distance MDt-1.
Then, a forward motion vector Vt-1 for the frame t is obtained using the matching distance MDt-1.

In step ST42, the CPU 3
Using the region constituting the moving image object B included in the frame t as a template, matching with the region constituting the moving image object B included in the frame t + 1 is performed. At this time, the CPU 3 sets the template to the frame t
+1 is the largest overlapping part near the region constituting the moving image object B and the position having the smallest distance is the matching distance MDt + 1, and the backward movement of the frame t is performed using the matching distance MDt + 1. Vector Vt + 1
Ask for.

In step ST43, the CPU 3
The matching distance MDt− obtained in step ST41 described above.
1 is compared with the matching distance MDt + 1 obtained in step ST42, and the matching distance MDt-1 is determined as the matching distance M
It is determined whether it is greater than Dt + 1. And CPU3
Proceeds to step ST44 when it is determined that the matching distance MDt-1 is larger than the matching distance MDt + 1,
When it is determined that the matching distance MDt-1 is not larger than the matching distance MDt + 1, the process proceeds to step ST45.

In step ST44, the CPU 3
The motion vector for the frame t is defined as a forward motion vector Vt-1.

In step ST45, the CPU 3
The motion vector for the frame t is defined as a backward motion vector Vt + 1.

More specifically, as shown in FIG. 8, for example, the moving image object B moves toward the still object A and moves toward the frame t-1, frame t, and frame t + 1. When the motion vector of the moving image object B included in the frame t is detected when the area of the moving image object B gradually decreases, the region configuring the moving image object B included in the frame t is used as a template, and in step ST41, the frame t− 1 by performing matching with the region forming the moving image object B included in the moving image object B.
1 and the motion vector Vt-1 are obtained, and in step ST42, matching is performed with the region constituting the moving image object B included in the frame t + 1 to obtain a matching distance MDt +
1 and the backward motion vector Vt + 1 are obtained. Then, in step ST43, it is determined that the matching distance MDt-1 is smaller than the matching distance MDt + 1, and the forward motion vector of the moving image object B included in the frame t is set to Vt-1.

Further, as shown in FIG.
When the moving image object B moves toward the still object A and progresses to the frame t and the frame t + 1, and the area of the moving image object B gradually decreases, the moving image object B included in the frame t is reduced.
When detecting a motion vector for, the region constituting the moving image object B included in the frame t is used as a template, and in step ST41, matching is performed with the region constituting the moving image object B included in the frame t-1. The matching distance MDt-1 and the forward motion vector Vt-1 are obtained, and the frame t +
The matching distance MDt + 1 and the backward motion vector Vt + 1 are obtained by performing matching with the region constituting the moving image object B included in No. 1. And step ST
At 43, the matching distance MDt + 1 is the matching distance MDt-1
Is determined to be smaller than Vt +
Set to 1.

The CPU 3 which performs the processing for determining the motion vector for the frame t shown in FIG. 7 has the matching distance MDt-1 obtained using the frame t-1 and the matching distance MDt obtained using the frame t + 1. +1
, And the matching distance MDt-1 and the matching distance MDt
+1 and either the forward motion vector or the backward motion vector using the smaller matching distance in frame t
Is a motion vector of an area constituting the moving image object B included in the moving image object B, so that the moving image object B is deformed so as to overlap the still object A as the frame t-1, frame t, and frame t-1 progress. Even if occlusion occurs, it is possible to determine a motion vector for each area constituting the moving image object B according to the matching distance, and to calculate a motion vector of an area near the boundary between the still object A and the moving image object B. It can be detected accurately.

Next, another process for determining a motion vector for the frame t, which corresponds to the above-described steps ST2 and ST3, will be described with reference to the flowcharts of FIGS.

At step ST51, the CPU 3
The same processing as the processing shown in step ST41 is performed. That is, the CPU 3 uses the region constituting the moving image object B included in the frame t as a template,
Matching is performed with the region constituting the moving image object B included in the frame t-1. At this time, CPU3
Is the distance when the overlapping portion is the largest in the vicinity of the region constituting the moving image object B included in the frame t-1 as the matching distance MDt-1, and the matching distance MDt-1 is used for the frame t. Is determined.

At step ST52, the CPU 3
A process similar to the process shown in step ST42 is performed. That is, the CPU 3 uses the region constituting the moving image object B included in the frame t as a template,
Matching is performed with the region constituting the moving image object B included in the frame t + 1. At this time, CPU3
Is the distance when the overlapping portion is the largest in the vicinity of the region constituting the moving image object B included in the frame t + 1 as the matching distance MDt + 1, and the motion for the frame t is calculated using the matching distance MDt + 1. Find the vector Vt + 1.

In step ST53, the CPU 3
The matching distance MDt− obtained in step ST51 described above.
It is determined whether 1 is smaller than a threshold value T1 and
It is determined whether or not the matching distance MDt + 1 normalized by the area of the template determined in step ST52 is smaller than a threshold. In the following description, it is assumed that the matching distance MD to be compared with the threshold is normalized by the area of the template. If the CPU 3 determines that the matching distance MDt-1 is smaller than the threshold value T1 and the matching distance MDt + 1 is smaller than the threshold value, the process proceeds to step ST54, where the matching distance MDt-1 is determined.
Is smaller than the threshold value T1, and the matching distance MDt +
If it is determined that the condition that 1 is smaller than the threshold is not satisfied, the process proceeds to step ST58 shown in FIG.

In step ST54, the CPU 3
A confidence mark is added to the motion vector Vt-1 obtained in step ST51 and the motion vector Vt + 1 obtained in step ST52. That is, the CPU 3 executes the above-described step S
For the motion vector for which the matching distance is determined to be equal to or smaller than the threshold value T1 in T53, a reliability mark is added as a flag indicating a motion vector with high reliability.

In step ST55, the CPU 3
The value of the matching distance MDt-1 is compared with the value of the matching distance MDt + 1 to determine whether the matching distance MDt-1 is greater than the matching distance MDt + 1. And CP
U3 indicates that the matching distance MDt-1 is equal to the matching distance MDt.
When it is determined that it is greater than +1, the process proceeds to step ST56, and when it is determined that the matching distance MDt-1 is not greater than the matching distance MDt + 1, the process proceeds to step ST57.

In step ST56, the CPU 3
The motion vector for frame t is represented by motion vector Vt +
Set to 1.

In step ST57, the CPU 3
The motion vector for frame t is calculated as motion vector Vt-
Set to 1.

More specifically, steps ST51 to ST
A series of processes of 57 will be described. For example, as shown in FIG. 12, a frame t-1, a frame t, a frame t +
When the area of the moving image object B having the motion vector expressed by the motion vector VRt-1 gradually decreases toward the still object A as the moving image object B progresses to 1, the moving image object B
Is small, assuming that the areas of the regions constituting the moving image object B are Bt-1, Bt, and Bt + 1, respectively, these Bt-1, Bt, and Bt + 1 have substantially the same area.

When detecting a motion vector of the moving image object B included in the frame t in FIG. 12, the region constituting the moving image object B included in the frame t is used as a template, and in step ST51, the matching distance MDt-1 and the matching distance MDt-1 are determined. A motion vector Vt-1 is obtained, and step S
At T52, the matching distance MDt + 1 and the motion vector Vt + 1
Ask for. When it is determined in step ST53 that the matching distance MDt-1 is smaller than the matching distance MDt + 1, the motion vector of the moving image object B included in the frame t is set to Vt-1, and the matching distance MDt-1 is set to match. When it is determined that the distance is not smaller than the distance MDt + 1, the motion vector of the moving image object B included in the frame t is set to Vt + 1.

When it is determined in step ST53 that the condition that the matching distance MDt-1 is smaller than the threshold value T1 and the condition that the matching distance MDt + 1 is smaller than the threshold value is not satisfied, in step ST58 shown in FIG. The CPU 3 compares the value of the matching distance MDt−1 with the value of the matching distance MDt + 1, and determines the matching distance MDt.
It is determined whether -1 is greater than the matching distance MDt + 1. When the CPU 3 determines that the matching distance MDt-1 is larger than the matching distance MDt + 1, the process proceeds to step ST65, and when the CPU 3 determines that the matching distance MDt-1 is not larger than the matching distance MDt + 1, the process proceeds to step ST59.

In step ST59, the CPU 3 sets
It is determined whether or not a confidence mark has been added to the motion vector of the area of the frame t-1 corresponding to the area used as the template in steps ST51 and ST52 for which the motion vector is to be detected.
Then, when the CPU 3 determines that the confidence mark has not been added to the motion vector in the area of the frame t-1, the process returns to step ST55 shown in FIG.
55 to perform the processing shown in step ST57 to obtain the frame t
Is determined, and when it is determined that the confidence mark has been added to the motion vector in the region of the frame t-1, the process proceeds to step ST60.

In step ST60, the CPU 3
Using the area having the motion vector VRt-1 determined to have the confidence mark added in step ST59 described above, matching is performed using the area used as the template to obtain a matching distance MDt-1.

In step ST61, the CPU 3 sets
The matching distance MDt-1 obtained in step ST60 is compared with the threshold value T2, and the matching distance MDt-1 is determined.
Is determined to be smaller than the threshold value T2, step S
Proceeding to T62, if it is determined that the matching distance MDt-1 is not smaller than the threshold value T2, the process proceeds to step ST63. Here, the threshold T2 has a value range wider than the threshold T1 for the value of the matching distance.

At step ST63, the CPU 3
The vicinity of the region having the motion vector VRt-1 for the moving image object B of the frame t-1 is searched for, and the matching distance MDt- with the template constituting the frame t is searched.
An area that satisfies that 1 is smaller than the threshold value T2 is detected. Then, the CPU 3 obtains the motion vector VRNt-1 using an area satisfying that the matching distance MDt-1 is smaller than the threshold value T2, and sets the motion vector VRNt-1 as a motion vector of a template constituting the frame t in step ST64. move on.

In step ST62, the CPU 3
The motion vector for frame t is represented by motion vector VR.
The process proceeds to step ST64 as t-1.

In step ST64, the CPU 3
A confidence mark is added to the motion vector VRNt-1 obtained in step ST63 and the motion vector VRt-1 in step ST62. That is, the CPU 3 adds a confidence mark as a flag indicating a motion vector having a high degree of reliability to the motion vector for which the matching distance is determined to be equal to or smaller than the threshold value T2 in step ST61 or ST63, and ends the process.

A series of processing in steps ST58 to ST64 is performed, for example, as shown in FIG.
When the moving image object B has a movement expressed by the motion vector VRt-1 toward the still object A as the moving image object B progresses to 1, frame t, and frame t + 1, the moving image object B gradually increases in area. Matching distance MDt-
1, when MDt + 1 is equal to or greater than the threshold value T1, and the area of the region constituting the moving image object B is Bt−
Assuming that 1, Bt and Bt + 1, the processing is performed when the relation of Bt-1>Bt> Bt + 1 is satisfied.

When detecting a motion vector for the moving image object B included in the frame t in FIG. 13, the region constituting the moving image object B included in the frame t is used as a template, and in step ST59, the frame t is detected.
-1 to determine whether a confidence mark is added to the motion vector of the area constituting the moving image object B,
When the confidence mark is not added, the motion vector for the area constituting the moving image object B included in the frame t is determined by the above-described processing from step ST55 to step ST57.

In step ST59, frame t-1
When the confidence mark is added to the motion vector of the region constituting the moving image object B included in the moving image object B, the motion vector VRt- of the frame t-1 to which the confidence mark is added is added.
1 or a motion vector VRNt-1 in the vicinity of VRt + 1, and a motion vector when the matching distance MDt-1 is equal to or less than the threshold value T2.
Is the motion vector for the region constituting.

On the other hand, when it is determined in step ST58 that the matching distance MDt-1 is larger than the matching distance MDt + 1, in step ST65, the CPU 3 is a target for detecting a motion vector.
It is determined whether or not a confidence mark is added to the motion vector of the area of the frame t + 1 corresponding to the area used as the template in 51 and step ST52. Then, when the CPU 3 determines that the confidence mark has not been added to the motion vector in the area of the frame t + 1, the CPU 3 returns to step ST55 shown in FIG.
The motion vector for the frame t is determined by performing the processing shown in steps ST57 to ST57. When it is determined that the confidence mark has been added to the motion vector in the area of the frame t + 1, the process proceeds to step ST66.

At step ST66, the CPU 3
Using the region having the motion vector VRt + 1 determined to have the confidence mark added in the above-described step ST65, matching is performed using the region used as the template to obtain a matching distance MDt + 1.

At step ST67, the CPU 3
The matching distance MDt + 1 obtained in step ST66 described above is compared with the threshold value T2, and the matching distance MDt + 1 is determined.
Is determined to be smaller than the threshold value T2, step S
Proceeding to T68, if it is determined that the matching distance MDt + 1 is not smaller than the threshold value T2, the process proceeds to step ST69.

At step ST69, the CPU 3
The vicinity of the area having the motion vector VRt + 1 for the moving image object B of the frame t + 1 is searched for, and the matching distance MDt + with the template constituting the frame t is searched.
An area that satisfies that 1 is smaller than the threshold value T2 is detected. Then, the CPU 3 obtains a motion vector VRNt + 1 using an area satisfying that the matching distance MDt + 1 is smaller than the threshold value T2, and sets the motion vector VRNt + 1 as a motion vector of a template constituting the frame t in step ST70. move on.

At step ST68, the CPU 3
The motion vector for frame t is represented by motion vector VR.
The process proceeds to step ST70 as t + 1.

In step ST70, the CPU 3
A confidence mark is added to the motion vector VRNt + 1 obtained in step ST69 and the motion vector VRt + 1 in step ST68. That is, the CPU 3 adds a confidence mark as a flag indicating a motion vector with high reliability to the motion vector for which the matching distance is determined to be equal to or smaller than the threshold value T2 in the above-described step ST67 or ST69, and ends the process.

A series of processes in steps ST65 to ST70 is performed, for example, as shown in FIG.
1. When the moving image object B has a movement represented by the motion vector VRt + 1 toward the still object A as the frame progresses to the frame t and the frame t + 1, and the area of the moving image object B gradually increases, Matching distance MDt-
1, when MDt + 1 is equal to or greater than the threshold value T1, and the area of the region constituting the moving image object B is Bt−
Assuming that 1, Bt, Bt + 1, the processing is performed when the relation of Bt + 1>Bt> Bt-1 is satisfied.

When detecting the motion vector of the moving image object B included in the frame t in FIG. 14, the region constituting the moving image object B included in the frame t is used as a template, and in step ST65, the frame t is detected.
It is determined whether a confidence mark has been added to the motion vector of the area included in +1 and constituting the moving image object B,
When the confidence mark is not added, the motion vector for the area constituting the moving image object B included in the frame t is determined by the above-described processing from step ST55 to step ST57.

In step ST65, frame t + 1
When the confidence mark is added to the motion vector of the region constituting the moving image object B included in the motion vector object B, the motion vector VRt + of the frame t + 1 to which the confidence mark is added
1 or a motion vector VRNt + 1 in the vicinity of VRt + 1, and a motion vector when the matching distance MDt + 1 is equal to or smaller than the threshold T2 using the motion vector VRNt + 1.
Is the motion vector for the region constituting.

When the matching distance MDt-1 and the matching distance MDt + 1 are equal to or less than the threshold value T1, the CPU 3 performing the processing for determining such a motion vector performs the determination shown in the above-mentioned step ST55 and includes the frame t. The motion vector for the area constituting the moving image object B to be moved is set to the forward motion vector Vt-1 or the backward motion vector Vt + 1. Even if the moving image object B overlaps with the still object A and deforms to generate occlusion, it is possible to determine a motion vector for each region constituting the moving image object B according to the matching distance. Can accurately detect the motion vector in the area near the boundary between the object and the moving image object B. That.

The CPU 3 calculates the matching distance M
Dt-1 and the matching distance MDt + 1 are equal to the threshold T1 or the threshold T
Since a motion vector to which a confidence mark indicating that the distance is equal to or less than 2 is used, and a forward motion vector or a backward motion vector for a region with a matching distance equal to or less than the threshold T2 is used as a motion vector for the frame t, the frame t Even when the moving image object B overlaps with the still object A and deforms as advancing to −1, frame t, and frame t−1, large occlusion occurs, each of the moving image objects B constituting the moving image object B according to the matching distance. A motion vector for the region can be determined,
A motion vector in an area near the boundary between the still object A and the moving image object B can be accurately detected.

The image processing apparatus 1 provided with the CPU 3 for performing such processing performs the above-described processing in step ST41 or ST51 on the frame t as shown in FIG. A motion vector is obtained, and an image composed of forward motion vectors as shown in FIG. 16 is generated. In the frame t shown in FIG. 15, the background mountain and the water surface correspond to the stationary object A, and the helicopter and the boat correspond to the moving image object B. It can be seen from FIG. 16 that an incorrect motion vector is detected due to occlusion near the boundary between the background and the helicopter or boat.

The image processing apparatus 1 performs the above-described step ST42 or step ST52 on the frame t constituting the moving image in the frame t as shown in FIG.
By performing the processing shown in (1), a backward motion vector is obtained, and an image composed of the motion vectors as shown in FIG. 17 is generated. It can be seen from FIG. 17 that an incorrect motion vector is detected due to occlusion near the boundary between the background and the helicopter or boat.

The image processing apparatus 1 performs the processing shown in steps ST43 to ST45 shown in FIG. 7 or the processing shown in steps ST53 to ST70 shown in FIGS. Using the motion vector, an image including the motion vector as shown in FIG. 18 is generated. From FIG. 18, it can be seen that an accurate motion vector is detected near the boundary between the background and the helicopter or boat even if it overlaps with the background.

On the other hand, as shown in FIG. 19, the motion vector of the frame t determined by the block matching is such that an occlusion occurs near the boundary between the background and the moving image object B and an incorrect motion vector is detected. You can see that it is done.

[0086]

As described above in detail, the motion detecting apparatus and method according to the present invention provide a first image constituting a moving image and a second image temporally located after the first image. In a motion detection method for detecting a motion of a moving image object included in a third image temporally preceding the image and the first image, a moving image object included in the first image is converted to a second image. And the forward motion vector is detected by obtaining a forward matching distance by matching the moving image object included in the first image, and the moving image object included in the first image is matched with the moving image object included in the third image. The backward matching distance is calculated to detect the backward motion vector, and based on the comparison result of comparing the forward matching distance and the backward matching distance, the forward In this case, the motion vector of the moving image object included in the first image is calculated as the motion vector of the moving image object included in the first image. Depending on the direction matching distance, a forward motion vector or a backward motion vector can be selected. Therefore, according to the motion detection device and method,
Even if the moving image object is deformed by the moving image object having a motion and overlapping the still object in the moving image, a forward motion vector or a backward motion vector obtained by a small matching distance is selected and a motion vector is selected. Can be determined, and the motion vector of the moving image object can be accurately detected.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus to which the present invention has been applied.

FIG. 2 is a diagram for explaining a relationship between a still object A and a moving image object B;

FIG. 3 is a diagram for describing processing performed on moving image data.

FIG. 4 is a flowchart illustrating a processing procedure according to a region division program performed for each of a frame t and a frame t + 1.

FIG. 5 is a flowchart illustrating a process of generating a region with a specified pixel threshold and a search range.

FIG. 6 is a diagram for explaining that an area dividing process is performed by changing a search range.

FIG. 7 is a flowchart illustrating an example of a process of determining a motion vector for a frame t by a CPU of an image processing apparatus to which the present invention has been applied.

FIG. 8 shows a frame t-1, a frame t, and a frame t + 1.
FIG. 9 is a diagram for explaining an example of a relationship between a still object A and a moving image object B that changes as the process proceeds to FIG.

FIG. 9 shows a frame t-1, a frame t, and a frame t + 1.
FIG. 10 is a diagram for explaining another example of the relationship between the still object A and the moving image object B that changes as the process proceeds to FIG.

FIG. 10 is a flowchart illustrating another process of determining a motion vector for a frame t by a CPU of an image processing apparatus to which the present invention has been applied.

FIG. 11 is a flowchart illustrating another process of determining a motion vector for a frame t by the CPU of the image processing apparatus to which the present invention has been applied.

FIG. 12 shows a frame t-1, a frame t, and a frame t +
FIG. 11 is a diagram for explaining another example of the relationship between the still object A and the moving image object B that changes as the process proceeds to 1.

FIG. 13 shows a frame t-1, a frame t, and a frame t +
FIG. 11 is a diagram for explaining another example of the relationship between the still object A and the moving image object B that changes as the process proceeds to 1.

FIG. 14 shows a frame t-1, a frame t, and a frame t +
FIG. 11 is a diagram for explaining another example of the relationship between the still object A and the moving image object B that changes as the process proceeds to 1.

FIG. 15 is a diagram showing a frame t constituting a moving image.

FIG. 16 is a diagram showing an image composed of forward motion vectors.

FIG. 17 is a diagram showing an image composed of a backward motion vector.

FIG. 18 is a diagram showing an image composed of motion vectors generated by an image processing device to which the present invention has been applied.

FIG. 19 is a diagram showing an image composed of motion vectors for a frame t determined by block matching.

FIG. 20 is a diagram for explaining that occlusion occurs.

[Explanation of symbols]

 1 image processing device, 3 CPU

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Keisuke Fuse 2-31-19 Shiba, Minato-ku, Tokyo F-term in the Communications and Broadcasting Corporation (reference) 5C059 KK00 MA05 MB04 NN01 NN08 NN11 NN28 PP04 UA31 5L096 DA02 FA41 FA59 FA66 GA07 GA19 GA34 GA51 HA04 JA09 9A001 HH30

Claims (10)

[Claims] 1. A first image constituting a moving image, a second image temporally located after the first image, and a third image temporally located before the first image. A motion detection device that detects a motion of a moving image object included in the first image; a moving image object included in the first image is matched with a moving image object included in the second image to obtain a forward matching distance; A forward motion detecting means for detecting a motion vector, and a backward moving distance is obtained by matching a moving image object included in the first image to a moving image object included in the third image to detect a backward motion vector. A forward motion vector or a backward motion vector based on a comparison result of comparing the forward matching distance and the backward matching distance. A motion vector detecting means for detecting a vector as a motion vector of a moving image object included in the first image. 2. The image processing apparatus according to claim 1, further comprising: an area dividing unit that divides the first image, the second image, and the third image into a plurality of areas based on an amount of change in a pixel value. The area generated by the area dividing means which constitutes the moving image object included in the first image is matched with the area generated by the area dividing means which constitutes the moving image object included in the second image. Calculating a direction matching distance to detect a forward motion vector; the reverse direction motion detecting means forming a moving image object included in the first image and generating an area generated by the area dividing means into a third image; And a reverse motion vector is detected by calculating a reverse matching distance by matching a moving image object included in the Motion detection apparatus according to claim 1. 3. Comparing a first threshold value set in advance with values of the forward matching distance and the backward matching distance,
The forward matching distance and the backward matching distance are the first
A confidence mark indicating that the reliability of the motion vector is high, the confidence mark indicating that the reliability of the motion vector is high, is added to the motion vector of the region constituting the moving image object included in the first image. The forward motion detecting means or the backward motion detecting means may define a region having a motion vector to which a trust mark has been added by the trust mark adding means and constituting a moving image object included in the second image or the third image. 3. The motion detecting apparatus according to claim 2, wherein a matching distance is determined by performing matching with a region included in the first image and constituting a moving image object. 4. The motion vector detecting means based on a comparison result of comparing a matching distance obtained using a region having a motion vector to which a confidence mark has been added by the confidence mark adding means with a second threshold value. 4. The motion detecting device according to claim 3, wherein the forward motion vector or the backward motion vector is detected as a motion vector of a moving image object included in the first image. 5. The motion vector detecting means compares a matching distance obtained by using a region having a motion vector to which a trust mark has been added by the trust mark adding means with a second threshold value, and determines that the matching distance is When it is smaller than the second threshold value, the motion vector included in the second image or the third image and to which the confidence mark is added is set as a motion vector of a region included in the first image and constituting a moving image object, When the matching distance is not smaller than the second threshold, a motion vector in an area where the matching distance is smaller than the threshold and located near the motion vector included in the second image or the third image to which the confidence mark is added is set. 5. The motion detection device according to claim 4, wherein the motion vector is a motion vector of a region included in the first image and constituting a moving image object. 6. A first image constituting a moving image, a second image temporally located after the first image, and a third image temporally located before the first image. In the motion detection method for detecting the motion of a moving image object included in a moving image object, a moving image object included in a first image is matched with a moving image object included in a second image to determine a forward matching distance, Detecting a motion vector and matching a moving image object included in the first image with a moving image object included in the third image to obtain a backward matching distance; detecting a backward motion vector; A moving image including the forward motion vector or the backward motion vector in the first image based on a comparison result of comparing the distance and the backward matching distance; Motion detecting method characterized by detecting such a motion vector of the object. 7. The first image, the second image, and the third image are divided into a plurality of regions based on a change amount of a pixel value, and regions forming a moving image object included in the first image To a region constituting a moving image object included in the second image, a forward matching distance is obtained to detect a forward motion vector, and a region constituting the moving image object included in the first image. 7. The motion detection method according to claim 6, further comprising: matching a region of the moving image object included in the third image with a region of the moving image object, calculating a backward matching distance, and detecting a backward motion vector. 8. Comparing a first threshold value set in advance with the values of the forward matching distance and the backward matching distance,
The forward matching distance and the backward matching distance are the first
Is smaller than the threshold value, a confidence mark indicating that the reliability of the motion vector is high is added to the motion vector of the region constituting the moving image object included in the first image, and the motion with the confidence mark is added. Using a region having a vector and constituting a moving image object included in the second image or the third image, matching is performed with a region constituting the moving image object included in the first image to obtain a matching distance. 8. The motion detection method according to claim 7, wherein: 9. A forward motion vector or a backward motion vector based on a comparison result obtained by comparing a matching distance obtained by using a region having a motion vector to which a confidence mark is added and a second threshold value, with a first threshold value. 9. The motion detection method according to claim 8, wherein the motion vector is detected as a motion vector of a moving image object included in the image. 10. A matching distance obtained by using an area having a motion vector to which a confidence mark is added and a second
When the matching distance is smaller than the second threshold value, the motion vector included in the second image or the third image to which the confidence mark has been added is included in the first image. When the matching distance is not smaller than the second threshold, the matching distance is set near the motion vector included in the second image or the third image to which the confidence mark is added, and the matching distance The motion detection method according to claim 9, wherein a motion vector in a region where is smaller than a threshold value is set as a motion vector of a region included in the first image and constituting a moving image object.