JPH0944681A - Moving object area detector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract a stable moving object area, regardless of the size of a motion vector, and to generate a background image which is not included in the moving object area, in a scene in which a moving object is tracked by a camera. SOLUTION: The motion vector of an input image generated accompanying the movement of a camera is detected, a field space to be processed according to the size of the motion vector in a field space calculation part 1b, the generation of a background image is performed by using a motion area, a motion vector, an input image and a background image in a background image generation part 1c, the motion area is detected from the motion vector, the input image and the background image in a motion area detection part 1d, a moving object area is detected from the motion area in a moving object area detection part 1e, and the processing space of the motion vector detection part 1a and the background image generation part 1c is varied by referring to the output of the field space calculation part 1b.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention accurately extracts a moving object area that moves differently from a background from a moving image obtained from a movable image pickup device such as a video camera, and provides a quick view of the moving image. The present invention relates to a moving object area detection device used for an image editing device, an image monitoring device in a security system, and the like, which facilitates editing and searching.

[0002]

2. Description of the Related Art A conventional method for detecting a moving object using a background image is roughly divided into (1) a method of integrating a input image with a camera fixed to generate a background image. (2) A method in which the camera is movable, the motion vector of the input image is detected, and the background image is integrated after motion compensation. (3) Motion compensation is performed as in (2), the background image is not updated for the moving area (moving object area), and the input image and the background image are added at a certain ratio for the static area. and so on. The type (1) is mainly used for an image monitoring apparatus, and there is one disclosed in Japanese Patent Laid-Open No. 7-79429. Japanese Patent Laid-Open No. 5-2253 discloses a method for generating a background image when a camera is moving as a moving image capturing condition.
As disclosed in Japanese Patent Publication No. 41-41, there is disclosed a method of changing the addition ratio of the input image and the background image according to the change of the screen. However, this reduces erroneous detections due to abrupt changes in illumination and hidden objects in motion. Further, as a feature amount of a moving object, as disclosed in Japanese Patent Application Laid-Open No. 5-145844, a method of using horizontal and vertical lengths, areas and barycentric position of a moving object as a feature amount of a moving object region is disclosed.

[0003]

FIG. 39 shows a block diagram of a conventional moving object area detecting apparatus. In the conventional moving object area detection device, the field interval of each process in the motion vector detection unit and the background image generation unit is fixed, and the motion area detected by the motion area detection unit is erroneously detected or may be missed. I have a problem to do. This cause will be described with reference to FIG. 7 by modeling the moving object area in the image. The horizontal axis of FIG. 7 represents the spatial coordinate x (or y), the vertical axis represents the brightness value of the image, the solid line graph represents the brightness value of the input image, and the dotted line graph represents the brightness value of the reference image. There is. 1) When the motion vector of the captured moving image is small First, consider a moving object region in which luminance values are uniformly distributed as shown in FIG. The moving region detected by taking the difference between the input image and the reference image is only a small region. Since the processing for removing the isolated points is performed on the moving area to detect the moving object area, the moving area may be erased by this processing if the moving area is small.

Further, in FIG. 7B, consider a case where the luminance of the moving object region changes smoothly. Although the range in which the difference is detected is wide, the difference value (absolute value or squared value of the difference) is small, so it is regarded as noise in the image, and the motion region may not be detected. Further, if the motion area cannot be accurately detected, the background image generation unit regards it as a static area even though it is actually a motion area, and adversely affects the next motion area detection process. 2) When the motion vector is large Consider the case where the field interval for processing is widened in order to solve the problem of 1). The motion vector detection unit searches for a motion vector within a certain search range, but if a motion vector exceeding the search range occurs, the motion vector detection will be an erroneous detection. Naturally, the subsequent processing results will also be inaccurate.

In view of the above points, the present invention accurately considers a moving object region in consideration of a method of generating a background image that does not include a moving object region in a moving image in various shooting states, and a ratio of adding the background image and the input image. The object of the present invention is to provide a moving object area detecting device for realizing a moving image search, quick viewing, editing device and monitoring device.

[0006]

In order to achieve the above object, the first configuration of the present invention is a motion vector detecting section for detecting a motion vector of an input image, and a background image for creating a background image from the input image. At least a generation unit is provided, and the processing interval between the motion vector detection unit and the background image generation unit or the processing interval only between the background image generation units is variable based on the magnitude of the motion vector.

The second configuration includes at least a motion vector detecting section for detecting a motion vector of the input image and a background image generating section for creating a background image from the input image, and the magnitude and movement of the motion vector. The processing interval between the motion vector detection unit and the background image generation unit or the processing interval between only the background image generation unit is variable according to the correlation value with the peripheral pixels of the object area.

Further, the third configuration comprises at least a motion vector detecting section for detecting a motion vector of the input image and a background image generating section for creating a background image from the input image, and the magnitude and movement of the motion vector. The ratio of adding the background image and the input image in the background image generation unit is determined based on the correlation value with the peripheral pixels of the object area.

[0009]

According to the present invention, with the above-described configuration, the processing intervals of the motion vector detection and the background image generation are narrowed when the motion vector size is large, and when the motion vector size is small. By increasing the processing interval, a background image that does not include a moving object region is generated, and the moving object region can be accurately extracted using this background image.

In addition, in the condition that the processing interval is variable,
By using the correlation value with the surrounding pixels of the moving object area,
It is possible to detect moving objects that are not affected by image texture.

Further, by using the correlation value between the size of the motion vector and the peripheral pixels of the moving object area, the ratio of addition of the input image and the background image is made variable to generate a background image not including the moving object area. However, it is possible to accurately extract the moving object region using this background image.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the moving object area detecting apparatus of the present invention will be described below. FIG. 1 is an overall block diagram of a first embodiment of a moving object area detecting apparatus of the present invention. The overall processing flow will be described.

First, the motion vector detecting section 1a calculates a motion vector using an input image and a reference image which is an image before a predetermined field from the input image. The field interval calculation unit 1b refers to the size of the motion vector and the field interval at the time of the previous processing to update or continue the reference image in the motion vector detection unit 1a, and the background image generation unit 1c.
To update or continue the background image in. Next, the background image generation unit 1c generates a new background image using the input image, the motion vector, the background image generated in the previous process, and the motion area. The motion area detection unit 1d detects the motion area by calculating the difference between the background image corrected by the motion vector and the input image. In the moving object area detection unit 1e,
The moving object area having the largest area area is detected in the moving area.

The motion vector detecting section 1a can be realized by various methods using image processing.
Technical Report "(National Technical Report
The method described in Vol.37 No.3 pp308-204 Jun. 1991) will be described in detail with reference to FIG. (1) The input image in the field f is denoted by S (f), and the input image and in
Image S (f-int) separated by t (f) fields
(F)) is held. (2) In the representative point matching processing unit 2b, the image S
(F) and S (f-int (f)) are divided into five areas shown in FIG. 8, and the reference image S (f-int) shown in FIG.
The representative point matching process is performed at the representative point 9a of each area of (f) and the point within the search range 9b of each area of the input image S (f) to calculate the correlation value C (n, z). However, n represents the region number 1 ≦ n ≦ 5, and z represents the coordinates within the search range, and 1
There are as many correlation values C for one image as the product of the number of area divisions and the size of the search range. (3) In the effective area determination unit 2c, the average value Cav within the search range is calculated from the set of correlation values {C (n, z)} of the nth area.
e (n), minimum value Cmin (n), x, y direction components grdx (n), grdy (n) of the gradient near the minimum value of the correlation value are obtained, and the effectiveness of each region is verified using these , The motion vector v (f, n) in the nth region is calculated. (4) The motion vector determination unit 2d determines the motion vector V (f) from the motion vector v (f, n) of the effective area. (5) Using the detected motion vector V (f), the field interval calculation unit 1b determines whether the reference image storage unit 2a should be updated or continued.

However, the motion vector detecting section 1a is not limited to this method, and may have a configuration for detecting a change in local motion of an image by an optical flow.

Next, the processing of the field interval calculator 1b will be described with reference to FIG. Here, in order to solve the problem, the processing interval for updating the reference image and the background image is variable. (1) The image update determination unit 3a determines the image update or continuation using the magnitude of the motion vector detected and the field interval stored in the field interval storage unit 3b during the previous processing. When the magnitude of the motion vector is small, it is determined to continue the image, and when the magnitude is equal to or larger than a certain threshold, it is determined to update the image. Further, if the interval between the input image and the reference image is too wide, the reliability of the detected motion vector is reduced. Therefore, a threshold is also set for the field interval to determine whether to update or continue the image. The update conditions in the image update determination unit 3a are shown below.

Let the motion vector in field f be V
(F) Int the field interval between the input image and the reference image
When expressed as (f), it is determined to be updated when either of the conditions (Equation 1) and (Equation 2) is satisfied.

[0018]

[Equation 1]

[0019]

[Equation 2]

However, TH in the equation represents a threshold value.
Note that in (Equation 1) and (Equation 2), the magnitude of the motion vector and the field interval use independent threshold values, but it is naturally conceivable to change the threshold value of the motion vector magnitude depending on the value of the field interval. . That is, the condition is (Equation 3)
become that way.

[0021]

(Equation 3)

In the actual video signal, the movement of the camera,
Since the motion of the moving object is mostly in the horizontal direction, it can be realized by using the lateral (x) direction component Vx (f) of the motion vector.

The background image generator 1c will be described in detail with reference to FIG. (1) Referring to the output from the field interval calculation unit 1b,
If this continues, the current background image is retained and new generation is not performed. When updating, the previously generated background image BG (f held in the background image storage unit 4c is stored.
-Int (f)) and the motion vector V are used to perform the coordinate conversion shown in (Equation 4) in the coordinate conversion unit 4a, and B (f-in
Calculate t (f)).

[0024]

(Equation 4)

(2) In the background image updating unit 4b, the background image B (f-int) whose coordinates have been converted by (Equation 4) is used.
(F)), the input image S (f), and the motion area are used to update the background image. At this time, the background image is not updated in the area determined to be the motion area, and the background image is updated using (Equation 5) for the area that is not the motion area (static area).

[0026]

(Equation 5)

(3) The background image storage unit 4c stores the background image B (f) created by the background image update unit 4b as BG (f) until the next update.

The moving area detecting section 1d will be described in detail with reference to FIG. (1) Background image BG (f-int (f)) using detected motion vector V to compensate for camera motion
To the same position as the input image S (f).
The coordinate conversion of (Equation 4) is performed with a. (2) The difference calculation unit 5b creates a difference image between the input image S (f) and the coordinate-converted background image B (f-int (f)). (3) Next, the motion area determination unit 5c executes motion area determination processing on the input and background difference images. Here, if the filter size is L (where L is an odd number) and the filter target pixel is (x0, y0), then (x, y) = (x0 ± (L-1) / 2, y0 ± (L −
For 1) / 2), the motion area is determined using (Equation 6).

[0029]

(Equation 6)

Pixels whose difference value is greater than or equal to a threshold THlab by calculating the difference with respect to L × L pixels around the filter target pixel (x0, y0) (pixels satisfying (Equation 6))
If the number of pixels is L × L / 2 or more, the target pixel (x
0, y0) is set as a motion area.

The moving object area detecting section 1e will be described in detail with reference to FIG. (1) In the labeling processing unit 6a, pixels adjacent to each other in the motion area are regarded as one group, and each group is sequentially numbered. (2) The adjacent region combination processing unit 6b is a process of combining adjacent ones of the labeled groups. (3) In the isolated point removal processing unit 6c, when the area forming the group is small, it is treated as noise and is changed to the still area. (4) The moving object area determination unit 6d detects the group having the largest area area as the moving object area after the above processing.

In order to confirm the effectiveness of the above-described first embodiment of the present invention, a comparison with the conventional system was made. As an evaluation image, a scene in which a camera tracks a person walking is selected. The area of the area as the feature of the moving object
Use the position of the center of gravity. If the moving object region is accurately detected in the tracking scene used for the evaluation image, the size of the area can be kept stable to some extent, and the position of the center of gravity can be obtained with a stable value with little change with time.

As the detection result of the moving object area, the area of the moving object area is shown in FIG. 10 and the center of gravity is shown in FIG. For comparison, the system 10a of the present invention equipped with the field interval calculator in FIG. 1 and the conventional system 10b with a fixed field interval.
It is. The horizontal axis represents the number of fields representing the time change, and the vertical axis represents the area of the moving object region and the position of the center of gravity in the x-axis direction. Here, since the movement of the camera or the moving object is mostly in the lateral direction, only the x-axis direction is referred to as the center of gravity position. 10 and 11
It can be seen that the area of the moving object and the position of the center of gravity are stable in the method of the present invention, but unstable in the conventional method.

Next, a second embodiment of the moving object area detecting apparatus of the present invention will be described. FIG. 12 is an overall block diagram of the second embodiment of the moving object area detecting apparatus of the present invention. The difference from the first embodiment is that the field interval calculator 12b refers to the moving object area in addition to the magnitude of the motion vector.

In the first method, the field interval is determined by comparing the magnitude of the motion vector with a threshold value, but in the second method, the presence or absence of texture in the moving object area is also included in the determination condition. Focusing on the edge part of FIG. 7 (A), the difference value is calculated even if the movement distance is small in a portion with texture, but conversely, the movement distance is small in an image with few textures as shown in FIG. 7 (B). If it is not large, the difference value will be small.

That is, the appropriate field interval is determined by referring to the texture of the moving object area instead of determining that the moving area is calculated only by the size of the motion vector. In order to determine the texture of the moving object area, the representative point matching process is performed on the moving object area to calculate the correlation value with the surrounding pixels, and from this correlation value, the pixel interval at which the correlation with the surrounding pixels becomes low is calculated. calculate.

The processing of the field interval calculator 12b will be described in detail with reference to FIG. (1) The reference area storage unit 13a stores a reference area that is a moving object area before the predetermined field from the current field. The update or continuation of the reference area is performed by the image update determination unit 13d. (2) In the representative point matching processing unit 13b, FIG.
The representative point matching process is performed at the representative point 9c of the reference area and the point within the search range 9d of the moving object area shown in FIG.
Calculate (m, z). However, z represents the coordinates within the search range, and m represents the moving object region. FIG. 14 shows an example of the correlation value C (m, z) for z = (x, y). (3) The pixel interval calculation unit 13c calculates the average value Cave of the correlation values from the set of correlation values {C (m, z)} of the moving object area.
(M), the minimum value Cmin (m), and the x- and y-direction components Gx (m) and Gy (m) of the gradient near the minimum correlation value are calculated, and the relationship between them is shown in FIG. Since the correlation is high in the vicinity of the minimum value, it can be said that the correlation with the surrounding pixels is low if the distance from the position showing the minimum value to the position showing the average value is increased. To obtain this, two in the horizontal (x) direction and two in the vertical (y) direction, a total of 4
The minimum value Gmi of the slopes among the existing slopes around the minimum value
Find n (however, Gmin ≠ 0). The pixel interval N with which the correlation with surrounding pixels is lower than this is calculated by (Equation 7).

[0038]

(Equation 7)

(4) In the image update determination unit 13d, the field interval T is calculated by (Equation 8) using the pixel interval N and the magnitude of the motion vector | V | that have a low correlation with the peripheral pixels of the moving object area. decide.

[0040]

(Equation 8)

That is, it means that T (f) fields are required between the background image and the input image in order to accurately detect the motion area. Field interval storage unit 3b
The field interval int (f) between the input image and the reference / background image held in the table and the calculated field interval T
Using (f), the condition for image update in the image update determination unit 13d is (Equation 9).

[0042]

[Equation 9]

Note that it is naturally conceivable to refer to the condition using the threshold value of int (f) in (Equation 2) as well as (Equation 9) at the time of determination. Further, in the actual video signal, since the movement of the camera and the movement of the moving object are mostly in the horizontal direction, it can be realized by using the horizontal (x) direction component Vx (f) of the motion vector.

The second embodiment of the moving object area detecting apparatus of the present invention can also be realized by using FIG. 16 instead of FIG. 13 for the processing of the field interval calculator 12b of FIG. Details of the field interval calculator 12b will be described with reference to FIG. (1) The variance of the moving object region can be calculated by the variance calculator 16a as in (Equation 10).

[0045]

(Equation 10)

(2) The pixel interval calculation unit 16b uses the function g in order to estimate the presence or absence of texture in the area by using the variance.
Is defined to determine the pixel interval N1 with which the correlation with surrounding pixels is low, using (Equation 11).

[0047]

[Equation 11]

There are various monotone decreasing functions such as 1 / σ and exp (-σ) as g (σ). (3) The image update determination unit 16c sets N1 to N in (Equation 7).
Equivalent to (f), N1 in (Equation 11) is substituted into N (f) in (Equation 8) to calculate the field interval T (f). This T (f) and int of the field interval storage unit 3b
Using (f), it is determined whether to update or continue according to the image update condition (Equation 8).

Next, a third embodiment of the moving object area detecting apparatus of the present invention will be described. FIG. 17 is an overall block diagram of the moving object area detecting apparatus according to the third embodiment of the present invention. The difference from the first and second embodiments is that the field interval calculator 17b refers to the correlation value between the input image and the reference image in addition to the magnitude of the motion vector.

First, although there are five areas in which the correlation value of the representative point matching processing section 2b is used from FIG. 8, the normal object is present in the center of the screen in a moving image in which the object is tracked. Use area. However, as a more general method, there is also a method of detecting the barycentric position of the moving object region as shown by the dotted line in FIG. 17 and determining the n-th region to be referred from the barycentric position. Here, it is assumed that the position of the center of gravity of the moving object area is detected by the moving object area feature amount detection unit 17c.

The motion vector detecting section 17a is constructed so that the correlation value output from the representative point matching processing section 2b in FIG. 2 can be referred to outside the motion vector detecting section 1a.

The field interval calculator 17 will be described with reference to FIG.
The details of b will be described. (1) The reference area determination unit 18a determines which of the five areas in FIG. 8 is to be referred to from the position of the center of gravity of the moving object area. When the position of the center of gravity is included in the fifth region, the fifth region is referred to, and when not included, the region including the position of the center of gravity is referred to. However, this process is not performed when the fifth area is referenced from the beginning. (2) The pixel interval calculation unit 18b performs the process of 13c not on the moving object region but on the nth region (1 ≦ n ≦ 5) shown in FIG. That is, the set of correlation values is {C
From (n, z)}, the average value Cave (n) of correlation values, the minimum value Cmin (n), and the x- and y-direction components Gx (n) and Gy (n) of the gradient near the minimum correlation value are calculated. The minimum value Gmin of the gradient is obtained among the four gradients around the minimum value (where Gmin ≠ 0). The pixel interval N with which the correlation with surrounding pixels is lower than this is calculated by (Equation 7). (3) The processing in the image update determination unit 13d and the field interval storage unit 3b is equivalent to the above description, and therefore the description is omitted.

In FIG. 1 in the first embodiment of the present invention, in FIG. 12 in the second embodiment, and in FIG. 17 in the third embodiment, the output of the field interval calculator is replaced by the motion vector detector. Although it is referred to by the background image generation unit, the process can be executed at a fixed field interval (or every field) without being referred by the motion vector detection unit.

In this case, the field interval calculation unit realizes by integrating motion vectors detected at fixed field intervals (or every field) and detecting motion vectors between variable fields. Details of the motion vector detecting units 1a and 17a in this case will be described with reference to FIG. (1) In the reference image storage unit 19a, the input image S (f) in the field f and the image S (f
-F) is held. (2) Representative point matching processing unit 2b, effective area determination unit 2
c, the processing of the motion vector determination unit 2d is the same as the above description, and thus the description is omitted.

When the field interval of the motion vector detecting unit 1a is fixed, the motion vector detecting unit 1a can be realized by a method of attaching a sensor to the photographing device and referring to the output thereof in addition to the method using the above image processing. .

The field interval calculator 1b when the motion vector detection is executed at the fixed field interval in the first embodiment will be described in detail with reference to FIG. (1) The motion vector integration unit 20a integrates the outputs of the motion vector detection units 1a and 17a to calculate the motion vector at the variable field interval. This is repeated until the image update determination unit 3a determines that the image is updated. When it is determined that the image is updated, the motion vector accumulated until that time is reset to zero. (2) Image update determination unit 3a and field interval storage unit 3b
Since the processing of is the same as the above description, it will be omitted.

FIG. 13 shows the process of the field interval calculator 12b in the second embodiment and the process of the field interval calculator 17b in the third embodiment when the motion vector detecting process is executed at a fixed field interval. ,
The image update determination unit 13d (or 16 in FIGS. 16 and 18)
The structure is such that the motion vector integration unit 20a is provided between c) and the motion vector that is the input of 13d (or 16c). Since the processing of the motion vector integration unit 20a is equivalent to the above description, the description will be omitted.

Next, the fourth embodiment of the moving object area detecting apparatus of the present invention will be described.
An example will be described. 21 and 24 are overall block diagrams of the moving object area detecting apparatus according to the fourth embodiment of the present invention. The difference from the first, second, and third embodiments is that the processes of the motion vector detecting unit and the background image generating unit are not executed at variable field intervals, but the background image updating unit 4b of the background image generating unit 1c. Coefficient calculation units 21a and 2 for calculating the integration parameter (see (Equation 5)) in
4a is provided. The reason why the field interval for processing is fixed is that it may not be possible to make the processing interval variable when the product is actually commercialized. The correlation between the magnitude of the motion vector and the peripheral pixels of the moving object area is used as an element for determining the integration parameter.

FIG. 21 shows a method of calculating the correlation with surrounding pixels by referring to the moving object area, and FIG. 24 shows the periphery of the moving object area from the correlation value of the nth area (usually the fifth area) of FIG. This is a method of estimating the correlation with pixels.

First, the processing performed by the coefficient calculation unit 21a will be described with reference to FIG.
2 and FIG. 23, and the processing performed by the coefficient calculation unit 24a will be described with reference to FIG. The reference area accumulator 13a, the representative point matching processor 13b, the pixel interval calculator 13c in FIG. 22, and the variance calculator 16a in FIG.
Since the pixel interval calculation unit 16b and the reference area determination unit 18a in FIG. 25 are the same as the above description, the description is omitted and the processing of the coefficient determination unit 22a will be described.

As an example, let us consider a case where each field processing is performed. In the formula (Formula 5) for generating the background image, i
Substituting nt (f) = 1, it is transformed as shown in (Equation 12).

[0062]

(Equation 12)

From this (Equation 12), it can be seen that images before the t field are referred to in the current background image at a ratio of a (1−a) ^t .

According to the above-mentioned principle, in the field interval variable system, the operation is awaited during the int (f) = N / | V | field. That is, even when the field interval is fixed, if the int (f) field is referred to when the background image is generated, the same effect as when the processing interval is made variable is obtained, and the detection accuracy of the moving object region is improved. The relationship between the number of fields t and the ratio y contained in the background image is (Equation 13)
Becomes

[0065]

(Equation 13)

FIG. 26 shows (Equation 13). From FIG. 26, t
If y increases, the ratio y decreases, but if t → ∞, then y
It does not become = 0. Therefore, it is defined that the differential straight line at t = 0 refers to the field that intersects the t-axis, and the number of fields is described as T1. (Equation 14) is derived by differentiating (Equation 13) using the logarithmic differentiation method.

[0067]

[Equation 14]

From equation (14), the slope a l at t = 0
If og (1-a) and a straight line having an intercept a are written as y1, then (Equation 15) is obtained.

[0069]

(Equation 15)

Substituting y1 = 0 into this and obtaining T1

[0071]

(Equation 16)

Is obtained. Solving for a from (Equation 16)

[0073]

[Equation 17]

Is obtained. (Equation 12) is a case where the processing is performed for each field, but it can be derived in the same manner even in the case of a fixed interval and a plurality of fields. Assuming that T1 in (Equation 17) is directly proportional to T (f) in (Equation 8), T1
= ΒN / | V |, and substituting this into (Equation 17) gives (Equation 18).

[0075]

(Equation 18)

The coefficient determining section 22a determines the integration parameter a by substituting the pixel interval N at which the correlation between the magnitude | V | of the motion vector and the surrounding pixels becomes low into (Equation 18).
The relationship of (Equation 18) is shown in FIG.

In order to examine the effect of (Equation 18), comparison of moving object area detection will be performed. As the feature amount of the moving object region, the area and the x-direction component of the center of gravity are used, and the area is shown in FIG.
FIG. 29 shows the position of the center of gravity. The evaluation image is an image obtained by tracking a walking scene of a person with a camera, but is different from the tracking scene used in FIGS. 10 and 11.

For comparison, the method 28a in which the processing interval is variable and a = 0.8 and the processing is executed in each field and a = 0.2, 0.4,
The respective methods 28b, 28c and 28d are set to 0.8. It is understood that the moving object area can be accurately detected by setting a = 0.2 when the field interval is fixed. When an appropriate value is substituted for β in (Equation 18) for this tracking scene and a is calculated, it is around 0.2 to 0.3, and it can be seen that the results almost match. However, the smaller a is, the earlier the field is referred to. Therefore, noise may be integrated and a motion area may be erroneously detected.

Next, the fifth embodiment of the moving object area detecting apparatus of the present invention will be described.
An example will be described. 30 and 31 are overall block diagrams of the moving object area detecting apparatus according to the fifth embodiment of the present invention. In this fifth embodiment, the first, second,
The third embodiment is provided with coefficient calculation units 30a and 31a for calculating an integration parameter (see (Equation 5)) in the background image update unit 4b of the background image generation unit 1c.

Now, let us consider the integration parameter a when the field interval is variable. The closer the value of a to 1, the better, if the moving object area can be detected completely. In other words, when the background image is generated: moving area: background image static area: input image, the background image changes due to camera movement (when the camera pans, the distance from the camera to the object differs , Etc.) can be dealt with quickly. Also, even if noise occurs on the screen, it will be affected by the noise on that screen, but it will not be related to the next processing,
The moving object can be detected correctly.

That is, with the coefficient calculation unit 30a, when the field interval calculation unit 12b updates the image under the condition of (Equation 9), the integration parameter is set to a = 1. In the case of performing, the configuration is such that the integration parameter a is determined by the same method as the coefficient determination unit 21a (details are shown in FIG. 22).

When the field interval calculation unit 17b updates the image under the condition of (Equation 9), the integration parameter is set to a = 1. The image update according to the condition of (Equation 2). In the case of performing, the configuration is such that the integration parameter a is determined by the same method as the coefficient determination unit 24a (details are shown in FIG. 25).

Next, another method of the motion area determination unit 5c of the motion area detection unit 1d in FIGS. 1 and 12, 17 and 21, 21 and 30 and 31 will be described. The motion area determination method described in the first embodiment refers to the values of the peripheral pixels on the same screen. Here, the moving region is determined in consideration of the temporal continuity of the moving object region. That is, the filtering process is performed not only within the same screen but also in the time axis direction. The weighting coefficient is Z (however, when Z> 1 and Z is increased, the weighting to the peripheral pixels is increased), and the pixel to be filtered (x0, y
The distance from 0) is defined as d, and the weight is reduced as the distance from the target pixel increases. Although the Euclidean distance is typical as the distance, it goes without saying that the distance can be designed by using other four-neighbor distance, eight-neighbor distance, and octagonal distance. As an example, a spatiotemporal filter designed using the three-dimensional Euclidean distance d is used. Used here 3
The dimension Euclidean distance is shown in (Equation 19). However, f0 represents the current field.

[0084]

[Equation 19]

Here, α is a coefficient that changes depending on the magnitude | V | of the motion vector, and an example thereof is shown in (Equation 20). A small motion vector indicates a high correlation in the time direction, and a large motion vector indicates a low correlation in the time direction.

[0086]

(Equation 20)

Element value W of the filter using (Equation 19)
(F, x, y) becomes (Equation 21).

[0088]

(Equation 21)

The filter size is represented by L in the spatial direction (where L is an odd number) and LF in the temporal direction. Z = 2, α = 1,
FIG. 32 shows an example of filter design when L = 3 and LF = 3. The motion region determination condition for the filter target pixel (x0, y0) is shown in (Equation 22).

[0090]

(Equation 22)

Here, I (f, x, y) is a luminance value, NW
Is a filter normalization coefficient, and THw is a threshold value. The normalization coefficient is shown in (Equation 23).

[0092]

(Equation 23)

That is, a pixel satisfying (Equation 22) is detected as a motion area. Next, FIG. 1, FIG. 12, FIG. 17, and FIG.
1, the moving object area detection unit 1e in FIGS. 30 and 31.
Another method will be described. Here, the respective processes shown in FIG. 6 are not executed for the moving area, and frequency conversion is used for detecting the moving object area. This method is characterized in that it is not possible to detect the details of the moving object area, but it is possible to detect the rough position and size. The method of using frequency conversion in the moving object area detection unit 1e will be described in detail with reference to FIG. <Procedure> (1) In the frequency conversion processing unit 33a, the motion area detection unit 1
Two-dimensional frequency conversion (FFT, DCT, etc.) is performed on the difference image of d by dividing the entire screen or block division. (2) The frequency filter processing unit 33b detects the AC component (excluding the DC component) having the maximum amplitude among the frequency components in each of the x and y directions. (3) The frequency inverse transform processing unit 33c inversely transforms the frequency component with the maximum amplitude (in each of the vertical and horizontal directions) detected in step 2 and returns it to the spatial domain. (4) The moving object region determination unit 33d calculates the correlation between the periodic function obtained by the frequency inverse transform and the difference image, and determines a period having a high correlation (a period from the minimum value of the function to the adjacent minimum value) as x. , Y, one for each direction. (5) The moving object area feature amount detection unit 33e sets the peak position of the section function detected in step 4 as the center of gravity position of the moving object. Further, the area of the moving object region can be derived as πuv / 4 using the ellipse area formula, where u and v are the periods of the extracted frequency components in the x and y directions. This is shown in FIG.

As a first application example of the present invention, there is a device which easily edits, looks at, and searches a moving image in accordance with the intention of a cameraman at the time of shooting. Here, the importance of the scene is determined based on the camerawork of the photographer rather than the action pattern of the moving object, and a moving image browsing system is constructed using the image of high importance as the representative image.

The next two scenes are similar panning (camera movement in the horizontal direction) scenes, but the intentions of the photographer are completely different. (1) Pan the camera to track a moving subject. (2) Pan the camera to shift the viewpoint from one subject to another.

That is, in (1), since the photographer pays attention to the subject during the panning period, it seems to be an important scene. On the other hand, in (2), the panning period is only a connection between two subjects, which seems to be an unnecessary scene. After that, the scene of (1) is the "tracking scene",
The scene (2) is called a "viewpoint transition scene".

FIG. 35 shows a scene discriminating method using the present invention for discriminating these two scenes. In FIG. 35, the processing in the first to fifth embodiments, that is, the processing until the moving object area is detected from the input image is referred to as a moving object detection unit 35a. The panning section detection unit 35b detects the scene in which the panning operation is performed, and the moving object feature amount detection unit 35c obtains the barycentric position, the area, and the like. Here, the center of gravity position is stable and the area is constant in the tracking scene, but both the center of gravity position and the area are unstable in the viewpoint transition scene. Utilizing this fact, the scene determination unit 35
In d, the tracking / viewpoint transition scene in the panning section is determined. The representative image determination unit 35e determines a representative image having high importance based on the result of the scene determination. The moving image display unit 35f has a configuration in which representative images are enumerated, and when the representative image is selected, the scene is reproduced.

A second application example of the present invention is a device for automatically determining the zoom magnification of a photographing device by detecting a moving object. FIG. 38 shows a moving object existing in the distance (see FIG.
(A) state is approaching ((B) and (C) in FIG. 38), which corresponds to a case where a skiing time is photographed. At this time, it is difficult for the photographer to change the zoom magnification of the photographing apparatus according to the size of the moving object area as shown in FIG. 38. If this automatically changes, the burden on the photographer is reduced, and a better image is obtained. Can be taken. FIG. 36 shows a block diagram of the image pickup apparatus having the automatic zoom magnification variable function, and the details will be described with reference to FIG. (1) First, the moving object detection unit 35a detects the moving object area using the input image obtained by the image capturing device 36a. (2) The zoom magnification calculator 36b calculates the optimum zoom magnification from the area of the moving object region. (3) The zoom magnification controller 36c controls the zoom magnification of the photographing device 36a according to the output of the zoom magnification calculator 36b. (4) In the image capturing device 36a, since the zoom magnification is variable depending on the area of the moving object region, the obtained image is a large image of the moving object region. This image is sent to the image recording device 36d and the image display device 36e to store and display the image.

The position of the photographing device 36a at the time of photographing may be fixed or the photographer may move according to a moving object. When the photographer moves the photographing device, the image may be disturbed due to camera shake, or the tracking photographing of the moving object may fail. Therefore, FIG. 37 shows a block diagram of an image pickup apparatus with a camera position automatic changing function for controlling the position of the camera according to the position of the center of gravity of the moving object area, and the details will be described with reference to this. However, the description of the same parts as those in FIG. 36 is omitted. (1) The photographing device 37a is designed so that the direction of the lens of the camera of the photographing device 36a can be moved. (2) The camera position calculation unit 37b calculates the direction of the camera lens from the position of the center of gravity of the moving object area. (3) The camera position controller 37c controls the direction of the lens of the camera of the photographing device 37a according to the output of the camera position calculator 37c. (4) In the imaging device 37a, the direction of the lens of the camera is variable depending on the position of the center of gravity of the moving object area, so that an image that follows the moving object can be obtained. This image is sent to the image recording device 36d and the image display device 36e to store and display the image.

Although the processing interval is the field interval in the embodiment, it may be a frame interval.

[0101]

As described above, in a scene in which a moving object is tracked, when the speed of the moving object is slow, in the conventional method in which the processing interval is fixed, the moving object area is often erroneously detected. . However, the processing interval can be changed according to the magnitude of the motion vector and the correlation value of the moving object area,
Alternatively, in the present invention in which the ratio of adding the background image and the input image is variable according to the magnitude of the motion vector and the correlation value of the moving object area, such a case can be dealt with, and a more general moving object area detection can be performed. Can be applied.

[Brief description of drawings]

FIG. 1 is an overall block diagram of a first embodiment of a moving object area detection device of the present invention.

FIG. 2 is a block diagram showing a motion vector detection unit 1a in FIG.

FIG. 3 is a block diagram showing a field interval calculator 1b shown in FIG.

FIG. 4 is a block diagram showing a background image generation unit 1c shown in FIG.

5 is a block diagram showing a motion area detection unit 1d in FIG.

6 is a block diagram showing a moving object area detection unit 1e in FIG.

7 (A) and (B) are diagrams showing difference values when the motion vector is small.

FIG. 8 is a diagram showing area division in a representative point matching processing unit.

9A and 9B are diagrams showing a representative point and a search range in the representative point matching processing unit.

FIG. 10 is a diagram showing the area of a moving object region.

FIG. 11 is a diagram showing the position of the center of gravity of a moving object region.

FIG. 12 is a second view of the moving object area detection device of the present invention.
Overall block diagram of the embodiment

13 is a block diagram showing a field interval calculation unit 12b (method using correlation value of moving object region) of FIG. 12;

FIG. 14 is a diagram showing correlation values calculated in representative point matching processing.

FIG. 15 is a diagram showing a minimum value, an average value, and a slope near the minimum value of correlation values.

16 is a block diagram showing a field interval calculation unit 12b (method using dispersion of moving object area) in FIG. 12;

FIG. 17 is a third view of the moving object area detection device of the present invention.
Overall block diagram of the embodiment

18 is a block diagram showing a field interval calculator 17b shown in FIG.

FIG. 19 is a block diagram showing a motion vector detection unit 1a that executes at fixed field intervals.

20 is a block diagram showing a field interval calculator 1b when the motion vector detector of FIG. 19 is used.

FIG. 21 is a fourth view of the moving object area detection device according to the present invention.
Overall block diagram of the embodiment (method using moving object area)

22 is a block diagram showing a coefficient calculation unit 21a (a method using a correlation value of a moving object region) in FIG. 21.

23 is a block diagram showing a coefficient calculation unit 21a (method using dispersion of moving object area) in FIG. 21.

FIG. 24 is a fourth view of the moving object area detection device of the invention.
Example (method using correlation value of motion vector detection unit)
Overall block diagram of

FIG. 25 is a block diagram showing a coefficient calculation unit 24a (method using a correlation value of a motion vector detection unit) of FIG.

FIG. 26 is a diagram showing the relationship between the number of fields and the ratio of the image included in the background image.

FIG. 27 is a diagram showing the relationship between the number of fields included in the background image and the integration parameter.

FIG. 28 is a diagram showing an area (comparison of integration parameters) of a moving object region.

FIG. 29 is a diagram showing a barycentric position (comparison of integration parameters) of a moving object region.

FIG. 30 is a fifth diagram of the moving object area detection device of the invention.
Overall block diagram of the embodiment (method using moving object area)

FIG. 31 is a fifth diagram of the moving object area detection device of the invention.
Example (method using correlation value of motion vector detection unit)
Overall block diagram of

FIG. 32 is a diagram showing a spatiotemporal filter.

FIG. 33 is a diagram showing a case where frequency conversion is used in the moving object area detection unit.

FIG. 34 is a diagram showing a feature amount of a moving object region when frequency conversion is used.

FIG. 35 is a diagram showing an application example of the present invention to moving image browsing.

FIG. 36 is a diagram showing an example of application of the present invention to an image pickup apparatus with a variable zoom magnification function.

FIG. 37 is a diagram showing an example of application of the present invention to an image pickup apparatus with a camera position automatic changing function.

FIG. 38 is a diagram showing a scene in which a moving object existing in the distance approaches.

FIG. 39 is a diagram showing a conventional example of a moving object area detection device.

[Explanation of symbols]

 1a Motion vector detection unit 1b Field interval calculation unit 1c Background image generation unit 1d Motion area detection unit 1e Moving object area detection unit 2a Reference image storage unit 2b Representative point matching processing unit 2c Effective area determination unit 2d Motion vector determination unit 2e Field interval Calculation unit 3a Image update determination unit 3b Field interval storage unit 4a Coordinate conversion unit 4b Background image update unit 4c Background image storage unit 5a Coordinate conversion unit 5b Difference calculation unit 5c Motion region determination unit 6a Labeling processing unit 6b Proximity region combination processing unit 6c Isolated point removal processing unit 6d Moving object region determination unit 9a Representative point (reference image) 9b Search range (input image) 9a Representative point (moving object region) 9b Search range (moving object region) 10a Method of the present invention 10b Conventional method 12b Field interval calculator 13a Reference area accumulator 13b Representative point map Stitching processing unit 13c pixel interval calculation unit 13d image update determination unit 16a variance calculation unit 16b pixel interval calculation unit 17a motion vector detection unit 17b field interval calculation unit 17c moving object region feature amount detection unit 18a reference image determination unit 18b pixel interval calculation unit 19a Reference image storage unit 20a Motion vector integration unit 21a Coefficient calculation unit 22a Coefficient determination unit 24a Coefficient calculation unit 28a Processing interval variable method (a = 0.8) 28b Processing interval fixed method (a = 0.2) 28c Processing interval fixed method (a = 0.4) 28d Fixed processing interval method (a = 0.8) 30a Coefficient calculation unit 31a Coefficient calculation unit 33a Frequency conversion processing unit 33b Frequency filter processing unit 33c Inverse frequency conversion processing unit 33d Moving object region determination unit 33e Moving object region feature amount detection unit 35a Moving object detection unit 35b Panning section size Part 35c Moving object feature amount detection unit 35d Scene determination unit 35e Representative image determination unit 35f Moving image display unit 36a Imaging device 36b Zoom magnification calculation unit 36c Zoom magnification control unit 36d Image recording device 36e Image display device 37a Imaging device 37b Camera position calculation 37c Camera position controller

Claims (7)

[Claims] 1. A moving object area detection device for detecting a moving object area of an input image by using a video signal of the image as an input image, and a motion vector detecting section for detecting a motion vector of the input image, At least a background image generation unit for generating a background image from an input image is provided, and a processing interval between the motion vector detection unit and the background image generation unit or a processing interval between only the background image generation unit based on the magnitude of the motion vector. A moving object area detection device characterized by making variable. 2. A moving object area detection device for detecting a moving object area of an input image by using a video signal of the image as an input image, and a motion vector detecting section for detecting a motion vector of the input image, At least a background image generation unit for generating a background image from an input image is provided, and a processing interval between the motion vector detection unit and the background image generation unit based on a correlation value between the size of the motion vector and peripheral pixels of the moving object area. Alternatively, the moving object area detection device is characterized in that the processing interval of only the background image generation unit is variable. 3. A moving object area detecting device for detecting a moving object area of the input image, using a video signal of the image as an input image, and a motion vector detecting section for detecting a motion vector of the input image, At least a background image generation unit that creates a background image from the input image, the background image and the input image in the background image generation unit based on the correlation value between the size of the motion vector and the peripheral pixels of the moving object region A moving object area detection device, characterized in that it determines a ratio for adding. 4. A moving object area detection device for detecting a moving object area of an input image by using a video signal of the image as an input image, and a motion vector detecting section for detecting a motion vector of the input image, At least a background image generation unit for generating a background image from an input image is provided, and a processing interval between the motion vector detection unit and the background image generation unit based on a correlation value between the size of the motion vector and peripheral pixels of the moving object area. Alternatively, the processing interval of only the background image generation unit is made variable, and the ratio of adding the background image and the input image in the background image generation unit according to the correlation value between the size of the motion vector and the peripheral pixels of the moving object region A moving object area detection device characterized by: 5. The movement according to claim 2, wherein a variance value of the moving object area or a correlation value calculated by a motion vector detection unit is used as a correlation value with peripheral pixels of the moving object area. Object area detection device. 6. A motion area detecting section for detecting a motion area from a background image and an input image, wherein the motion area is filtered in the time axis direction.
5. The moving object area detection device according to any one of 5 above. 7. A moving object area detection unit for detecting a moving object area of interest from a moving area, wherein the moving object area is detected by using frequency conversion for the moving area. The moving object area detection device according to any one of 1 to 6.