JP3149839B2 - Object extraction device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object extracting apparatus, and more particularly to an object extracting apparatus for extracting a region where a moving object exists from an image.

[0002]

2. Description of the Related Art Conventionally, as an object extraction method using an object extraction apparatus, for example, the Transactions of the Society Conference of Electronic Information and Communication 1996, D-404, "Moving Object Detection Using Stereo Images" (Satota et al., 1996. 9). There is an object extraction method using a stereo image proposed by J.

FIGS. 5A and 5B are diagrams showing this object extraction technique. According to this object extraction method, first, as shown in FIG. 5A, an image without a target object is photographed by a stereo camera. For a stereo image composed of a left image (imaging surface S _{l of the} left camera) and a right image (imaging surface S _{r of the} right camera) photographed by this stereo camera, the correspondence between pixels of these images is calculated. I do. Corresponding calculation The relationship is to determine the pixel P _r on any of the left image corresponding to a point P _b pixel P _l and the right image on the background. Incidentally, in FIG., C _l is C _r represents the center of the left camera lens represents the center of the left camera lens. The calculation of the correspondence is performed for the entire image, and the result is stored in the form of an image file or the like.

[0005] Next, as shown in FIG. 5 (b), in a frame in which a target object is present, an input stereo image is set as a target, and reference is made to a stored correspondence relationship between pixels.
Either a correlation value or a difference value is calculated as a similarity measure between pixels in a correspondence relationship, and the calculated value is compared with a threshold value to determine whether the correspondence is broken. Then, a pixel whose correspondence has been lost (the degree of similarity is low) is extracted as a region where an object exists.

[0005] That is, as shown in FIG. 5A, in an image in which no target object exists, pixels P _l and P _r in a corresponding relationship are present.
Refers to the same point P _b. Thus, the pixel P _l, corresponding is well-P _r, the degree of similarity is high. However, FIG.
As shown in (b), when the target object O exists, in the region where the object O exists, the pixels P _l , P
_r is it would point to the another point P _ol, P _or on the object O, the pixel P _l, the similarity similarity collapse of P _r decreases. The area where the similarity is reduced is extracted as the display area of the object O.

[0006]

However, according to the object extraction method using the conventional object extraction device, only one value is used as the threshold value used to determine the collapse of the correspondence between pixels. Therefore, there are the following problems. (1) For example, when the background is complicated, the threshold value must be set higher in order to prevent the background from being extracted as an object. This makes it difficult to detect a change in the similarity due to the sensibility, and there is a problem that an object cannot be extracted much. (2) The texture and brightness of the object and the background affect the extraction of the object. However, if only one threshold is used, it cannot be dealt with, and the extraction accuracy of the object is significantly reduced. There is a problem of letting them do it. This problem will be described with reference to FIG.

FIG. 6 is a diagram for explaining a change in a difference value between pixels due to brightness (pixel brightness) and texture. Corresponding pixels (pixels P _l ,
And P _r), the pixel P _l at pixel correspondence is lost (FIG. 5 (b), the correlation between the luminance difference value of P _r), is a representation divided into the presence or absence of texture. From this figure, it can be seen that the difference value greatly changes according to the luminance of the pixel and the presence or absence of the texture. In FIG. 6, the threshold value is set based on the correlation with non-correspondence and texture.
Therefore, for example, if the background has a texture and the object has no texture, the object cannot be extracted from the texture. This is the same when the correlation value is used instead of the difference value.

The presence or absence of a texture in FIG.
The presence of a texture means that there is a very fine texture, and the absence of a texture means that no texture can be seen in an image. Therefore, when an actual environment is assumed, it is considered that the actual correlation between the difference value and the luminance value fluctuates between the correlations shown in FIG. 6 according to the fineness of the texture. For this reason, highly accurate extraction cannot be expected by the object extraction method using the conventional object extraction device.

Accordingly, an object of the present invention is to provide an object extracting apparatus capable of performing high-precision extraction without being affected by differences in texture of backgrounds and moving objects and brightness.

[0010]

In order to achieve the above object, an object extracting apparatus according to the first aspect of the present invention includes a plurality of illumination devices.
An image group obtained by imaging at a plurality of different positions according to light conditions is displayed in an initial state where no moving object exists and in a state other than the initial state.
The image input unit that is acquired in the state of
A correspondence detection unit that detects a correspondence between pixels among an image group acquired under an arbitrary lighting condition among an image group in an initial state, and at least one of an image group acquired by an image input unit. A brightness calculation unit that calculates the brightness of the pixels constituting the two images,
Accordance correspondence between pixels corresponding detection unit detects the threshold measuring section for measuring brightness, one or more thresholds of similarity measure is defined by an indicator such as color separately by a pixel unit luminance value of the pixel And the state other than the initial state acquired by the image input unit
Among the image group, the image group acquired in the same time, between pixels
By examining according to the correspondence of
And calculate the similarity measure according to the pixel brightness.
The threshold is compared with the pixel for each index, and the
The similarity measure of one or more indicators is determined by a threshold
And a region detection unit that detects a pixel out of the predetermined range as a pixel whose correspondence is broken.

According to a second aspect of the present invention, there is provided the object extracting method according to the first aspect.
In the invention of out device, the threshold measurement unit, the initial state
For each image group , an average calculation unit that calculates an average value of the similarity measure between pixels according to the correspondence detected by the correspondence detection unit, and a deviation calculation unit that calculates a deviation within an allowable range from the average value calculated by the average calculation unit. And the area detection unit acquires the image
For the image group in a state other than the initial state, a similarity measure between pixels is calculated according to the correspondence detected by the correspondence detection unit, and after subtracting the average value calculated by the average calculation unit from the similarity scale, a shift calculation is performed. Using the shift calculated by the unit as a threshold, a pixel whose absolute value of the subtraction result is larger than the shift is detected as a pixel whose correspondence is broken.

The invention described in claim 3 is the first or second invention.
In the object extraction device of
The detection unit uses the pair detected by the correspondence detection unit as a similarity measure .
A difference value between each index between pixels respond, is characterized by using a value obtained by dividing the luminance value of the one pixel.

[0013]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a block diagram showing a first embodiment of the object extraction device of the present invention.

The object extracting apparatus includes an image input unit 1 for inputting images captured by a stereo camera at different positions, and a correspondence detecting unit 2 for detecting a correspondence between pixels of each image input to the image input unit 1. A correspondence storage unit 3 that stores the coordinates of each pixel whose correspondence has been detected by the correspondence detection unit 2, and each of the pixels input to the image input unit 1 based on the correspondence between the pixels stored in the correspondence storage unit 3. A threshold value measuring unit 4 that measures a threshold value for detecting a corresponding collapse for all pixels of the image in pixel units, and a threshold value measured by the threshold value measuring unit 4 are stored. Threshold storage unit 5 and threshold storage unit 5
The correspondence between the pixels stored in the correspondence storage unit 3 for all the pixels of each image input to the image input unit 1 is detected based on the threshold value stored in And an area detecting unit.

The operation will be described based on the above configuration.
First, as an initial stage, a scene without a moving object (that is, a background image) is photographed by a stereo camera and input to the image input unit 1. The background image input to the image input unit 1 is input to the correspondence detection unit 2 and detects correspondence between pixels of the background image. This correspondence detection is performed by obtaining an epipolar condition from the positional relationship of the stereo camera that captures the background image.
That is, a set of two straight lines formed by intersecting a stereo image with a straight line connecting the camera centers of two stereo cameras and a point on the background scene is an epipolar straight line,
It is a condition that any point in the background is reflected on a pixel on the pair of epipolar straight lines in the stereo image. The corresponding points of the pixels are obtained on all the epipolar straight line pairs of the left and right images. Then, the coordinates of each pixel for which the correspondence has been detected are confirmed and stored in the correspondence storage unit 3.

Next, for each pixel in the above image, luminance, hue, lightness, and index, which are indices for each image of that pixel,
Calculate saturation. The calculation of each index is described in the University of Tokyo Press “Image Analysis Handbook” (supervised by Mikio Takagi), p. 103, p
Utilize the technique described in pages 486-489. Then, the difference value of each index between the associated pixels is calculated. Thereafter, a value obtained by adding a constant value (a value of about 1/100 of the entire scale) to the difference value is determined as the threshold value of each index. The above is the processing of the threshold value measuring unit 4. Then, the threshold value of each index determined by the threshold value measurement unit 4 is stored in the threshold value storage unit 5.

After the above settings are made, the input image of the image input unit 1 is input to the area detection unit 6. Area detector 6
Then, the area of the object other than the background is detected by using the coordinates of the pixels having the correspondence stored in the correspondence storage unit 3. This will be described with reference to FIG. When the correspondence of the background image at the initial stage, is about the point P _b backgrounds, and the pixel P _r of the left image pixels P _l and the right image is supported. Object O
When taken, if the point P _b is hidden by the object, P _l and P _r different points of it are points on the object reflects the P
_ol, P _or next to, not correspond. Here, the following calculation method is used to determine whether or not the correspondence has been made. (1) the luminance of the pixel P _l and P _r, calculates the hue, lightness, a difference of saturation. (2) When at least one of the calculated similarity measures is larger than the threshold value stored in the threshold value storage unit 5, it is determined that the correspondence is broken. (3) If not, it is determined that the correspondence is taken. As described above, the threshold value is set in the pixel unit in the threshold value measuring unit 4, the threshold value is extracted in the area detecting unit 6 with reference to the pixel value in the pixel unit, Since the threshold value is determined and the region detection unit 6 performs extraction using each index, highly accurate extraction can be performed.

FIG. 2 shows a second embodiment of the object extracting apparatus according to the present invention.
It is a block diagram showing an embodiment. In the second embodiment, the threshold value measuring unit 4 includes an average calculating unit 41 and a shift calculating unit 42, and the threshold storing unit 5 includes an average storing unit 51 and a shift storing unit 52. The feature is that it is done. Here, the average calculation unit 41 calculates each similarity measure between corresponding pixels and calculates an average value thereof, and the shift calculation unit 42 is allowed from the average value obtained by the average calculation unit 41. This is to calculate the range deviation. Further, the threshold value storage unit 5 stores the average value obtained by the average calculation unit 41, and the shift storage unit 52 stores the shift obtained by the shift calculation unit 42. FIG.
The same reference numerals are given to the same parts as those described above, and the duplicate description will be omitted.

The operation of the above configuration will be described. The processing up to the correspondence storage unit 3 is the same as in the first embodiment. In the second embodiment, after the processing up to the correspondence storage unit 3 is finished, N × 2 (N = 10 is sufficient in practical use) stereo images of frames without moving objects. An image is taken and input to the threshold value measuring unit 4 composed of the average calculating unit 41 and the shift calculating unit 42. Average calculator 41
In each of the input N × 2 images, the indices of luminance, hue, lightness, and saturation are calculated for all pixels.
Then, an average value of N differences between the luminance, hue, lightness, and saturation of the pixels in the corresponding relationship is calculated. Then, the result of the calculation is stored in the average storage unit 51.

The shift calculator 42 calculates differences in luminance, hue, lightness, and saturation of the corresponding pixels in all of the input N images, and calculates the differences based on the average values. Calculate the standard deviation of. Then, a value three times the standard deviation is stored in the shift storage unit 52. This is because the noise shift assumes a normal distribution,
The probability that the deviation from the average is more than three times the standard deviation is
This is because it is as small as 0.0027. Here, the value of this deviation is obtained using the standard deviation.
For each of the images, the deviation from the average value may be calculated, and the maximum value of the deviation may be sent to the deviation storage unit 52.

The area detection section 6 determines the object area as follows. The other processes in the area detection unit 6 are the same as those in the first embodiment. (1) the luminance of the pixel P _l and P _r, calculates the hue, lightness, a difference of saturation. (2) The average storage unit 5 for the calculated similarity measure
The absolute value of the difference from the value stored in 1 is calculated. (3) Then, the value of the calculated absolute value is shifted
If there is at least one index that is larger than the threshold value stored in, it is determined that the correspondence is broken. (4) If not, it is determined that a response has been taken.

As described above, the difference between the average value of each similarity scale calculated by the average calculation unit 41 and the pixel to be determined whether or not to extract and the pixel corresponding thereto is determined by the shift calculation unit 42. By judging whether the object is a moving object or a background based on whether or not the moving object is within the range of the shift due to the noise calculated in step (1), the moving object can be extracted flexibly in response to the difference in texture between the moving object and the background. This means that each similarity measure has a corresponding
This is because it greatly changes depending on the texture around the pixel to be determined whether to extract.

FIG. 3 shows a third embodiment of the object extracting apparatus according to the present invention.
It is a block diagram showing an embodiment. 2 are denoted by the same reference numerals, and duplicate description will be omitted. However, in the third embodiment, a luminance calculation unit 7 is provided between the image input unit 1 and the area detection unit 8. And a threshold determining unit 8 is provided.

Here, the luminance calculation section 7 calculates the luminance of at least one of the images input from the image input section 1, and the threshold value determination section 8 stores the luminance in the average storage section 51. The luminance value calculated by the luminance calculation unit 7 is the smallest among the combinations of the average luminance and the average value of each similarity scale and the combination of the luminance and the average deviation value stored in the deviation storage unit 52. The average value and the deviation value corresponding to the near luminance are selected.

The operation of the above configuration will be described. The processing up to the correspondence storage unit 3 is the same as in the second embodiment. Therefore, only the parts different from the second embodiment will be described. In the third embodiment, after the processing up to the correspondence storage unit 3 is completed, a frame without a moving object is photographed. At this time, the brightness of the lighting is set in a plurality of levels (M levels).
While photographing, N images are photographed and input to the threshold value measuring unit 4 composed of the average calculating unit 41 and the shift calculating unit 42 (in practice, about M = 3, N = 10 is sufficient. Is.). However, in such a case, one image with the same illumination as that for shooting a scene including a moving object is shot, and two images, a slightly darker image and a slightly brighter image, are shot.

In the average calculation section 41, the inputted M × N ×
In all of the two stereo images, luminance, hue, lightness, and saturation are calculated for all pixels. Then, in each luminance stage, an average value of N difference values of luminance, hue, lightness, and saturation of the pixels having the corresponding relationship is calculated. When the result of this calculation is stored in the average storage unit 51, the luminance and the average value of the pixels of one image (here, the left image but may be the right image) are stored in association with each other.

In the shift calculator 42, the input M × N ×
In all of the two stereo images, the luminance, hue, lightness, and saturation are calculated for all the pixels, and the respective standard deviations are calculated based on the above-described average values.

Then, the luminance of the pixel of the left image and the value of three times the standard deviation thereof are stored in the shift storage unit 52 in association with each other. In this case as well, the value of the shift is obtained using the standard deviation. However, the shift from the average value is calculated for each of the N images, and the maximum value of the shift is stored in the shift storage unit 5.
2 may be stored.

Next, the luminance calculator 7 calculates the luminance value of the pixel (P _{l in} FIG. 5) corresponding to the determination of the object area. Then, the threshold value determination unit 8 refers to the luminance value obtained by the luminance calculation unit 7 and refers to the M average values stored in the average storage unit 51 and the M average values stored in the shift storage unit 52. From the deviation values, an average value and a deviation value corresponding to the luminance closest to the luminance value calculated by the luminance calculation unit 7 are selected,
This is sent to the area detection unit 6.

The average value and the shift value are selected from those corresponding to the two luminance values closest to the luminance value calculated by the luminance calculation unit 7, and finally determined by a linear sum of them. Is also good. Specifically, each index corresponding to the luminance value of the pixel to be determined whether to extract in the left image I _0, two near I ₀ intensity I _5, I _l mean, deviation Assuming that the values are a ₅ , d ₅ , a _l , and d _l (where I ₅
<I ₀ <I _l ), an average value a ₀ used as a threshold,
The shift value d ₀ is obtained by the following equation.

[0031]

(Equation 1)

In the area detector 6, the determination of the object area is performed as follows. (1) the luminance of the pixel P _l and P _r, calculates the hue, lightness, a difference of saturation. (2) With respect to the calculated similarity measure, the absolute value of the difference from the average value sent from the threshold value determination unit 8 is calculated. (3) If the calculated absolute value is larger than the threshold value sent from the threshold value determination unit 8 by one similarity measure, it is determined that the correspondence is broken. (4) If not, it is determined that a response has been taken. Thus, the object area can be detected.

FIG. 4 shows a fourth embodiment of the object extracting apparatus according to the present invention.
It is a figure showing an embodiment. In the third embodiment,
As an initial setting, shooting is performed while changing the brightness in a frame without a moving object, and it is necessary to set a threshold value according to the brightness. In the fourth embodiment, in the second embodiment, By using a value obtained by dividing by a luminance value of a pixel for determining whether to extract a difference value of each index as a similarity measure, an effect similar to that obtained in the third embodiment can be obtained. I have.

To be more specific, FIG. 4 shows the change due to the luminance obtained by dividing the difference value between the corresponding and non-corresponding pixels by the luminance value when there is the texture shown in FIG. However, according to this, it is understood that this index does not change much within a certain luminance range (0.1-0.7). This is because the amount of change in the luminance due to the texture change is large when the image is bright and small when the image is dark, assuming that the change in the texture changes at a fixed rate with respect to the luminance. Therefore, when the background and the moving object are within the range of the luminance (0.1-0.7), only the similarity scale is used, and the luminance is changed according to the luminance as in the third embodiment. Even without setting a threshold,
Without being affected by changes in brightness in the background / animals,
The animal body can be extracted.

[0035]

As is apparent from the above description, the claims
According to the first aspect of the present invention, the luminance of a pixel to be determined whether or not to extract is calculated by the luminance calculation unit, and based on the calculated value, the threshold value determination unit calculates the average value of each similarity scale, Is determined and the moving object is extracted, so that the moving object can be extracted without being affected by the brightness of the similarity scale.

According to the second aspect of the present invention , the threshold value measuring section measures the threshold value for a plurality of images, and the area detecting section extracts the threshold value based on the threshold value. Can be extracted.

According to the third aspect of the present invention, a value obtained by dividing a difference value of each index by a luminance value of a pixel for determining whether to extract the index value is used as a similarity measure. It is possible to extract a moving object that is not affected by the brightness of the similarity measure without changing.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a block diagram showing a second embodiment of the present invention.

FIG. 3 is a block diagram showing a third embodiment of the present invention.

FIG. 4 is a graph showing a fourth embodiment of the present invention.

FIG. 5 is an explanatory diagram of a conventional object extraction method using a stereo image.

FIG. 6 is a graph showing the influence of the brightness and the texture change of the difference value between the corresponding pixel and the non-corresponding pixel in the conventional object extraction method.

[Description of Signs] 1 Image input unit 2 Correspondence detection unit 3 Correspondence storage unit 4 Threshold measurement unit 5 Threshold storage unit 6 Area detection unit 7 Luminance calculation unit 8 Threshold determination unit 41 Average calculation unit 42 Deviation calculation Unit 51 average storage unit 52 shift storage unit

Claims (3)

(57) [Claims] An image group obtained by imaging at a plurality of different positions according to a plurality of lighting conditions does not include a moving object.
An initial state and an image input unit acquired in a state other than the initial state; and an image group in the initial state acquired by the image input unit.
That is, a correspondence detection unit that detects correspondence between pixels between image groups acquired under arbitrary lighting conditions, and a pixel that constitutes at least one image of the image group acquired by the image input unit. a luminance calculation unit for calculating the brightness of, according to the correspondence between the pixels the correspondence detection unit detects,
Brightness, and threshold measurement unit that measures at the pixel unit 1 or more thresholds of similarity measure is defined by an index for each luminance value of the pixel of the color or the like, the said image input unit obtains Images in states other than the initial state
Of the image group, the image group acquired in the same time, between the pixel
By examining the correspondence according to
Calculated in pixel units for each index, and the similarity scale is calculated as
The threshold value according to the luminance of the pixel and the pixel unit
And rank one or more indicators in the result of the comparison.
An area detection unit that detects a pixel whose similarity measure is out of the range defined by the threshold value as a pixel whose correspondence is broken. 2. The method according to claim 1, wherein the threshold value measuring unit is configured to determine the initial state.
For each image group , an average calculation unit that calculates an average value of the similarity measure between pixels according to the correspondence detected by the correspondence detection unit, and a deviation calculation that calculates a deviation that is an allowable range from the average value calculated by the average calculation unit. And an area in which the area detecting section is in an initial state acquired by the image input section.
After calculating the similarity measure between pixels according to the correspondence detected by the correspondence detection unit, and subtracting the average calculated by the average calculation unit from the similarity measure, Using the deviation calculated by the calculation unit as a threshold,
2. The object extracting apparatus according to claim 1, wherein a pixel whose absolute value of the subtraction result is larger than the deviation is detected as the pixel whose correspondence is broken. 3. The apparatus according to claim 2, wherein said threshold value measuring section and said area detecting section are provided.
The output unit is detected by the correspondence detection unit as the similarity measure.
The object extraction device according to claim 1, wherein a value obtained by dividing a difference value of each index between corresponding pixels by a luminance value of one of the pixels is used.