JP2001352440A - Image processing unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing unit that can cut out an area, even if the background color is equal to the color of the cutout area. SOLUTION: The image processing unit comprises an image pickup means, consisting of an image pickup control section 103 and a CCD 104 that conduct consecutive shots by changing image pickup conditions; a discrimination means consisting of a focal distance estimate section 108, that calculates the feature quantity with respect to an object from the pixel values of images in each of two image areas and discriminates whether the two image areas belong to the same object on the basis of the feature quantity, a strobe image generating section 107, a refractive index estimate section 109, a strobe distance estimate section 110, and an area extension section 112; an area generating means consisting of an area extension section 112 that generates an area consisting of pixels, belonging to the same object on the basis of the discrimination of the discrimination means, and a selection means consisting of a foreground extract section 111 that selects the area-generating result, corresponding to the object satisfying a prescribed condition.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus having an area cutout unit for extracting an image area belonging to a specific subject from an image.

[0002]

2. Description of the Related Art With the spread of digital cameras, there is an increasing need for various types of image processing for captured images, including a process of combining a captured image with another image. For example, taking a portrait and leaving the background of the person as it is as another image is an example. In the case of such a combining process, it is necessary to specify an area to be combined with each other to form a combined image. Conventionally, the area is specified by a method of automatically removing the background using a so-called blue background, or a method of specifying a cutout range by a retouch tool with an operator.

[0003] Most of these prior arts perform clipping by relying only on the difference between the color information of the region to be clipped and the color information of the background. For example, Japanese Patent Application Laid-Open No. 4-13317
In Japanese Patent Application Publication No. 9-29, after the number of colors in an image is quantized and reduced by clustering, the image is divided into regions by expanding a region by a region expansion method until a pixel is encountered starting from a point without an edge, There is disclosed an image clipping method in which only a necessary part is selected. Also,
Japanese Unexamined Patent Application Publication No. 10-2103, which does not rely on color information
No. 40 discloses an area extraction method using a change in illumination. This method separates a background with a small change in illumination and a foreground that changes greatly by extracting a region that changes between an image with illumination and an image without illumination.

[0004]

However, these conventional techniques have the following problems. First, in the clipping technique relying solely on color information, first, the color of the region to be clipped and the region not to be clipped must be different, and if the difference is small, various parameters related to the clipping algorithm are frequently changed. Had to adjust.
Second, in the case of automatic clipping, there is a problem that the background must be a specific color in order to explicitly specify the background, and a special environment must be prepared. Third, when a human intervenes, depending on the image, a portion to be clipped may be made up of a very large number of colors, or a region that is fragmented as a result of processing according to the prior art may result in a human being. However, there is a problem that it is necessary to pick up a large number of small areas, which is troublesome. Further, in the conventional technology based on only the illumination change, if the subject to be cut out wears clothes with low reflectivity such as black pants, or if the subject itself has severe irregularities and a shadow is formed, the area to be extracted is the subject in the foreground. There was a problem that there was no consistent guarantee.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems in the conventional region clipping technique, and an image processing apparatus capable of clipping even if the background color and the color of the clipped region match. The purpose is to provide.
It is another object of the present invention to provide an image processing apparatus capable of automatically cutting out parts other than the background even when the background is composed of various colors. It is another object of the present invention to provide an image processing apparatus capable of selecting a coherent area that is not subdivided when a human intervenes, thereby reducing annoyance. Still another object of the present invention is to provide an image processing apparatus capable of performing stable clipping independent of the state (color, shape) of the surface of a subject.

[0006]

According to a first aspect of the present invention, there is provided an image pickup means for obtaining a plurality of images by performing continuous shooting while changing image pickup conditions, and two image areas. Determining means for calculating a feature amount related to a subject from pixel values of the plurality of images in each of the plurality of images, and determining whether two image areas belong to the same subject based on the feature amounts; Area generating means for generating an area consisting of pixels belonging to the same subject based on the determination by the determining means; and area selecting means for selecting a generation result of the area generating means corresponding to a subject satisfying a predetermined condition. It constitutes an image processing apparatus.

The first and second embodiments described below correspond to the invention according to claim 1, and the image pickup means includes the image pickup control unit 103, CCD 104, and CCD of the first embodiment.
The imaging control unit 206 and the CCD 203 of the second embodiment correspond to each other, and the discriminating means includes a focus distance estimating unit 108, a strobe image generating unit 107, and a reflectance estimating unit of the first embodiment.
109, a strobe distance estimating unit 110, an area expanding unit 112, and a strobe image generating unit 107, a reflectance estimating unit 109, a strobe distance estimating unit 110, and a personal computer 208 according to the second embodiment. Area expansion unit according to the first embodiment
112, and the personal computer 208 of the second embodiment corresponds to
The selection means corresponds to the foreground extraction unit 111 of the first embodiment and the personal computer 208 of the second embodiment.

[0010] In the invention according to claim 1 configured as described above, the imaging means obtains a plurality of images by continuous shooting, and the determination means calculates the feature amount of the image area from the information of the plurality of images. A discriminant function for determining the similarity between the two image regions is generated based on the discrimination function. The area generating means generates an area composed of pixels belonging to the same object based on the discriminant function, and the selecting means extracts an area corresponding to the object satisfying a predetermined condition from the area. Alternatively, the selection unit specifies a pixel or a region corresponding to a subject satisfying a predetermined condition, and the region generation unit generates a region belonging to the same subject as the specified pixel or region based on a discriminant function.

As described above, according to the first aspect of the present invention, a discriminant function for judging whether two image areas belong to the same subject is generated based on a plurality of images obtained by continuous photographing, thereby providing one function. A more accurate discriminant function can be created than when only images are used, and the burden on the user at the time of shooting is small. Then, a region corresponding to each subject is generated with high accuracy based on the discriminant function, and by selecting the region with conditions, clipping of the specific subject can be realized.

According to a second aspect of the present invention, there is provided an imaging means for obtaining a plurality of images by performing continuous shooting while changing imaging conditions;
A distance estimating means for obtaining an estimated value related to a distance to a subject at each pixel based on the plurality of images; and a statistic regarding a distance estimated value distribution and a pixel value distribution inside each of the two image regions. Is calculated as a feature value, and a discriminating means for generating a discriminant function for determining whether two image areas belong to the same subject based on the feature value, and belonging to the same subject based on the discrimination of the discriminating means. An image processing apparatus comprises an area generating means for generating an area composed of pixels, and an area selecting means for selecting a generation result of the area generating means corresponding to a subject satisfying a predetermined condition.

The invention according to claim 2 corresponds to the first and second embodiments also described below. The image pickup means includes the image pickup control section 103 of the first embodiment and a CCD.
104, the imaging control unit 206 of the second embodiment, the CCD 2
03, the distance estimating means includes a focus distance estimating unit 108, a strobe image generating unit 107, a reflectance estimating unit 109, a strobe distance estimating unit 110 according to the first embodiment, and a strobe according to the second embodiment. Image generator 107, reflectance estimator 10
9, the strobe distance estimating unit 110, the discriminating unit corresponds to the area expanding unit 112 of the first embodiment, and the personal computer 208 of the second embodiment corresponds to the discriminating unit. The area expansion unit 112 of the second embodiment corresponds to the personal computer 208 of the second embodiment, and the selection means corresponds to the foreground extraction unit 111 of the first embodiment and the personal computer 208 of the second embodiment. ing.

According to the second aspect of the present invention, the distance estimating means calculates the distance estimation value from the plurality of images obtained by the imaging means, and the discriminating means calculates the distance estimation value in the image area and the plurality of distance estimation values. The statistic of the distribution is determined as the feature value for each pixel value of the sheet image. Then, a discriminant function for judging whether or not the two image areas are composed of the same subject is generated based on the feature amount. The area generating means generates an area including pixels belonging to the same subject based on the discriminant function,
The selection unit extracts a region corresponding to a subject satisfying a predetermined condition from the selection unit. Alternatively, the selection unit specifies a pixel or a region corresponding to a subject satisfying a predetermined condition, and the region generation unit generates a region belonging to the same subject as the specified pixel or region based on a discriminant function.

As described above, according to the second aspect of the present invention, it is possible to use the distance information to determine whether or not the object belongs to the subject with the background of physical knowledge. A highly accurate discriminant function can be configured.

According to a third aspect of the present invention, in the image processing apparatus according to the first aspect, the discriminating means has a reliability evaluation means for evaluating the reliability of the feature quantity, And the priority of the feature amount used for determining whether or not the same subject is present is determined in accordance with the first and second aspects of the present invention. In the image processing apparatus, the determination unit includes:
It has a reliability evaluation means for evaluating the reliability of the feature amount,
According to another aspect of the present invention, the priority of the feature amount used to determine whether or not the same subject exists is determined in accordance with the evaluation of the reliability evaluation means.

The invention according to claims 3 and 4 corresponds to the first embodiment described below, and the reliability evaluation means includes a focus distance estimating unit 108 according to the first embodiment.
And the area extension unit 112. The feature quantity in claim 4 is the distance estimation value of each pixel itself.

In the third and fourth aspects of the present invention, a reliable feature quantity is preferentially used as a basis for the determination by the determining means in accordance with the evaluation by the reliability evaluating means. Even when there is an error in the calculated feature amount, since the degree of error is evaluated by the reliability evaluation unit and the feature amount which is likely to have an error is not used, erroneous determination by the determination unit is reduced, and stable determination can be performed. .

According to a fifth aspect of the present invention, in the image processing apparatus according to the first aspect, the discriminating means includes subject feature estimating means for estimating a range in which the feature amount can be taken in an image area corresponding to the subject. Wherein when the ranges to which the feature amounts of the two image regions belong are the same, it is determined that the two image regions belong to the same subject. The invention is claimed in claim 2
In the image processing apparatus according to the above, the discriminating means includes subject feature estimating means for estimating a range in which the feature amount can be taken in an image region corresponding to the subject, and the ranges to which the feature amounts of the two image regions belong are the same. In this case, the two image areas are determined to belong to the same subject.

The invention according to claims 5 and 6 corresponds to the first and second embodiments described below. The foreground extraction unit 111 according to the first embodiment includes a subject feature estimating unit. And the area expansion unit 112, and the personal computer 208 and the slider 211a of the second embodiment.

As described above, according to the fifth and sixth aspects of the present invention, two image areas are determined based on whether or not the characteristic amount belongs to a specific range. , The characteristics can be reflected, and more reliable determination can be made.

According to a seventh aspect of the present invention, in the image processing apparatus according to the sixth aspect, the subject feature estimating means approximates a histogram of distance estimation values within a predetermined area of the image by a sum of a predetermined number of basic distributions. , Wherein a range in which each basic distribution has a value equal to or greater than a predetermined threshold value is set as a range of the estimated distance value occupied by each subject.

The invention according to claim 7 corresponds to the first embodiment described below. In this invention, the distance distribution histogram is approximated by the sum of a plurality of basic distributions, and based on this, The method of setting the distance range by
In particular, in a scene in which a subject only exists discretely in the depth direction, the distance range in which each subject exists can be reliably estimated, and as a result, the cutting accuracy can be increased.

According to an eighth aspect of the present invention, in the image processing apparatus according to the fourth aspect, a plurality of the distance estimating means are provided, and the discriminating means includes a plurality of the distance estimating means for each pixel. It is characterized in that the evaluation means is configured to preferentially use a distance estimation value determined to be most reliable.

The invention according to claim 8 corresponds to a first embodiment also described below. A plurality of distance estimating means include the focus distance estimating unit 108 in the first embodiment, and a strobe light. Image generator 107, reflectance estimator 10
9. Strobe distance estimation unit 110 and focus distance estimation unit 1
08 corresponds to the reliability evaluation means, and the area expansion unit 112 in the first embodiment corresponds to the reliability evaluation means.

In the invention according to claim 8, the reliability estimating means evaluates a plurality of estimated values obtained for each pixel by the plurality of distance estimating means, and determines that the discriminating means is most reliable. The determination is performed using the distance estimation result obtained.

In this way, in the invention according to the eighth aspect, by providing a plurality of distance estimating means, even if the reliability of one distance estimating means is reduced, the other distance estimating means does not reduce the reliability under the condition. The distance estimation value of the estimating means can be used, and the accuracy of the discriminant function can be improved.

According to a ninth aspect of the present invention, in the image processing apparatus according to the second aspect, the imaging means is configured to perform continuous shooting under a condition in which a strobe emits light and a condition in which the strobe does not emit light, and the distance estimating means includes: A difference means for obtaining an image illuminated only with strobe light from a difference between two continuous shot images; and a strobe distance estimating means for correcting an image illuminated with only the strobe light to obtain a distance estimation value. It is a feature.

The ninth aspect of the present invention corresponds to the first and second embodiments also described below. The difference means is provided by the strobe image generation unit in the first and second embodiments. 107 corresponds to the strobe distance estimating means, and corresponds to the strobe distance estimating unit 110 of the first and second embodiments.

According to the ninth aspect of the present invention, the imaging means performs continuous shooting under the condition that the strobe emits light and the condition that the strobe does not emit light, and the difference means determines only the strobe light from the difference between the two continuous shot images. And the strobe distance estimating means corrects the image illuminated only by the strobe light to obtain a distance estimation value.

As described above, according to the ninth aspect of the present invention, since an image illuminated only by the strobe light is obtained by the difference means, the fact that the strobe light is attenuated according to the distance from this image is used for each image. The distance for each pixel can be estimated,
Pixels can be easily classified for each subject under specific conditions. Further, the strobe distance estimating means further corrects the estimated value, so that the reliability can be improved.

According to a tenth aspect of the present invention, in the image processing apparatus according to the ninth aspect, the strobe distance estimating means obtains the brightness of an image illuminated only by the strobe light under the condition that no strobe light is emitted. The distance estimation value is obtained by performing a correction by multiplying by a correction coefficient that is inversely proportional to the luminance of the image.

The invention according to claim 10 corresponds to the following first and second embodiments. In this invention, the strobe distance estimating means is illuminated only by strobe light. The distance estimation value is obtained by correcting the luminance of the image by multiplying by a correction coefficient inversely proportional to the luminance of the image obtained under the condition that the strobe is not emitted.

As described above, according to the tenth aspect of the present invention, the image under the condition that the strobe does not emit light is a good estimated value of the reflectance if the illumination is close to parallel light and there is no uneven illumination. By correcting the luminance of an image illuminated only with light, it is possible to mitigate the influence of the color of clothes of a human subject when estimating a distance, for example.

According to an eleventh aspect of the present invention, in the image processing apparatus according to the second aspect, the image pickup means is configured to perform continuous shooting while shifting a focus position, and the distance estimating means is provided with different focus positions. It is characterized in that it has a focus distance estimating means for obtaining a distance estimation value from the continuously shot images.

The invention according to claim 11 corresponds to the first embodiment described below, and the focus distance estimating means corresponds to the focus distance estimating unit 108 in the first embodiment. I have.

In the eleventh aspect of the present invention, the imaging means shifts the focus position to perform continuous shooting,
The focus distance estimating means obtains a distance estimation value from the continuous shot images having different focal positions. Therefore, it is possible to estimate the distance of the edge portion by a simple method of changing the focus position and taking an image, and unlike the case of using a strobe, it is possible to prevent the influence of uneven illumination or the like.

According to a twelfth aspect of the present invention, in the image processing apparatus according to the second aspect, the discriminating means changes the estimated luminance and the distance between two overlapping image areas between the overlapping area and the other area. Are continuous, and the two image areas are determined to belong to the same subject only when the change amounts of the hue and the saturation are within a predetermined threshold value. is there.

The invention according to claim 12 corresponds to the following first and second embodiments. In this invention, the discriminating means is provided for two overlapping image areas. It is assumed that two image regions belong to the same subject only when the change of the luminance and the distance estimation value between the overlap region and the other region is continuous, and the change amount of the hue and the saturation is within a predetermined threshold. I am to judge.

In a general object, the hue is relatively stable in the same material region, but the luminance greatly varies depending on the direction of the surface. Therefore, the luminance varies greatly even within the same subject. According to the twelfth aspect of the present invention, since only the change rate is limited only with respect to the luminance, it is possible to reduce the possibility that two image areas of the same material in the same subject belong to another subject. be able to.

According to a thirteenth aspect of the present invention, in the image processing apparatus according to the second aspect, the area generating means generates, as a basic area, a connected area consisting of pixels determined to belong to the same subject by the determining means. And a region expansion unit that extracts pixels determined to belong to the same subject as a plurality of pixels on the outline of each connected region by the determination unit and adds the extracted pixels to the basic region. It is characterized by the following.

The invention according to claim 13 corresponds to a second embodiment described below. The personal computer 208 and the slider 211a according to the second embodiment correspond to the basic area generating means. The personal computer 208 corresponds to the area expanding means.

In the invention according to the thirteenth aspect, first, the basic area generating means generates a connected area composed of pixels belonging to the same subject as the basic area using the determination result of the distance determining means. Next, based on the judgment of the discriminating means, the area expanding means searches each contour pixel of each basic area for a pixel belonging to the same subject as the contour pixel outside the basic area. After the search has been completed for all the contour pixels, the region expanding means selects only pixels determined to be the same subject as a predetermined number or more of the contour pixels from the searched pixels, and adds the selected pixels to the basic region. This operation is repeated to obtain a region corresponding to the subject.

Thus, in the invention according to the thirteenth aspect, by forming an area based on only the distance information as the basic area, a basic area which does not depend on the surface color of the subject and is difficult to be subdivided is generated. Since this area is extended based on the high-precision determination of the determination means, it is possible to cope with a case where the distance information contains an error and the contour of the basic area does not match the contour of the subject. Moreover, since expansion is performed using only external pixels determined to be the same as many contour pixels in the basic area at the time of expansion, the risk of extending beyond the outline of the subject is reduced.

According to a fourteenth aspect of the present invention, in the image processing apparatus according to the first or second aspect, the area generating means determines a circumscribed pixel adjacent to the outside of a basic area composed of pixels determined to belong to the same subject. Circumscribing pixel extracting means for extracting,
An inner / outer determination unit for evaluating a probability that each circumscribed pixel extracted by the circumscribed pixel extraction unit belongs to a basic region; and a circumscribed pixel evaluated as having the highest probability by the inner / outside determination unit is newly added to the basic region. It is characterized by having such a configuration.

The invention according to claim 14 corresponds to the first embodiment described below, wherein the circumscribed pixel extraction means and the inside / outside determination means are provided with the area extension unit of the first embodiment.
112 are supported.

In the invention according to claim 14, the circumscribing pixel extracting means extracts a pixel set circumscribing the designated basic area, and the inside / outside judging means sets each pixel of the circumscribing pixel set to the basic area. Evaluate the probability of belonging. And
The area generating means expands the basic area by newly adding the circumscribed pixel evaluated as having the highest probability by the inside / outside determining means to the basic area.

As described above, in the invention according to claim 14, a pixel having the highest probability of belonging to the basic region among the circumscribed pixels of the basic region is newly added.
Pixels that do not belong to the same subject are added to the basic region, and the possibility of occurrence of a breakdown such as the basic region suddenly protruding greatly from the region of pixels belonging to the same subject is reduced.

According to a fifteenth aspect of the present invention, in the image processing apparatus according to the fourteenth aspect, the inside / outside determination means includes a pixel set belonging to a basic area in the vicinity of the circumscribed pixel and the circumscribed pixel itself. It is configured to calculate the probability that the circumscribing pixel belongs to the basic region based on both the determination result and the determination result of the determination unit for the two regions of the circumscribing pixel and the pixel set belonging to the outside of the basic region in the vicinity of the circumscribing pixel. It is characterized by having.

The invention according to claim 15 corresponds to the first embodiment described below. In this invention, the inside / outside determination means includes a pixel set belonging to the vicinity of the circumscribed pixel and belonging to the inside of the basic area. Extract a pixel set belonging to the outside of the basic region, determine whether the circumscribed pixel itself belongs to each region,
The probability that the circumscribed pixel belongs to the basic region is calculated based on the two determinations.

Thus, in the invention according to claim 15, it is determined whether or not the circumscribed pixel is included in the basic region in addition to whether the circumscribed pixel is included in the basic region. In order to calculate the probability of being included in the region, it is difficult to judge the difference in color and distance information between the inside and outside of the basic region, preventing the probability that the circumscribing pixel is included in the basic region from being overestimated, Enables accurate inside / outside judgment.

According to a sixteenth aspect of the present invention, in the image processing apparatus according to the seventh aspect, the area selecting means includes a distance estimating unit which is set by the subject feature estimating means among the areas generated by the area generating means. It is characterized in that it is configured to select a region corresponding to a subject corresponding to the closest range among the value ranges.

The invention according to claim 16 corresponds to a first embodiment described below. In the present invention, the area selecting means determines the distance estimation value set by the subject feature estimating means. The closest range is specified, and a subject corresponding to this range is regarded as a foreground subject. Then, an area corresponding to the foreground subject is selected from the areas generated by the area generating means. Therefore, by deciding to cut out the foreground subject in this way, it becomes possible to automate the work of cutting out the foreground such as a portrait.

According to a seventeenth aspect of the present invention, in the image processing apparatus according to the seventh aspect, the area selecting means has a mask image generating means for forming a mask image representing the selected area, and the image processing means obtains the mask image. An image obtained under a standard imaging condition is selected from the plurality of obtained images and output together with a mask image.

The invention according to claim 17 corresponds to a first embodiment described below.
Foreground extraction unit 111 and area expansion unit 112 in the embodiment
, A mask memory 113 and a recording unit 114, and the area expansion unit 112 and the mask memory 113
Is supported.

In the invention according to claim 17 configured as described above, the mask image generating means creates a mask image having a pixel value of 1 in a selected area and a pixel value of 0 in other areas. The selection unit extracts only an image obtained under standard imaging conditions from the images obtained by continuous shooting, and outputs the image together with the mask image.

As described above, according to the seventeenth aspect of the present invention, the output image is limited to one of continuous shot images, and a mask image as a processing result is added, so that only necessary and sufficient data is output. Therefore, waste of the storage capacity of the output destination can be avoided.

[0056]

Next, an embodiment will be described. FIG. 1 is a block diagram showing a digital camera 100 according to a first embodiment of the image processing apparatus according to the present invention.
As shown in FIG. 1, in a digital camera 100, an optical system 101, a strobe light 102, and a CCD capable of performing focus control are provided.
104 is connected to an imaging control unit 103 for controlling these,
The CCD 104 is connected to an image memory 106 and an imaging control unit 103 via an interpolation unit 105. In the image memory 106,
In addition, a strobe image generator 107, a reflectance estimator 109,
Focus distance estimating unit 108, area expanding unit 112, and recording unit
The strobe image generating unit 107 and the reflectance estimating unit 109 are both connected to a strobe distance estimating unit 110. The strobe distance estimating unit 110 is connected to the foreground extracting unit 111 and the region extending unit 112, and the foreground extracting unit 111 and the focus distance estimating unit 108 are both connected to the region extending unit 112.
The area extension unit 112 is connected to a mask memory 113, and the mask memory 113 is connected to a recording unit 114.

Next, the operation of the digital camera 100 configured as described above will be described focusing on the signal flow. When the user shoots in portrait mode, C
The imaging control unit 103 calculates an in-focus position and an appropriate amount of strobe light based on an imaging signal from the CD 104. Then, the strobe 102 and the optical system 101 are controlled so that the strobe light is emitted and focused, the strobe light is not emitted and the focus is achieved, and the strobe light is not emitted and the focal position is slightly shifted. Imaging is performed under three types of imaging conditions. An image pickup signal from the CCD 104 is converted into a color image by an interpolation unit 105.
Hereinafter, the color images captured under the above three conditions are sequentially referred to as an image I1, an image I2, and an image I3. Image I
1 to I3 are sequentially written to the image memory 106.

When the imaging of the three images I1 to I3 is completed, the flash image generating unit 107 reads out the images I1 and I2 from the image memory 106, and outputs the image I4 that would be obtained when the illumination light is only the strobe light. , Strobe distance estimation unit
Output to 110. Since the image I1 and the image I2 are captured with proper exposure, the exposure amounts are different. Therefore, after correcting the difference in the exposure amount, the image I1 is subtracted from the image I2, and the image I4 having only the strobe light is synthesized. This process is performed for each color component for both the image I1 and the image I2.
In the image I4 using only the strobe light, the pixel value is a function of the subject distance and the orientation of the subject surface. For example, considering the situation as shown in FIG. 2, if the reflectance of the subject 301 is r and the angle between the surface of the subject 301 and the optical axis 303 of the camera 302 is α1, the irradiation direction of the strobe light 304 and the camera optical axis 303 Are substantially the same, the pixel value V4 of the image I4 using only the strobe light 304 as illumination is expressed by the following equation (1). V4 = r × cos (α1) × Ls / d ² (1) where Ls is the brightness of the strobe light and d is the distance to the subject.

On the other hand, the reflectance estimating unit 109 is provided in the image memory 10
6 Estimate the object reflectance from the image in the middle, and calculate the image I5 in which the value of each pixel is correlated with the object reflectance by the strobe distance estimation unit.
Output to 110. In the present embodiment, the image I5 is obtained by replacing the pixel value of a portion having a pixel value of 0 with 1 in the image I2. Considering the situation of FIG. 2 again, the pixel value V2 of the image I2 is represented by the following equation (2), and is proportional to the subject reflectance r except for the direction of the surface. V2 = r × cos (α2) × La (2) where La is the brightness of the light source 305 other than the strobe, and α2 is the brightness of the subject 301. The angle between the surface and another light source 305.

The image I 4 illuminated only by the strobe light is input from the strobe image generation unit 107, and the reflectance estimation unit 109
When the image I5 corresponding to the subject reflectance is input from
The strobe distance estimating unit 110 obtains a value obtained by dividing the pixel value V4 of the image I4 by the pixel value of the image I5 for each color component, and calculates an image I6 having an average value between the color components as a pixel value. From equations (1) and (2), the following equation (3) is obtained.
V2 is inversely proportional to the square of the distance d except for the influence of the surface orientation. V4 / V2 = 1 / d ² × cos (α1) / cos (α2) × Ls / La (3) Hereinafter, the relationship is monotonic with the distance (or its square) or its reciprocal. Are collectively referred to as distance estimates. The strobe distance estimating unit 110 outputs the image I6 to the foreground extracting unit 111 and the area expanding unit 112.

From the strobe distance estimating unit 110 to the foreground extracting unit 11
When the image I6 is input to 1, the foreground extraction unit 111 determines an image area determined to be a foreground. For this purpose, a histogram is created by plotting the distance estimation value on the horizontal axis and the number of pixels having the distance estimation value on the vertical axis, and the histogram is approximated by the sum of two normal distributions. A known algorithm such as the maximum likelihood method is used for the approximation. If the approximation error is equal to or greater than a predetermined value, it is determined that the foreground and the background cannot be properly separated, and an image having all pixels of 0 is generated.

On the other hand, when the approximation error is small, the distance between the foreground and the background is generally large, and the histogram has two peaks corresponding to the foreground and the background as shown in FIG. 3 and a normal distribution as shown in FIG. Thus, the distance estimation value distribution of each of the foreground and the background can be roughly estimated.

Therefore, the foreground extraction unit 111 calculates the range of the distance estimation value having a high probability that the foreground and the background are located based on the respective distance estimation value distributions under the following conditions. Foreground range: left and right ± m1σ1 or less (σ1 is the standard deviation of the foreground distance estimation value distribution) of the distance estimation value corresponding to the peak of the foreground distance estimation value distribution, and the value of the background distance estimation value distribution is β1 times the peak value. Below background range: ± m2σ2 or less of the left and right of the distance estimation value corresponding to the peak of the background distance estimation value distribution (σ2 is the standard deviation of the background distance estimation value distribution), and the value of the distance estimation value distribution of the foreground is β2 of the peak value. Here, β1, β2, m1, and m2 are appropriately set thresholds. For example, in the case of FIG. 4, the range indicated by the arrow is the range of the distance estimation value satisfying each condition. Then, a pixel having a distance estimation value falling within the foreground range has a pixel value of 1,
An image I7 is generated in which a pixel having a distance estimation value falling within the background range is -1 and other pixels have a pixel value of 0.

After the above processing, the generated image I7 is output to the area expanding section 112. In this image I7, the area of the pixel value 1 corresponds to the area determined to be included in the foreground.

In parallel with the processing up to this point, the relative distance of each pixel of the image in the image memory 106 is estimated in the focus distance estimating unit 108, and the image I8 whose pixel value is the estimated distance is used as the area expanding unit. Output to 112. Image memory 10
Images I2 and I3 in FIG. 6 are obtained by capturing the same subject at different focal positions. Known “Depth from defocus”
According to the method, the subject distance can be estimated from two images having different focal positions. In the present embodiment, the image I2 which is a focused image is blurred by Gaussian to the same extent as the image I3 to calculate an image I2 '. After calculating a luminance component for each of the three images, a Laplacian Gaussian filter is applied. , And an edge image E2, E3, E2 ', and an image I8 having a correlation with the subject distance is calculated by the following equation (4). I8 = (E3−E2) / (E3 + E2) × (E2 ′ + E2) / (E2′−E2) (4)

The area extension unit 112 is used to generate images I6, I7, I8
And I2 are input, the foreground region is extended so that the actual contour of the foreground and the contour shape of the foreground region are close to each other. In the present embodiment, the foreground candidate area obtained by using the strobe is extended based on the color information and the distance information obtained by using the strobe, and the distance information obtained by the focus movement and the pixels of the image I7 are supplementarily added. Use values to prevent errors during expansion. Area extension
FIG. 5 shows the flow of the processing in 112. The operation of each stage (steps S1 to S10) of the processing flow is performed as follows.

S1: A pixel set adjacent to the outline outside the outline of the foreground candidate area and having a pixel value of 0 in the image I7 is taken out as B. For example, if the gray area (dot pattern area) in FIG. 6A is a foreground candidate area 401, a pixel set B to be extracted is a shaded portion 402. In the processing flow of S2 and thereafter, one pixel having the highest possibility of being included in the foreground on the pixel set B is determined by the processing flow of S2 to S10, and is added to the foreground candidate area 401 in S12. For the foreground candidate area after the addition, a pixel set adjacent to the contour is calculated again in S1, and the processing flow from S2 is repeated. At the start, the foreground candidate area is the image I
7 is an area (pixel area determined to be foreground by using a strobe light) having a pixel value of 1. Also, a variable named max is initialized to 0. The variable max is used to select a pixel on the pixel set B having the highest probability of belonging to the foreground in the processing flows S2 to S10.

S2: Pixels on the pixel set B are taken out one by one, and the processing flow from S3 to S10 is repeated.

S3: The pixels extracted from the pixel set B are P
And A predetermined size centered on the pixel P, for example, 5 × 5
Is a pixel set included in the foreground candidate area 401 in the neighborhood area 403 of In, and a set of pixels not included and not P is Out. In FIG. 6A, the pixel indicated by the arrow is P, and the surrounding area 5 × 5 is the neighboring area 403.

S4: For each of the sets In and Out, the image I
2, the average values Avg-in (c) and Avg-out (c) and the variance Var-in
(c), Var-out (c) is calculated. c is a three-color component r, g, b
Indicates one of Further, for each of the sets In and Out,
The average values Avg-d-in and Avg-d-out on the image I5 are calculated.

S5: Calculate the probability that the pixel P belongs to each of the sets In and Out. The pixel value of pixel P is (pr, pg, pb)
Then, the probability Prob-in belonging to the set In is expressed by the following equation (5)
a). Prob-in = Π · 1 / ｛2πVar-in (c)｝ ^1/2 · exp [− ｛p (c) −Avg-in (c)｝ ² / 2Var-in (c)] (c = r, g, b) (5a) The probability Prob-out belonging to the set Out is represented by the following equation (5b). Prob-out = Π · 1 / ｛2πVar-out (c)｝ ^1/2 · exp [− ｛p (c) −Avg-out (c)｝ ² / 2Var-out (c)] (c = r, g, b) (5b) In the example of FIG. 6A, the sets In, Out are shown in FIG.
The pixel value distribution in each RGB space is shown in gray, and points corresponding to the pixel values of the pixels P are shown by black circles. In this case, Prob-in> Prob-out, and the probability that the pixel P belongs to the set In is higher.

S6: If Prob-in> Prob-out, it is determined that the pixel P can belong to the foreground, and the flow advances to S7. On the other hand, the process proceeds to S2 assuming that the pixel cannot belong to the foreground, and the next pixel on the pixel set B is extracted.

S7: If the pixel P can belong to the foreground, the reliability of the judgment is evaluated using the distance estimation information obtained by the focus movement. In the image 18 whose pixel values are distance estimation values, the average Dist-in of the pixel values in the region corresponding to the set In and the value Dist-P of the pixel corresponding to the pixel P are calculated.

S8: If both Dist-in and Dist-P are larger than the threshold value Td, the absolute value of the difference between Dist-in and Dist-P is set to d. Otherwise, let the absolute value of the difference between Avg-d-in and Avg-d-out be d. If d is smaller than threshold value Tf, S
Go to 9. If it is larger, the determination that P belongs to the foreground is determined to be incorrect, and the process proceeds to S2.

S9: Probability Prob-i that pixel P is included in the foreground
It is checked whether or not n is larger than the maximum value max of the probability that the pixels examined so far in the pixel set B are included in the foreground. If larger, the process proceeds to S10, and if smaller, the process proceeds to S2.

S10: Update max with Prob-in, and update the variable maxp holding the pixel in which the maximum value is recorded to P.

S11: After examining all the pixels on the pixel set B, the maximum probability max recorded by the pixel maxp having the highest probability of belonging to the foreground is compared with a predetermined threshold value Tp. The process ends because there is not a sufficient probability that any pixel on the set B belongs to the foreground. If it is larger, the process proceeds to S12.

S12: The pixel having the highest probability maxp belonging to the foreground in the pixel set B is added to the foreground candidate area.

According to this processing flow, the pixels for which it cannot be determined whether they belong to the foreground or the background in the distance estimation value distribution obtained by using the strobe light, that is, the pixels having the pixel value 0 on the image I7 are classified. Since the classification is performed mainly based on the color information, the continuity of the area, and the distance estimation value of the edge part due to the focus movement, the foreground area can be extracted with higher accuracy than when only the distance estimation value using the strobe light is used. Further, since the distance estimation value obtained by the focus movement is not affected by the uneven lighting, the foreground candidate area can be expanded without being affected by the uneven lighting.

The foreground candidate area extended in the above processing flow is converted into a mask image having a value of 1 in the foreground candidate area and a value of 0 in other areas by the area expansion unit 112 as a mask image.
Written to 113.

When the writing of the mask image to the mask memory 113 is completed, the recording unit 114 reads out only the image I2 obtained without the flash at the in-focus position from the image memory 106, and stores the image I2 together with the mask image in the internal recording medium. Write.

With the above operation, when the user presses the shutter, three images are read into the image memory 106 by changing the imaging conditions by the imaging control unit 103, and two images having different strobes are used. The foreground candidate area determined as the foreground is extracted by the foreground extraction unit 111 based on the foreground. Next, based on the two images having different in-focus positions, the area expanding unit 112 expands the foreground candidate area to bring the contour of the foreground candidate area closer to the actual foreground contour, and the processing result area is the mask image. Is recorded by the recording unit 114. Since the three images are taken continuously, the user does not need to be particularly conscious of the imaging for extracting the foreground.
Since only one of the images is recorded together with the foreground mask image, the recording medium is not wasted. In the clipping process, a method of using a strobe light method that can roughly extract the distance information of the surface is used to roughly determine the area belonging to the foreground and then expand it, but the distance information by the focus movement method that is not affected by uneven lighting etc. Thus, since the area is expanded in combination with the similarity of the color information while determining the reliability of the distance information of the surface, the accuracy of the area extraction is higher than when each method is used alone. Moreover, since the distance information is used to distinguish between the background and the foreground, the color of the background is not limited, and clipping can be performed even if the colors of the foreground and the background are very similar.

Various modifications and changes can be made to this embodiment. For example, the area expansion algorithm of the area expansion unit 112 does not need to be specified in the processing flow shown in FIG. 5, and may be any method that takes into account the continuity of distance information in addition to the area expansion method. Even in the case of following the processing flow shown in FIG. 5, the neighboring area extracted in S3 is not limited to a fixed shape, but may be a shape along the contour of the foreground candidate area. Alternatively, only the contour of the foreground candidate area may be extended using a known technique such as Snake, and control may be performed so as to converge to the edge at the distance detected by the focus moving method.

Next, a second embodiment will be described. FIG. 7 is a block diagram showing a configuration of the second embodiment. The digital camera 200 according to this embodiment includes an optical system 201, and an optical image is formed on a CCD 203. The digital camera 200 further includes a strobe 202 and an optical system 201.
, The strobe 202 and the CCD 203 are controlled by an imaging control unit 206. The CCD 203 outputs to the image memory 205 via the interpolation unit 204. The image is output from the image memory 205 to the flash image generating unit 107 and the reflectance estimating unit 109. Strobe image generator 107 and reflectance estimator 109
Is output to the strobe distance estimation unit 110. Image memory 20
5 and the strobe distance estimating unit 110 are both connected to a PC (personal computer) 208 via an output terminal 207. PC20
8 has a monitor 209 and a mouse 210 so that the user can perform an interactive clipping operation with the user. In that case, FIG.
Slider 211a ~ 211d and cutout execution button on the screen shown in
It is designed to be operated using 211e.

Next, the operation of the second embodiment will be described along the flow of signals. When the user performs photographing, the imaging control unit 206 calculates an appropriate exposure condition and a strobe light amount based on a signal from the CCD 203. And strobe 20
2 and the optical system 201 are controlled to perform imaging under two types of imaging conditions: a state in which the strobe 202 emits light and a state in which the strobe 202 does not emit light. The image pickup signal is converted into a color image by the interpolation unit 204, and two images are obtained under the above two conditions. The obtained images are sequentially called images I11 and I12. The subscript numbers 11 and 12 specify the imaging conditions.

The images I1 and I2 are sequentially stored in the image memory 205.
Is written to. After the two images are written in the image memory 205, the strobe image generation unit 107 calculates an image using only the strobe light as in the case of the first embodiment. The calculation method is obtained by subtracting the image I12 from the image I11 after correcting the difference in the exposure amount due to the presence or absence of the strobe light. Let the obtained image be I13. Then, the image I13 is output to the strobe distance estimating unit 110, and the image I12 is output to the output terminal 207.

At the same time, the reflectance estimating unit 109 estimates the subject reflectance from the output image of the CCD 203 in the image memory 205, and converts the image I14 in which the value of each pixel is correlated with the subject reflectance into a strobe distance estimating unit. Output to 110. The image I14 is calculated from the image I12 in exactly the same way as in the first embodiment.

When the image I13 is input from the strobe image generating unit 107 and the image I14 is input from the reflectance estimating unit 109, the strobe distance estimating unit 110 calculates an image I15 having a pixel value having a monotonic relationship with the subject distance. Then, the signal is output to the output terminal 207. In the present embodiment, the image I15 is calculated by the following equation (5), as in the first embodiment. I15 = I13 / I14 (5)

Thereafter, the image I15 is output to the output terminal 207 as a distance estimation image. From the output terminal 207, the image I12,
When the output of I15 is completed, the PC 208 displays a screen as shown in FIG. 8 on the monitor 209, and proceeds with the user interactively using the mouse 210. In the example shown in FIG. 8, an image I12 photographed under normal conditions is displayed on the left side, and an image I15 having a subject distance estimated value as a pixel value is displayed on the right side. On the right side of the image I15, a parameter adjustment unit 211 for a clipping process is provided.
Is displayed.

When the screen of FIG. 8 is displayed, the user
The distance range slider 211a in the parameter adjustment unit 211 is adjusted so that the range to be clipped can be roughly selected. Adjustment of the slider and the like is performed by the mouse cursor 211f. When the upper limit value and the lower limit value of the distance estimation value to be cut out are designated by the slider 211a, the PC 208 performs a threshold process on the image I15, and 1 is set at the pixel position where the distance estimation value is between the upper limit value and the lower limit value. Generate an image I16 that takes 0 at other positions,
Display to the right. At the same time, RGB of image I16 and image I12
The product of the components is taken as an image I17, which is displayed on the left. FIG. 9 is a diagram illustrating an example of the result of performing this processing. After this processing is completed, the pixel value of the image I17 displayed on the left is 0
The region expansion based on the color information and the estimated distance value is performed with the region not being the initial region. To adjust for this expansion,
The user uses the luminance change threshold slider 211b and the hue change threshold slider 211c of the parameter adjustment unit 211 to set the threshold Ty for luminance, the threshold Tc for hue and saturation, and the threshold Td for the estimated distance using the distance change threshold slider 211d. Is specified, and an extension execution is instructed by pressing the button 211e. The area expansion is performed by the processing flow shown in FIG.
1 to S10).

S1: Set D is a set of pixels belonging to the initial area. S2: From the set D, a pixel set B belonging to the contour of the region formed by the pixels in D is extracted. Set U is an empty set. S3: In a loop from S4 to S7, a pixel set included in the same subject as each pixel of the contour part pixel set B is extracted. Therefore, it is checked whether all the pixels of the contour part pixel set B have undergone the processing of S4 to S7. When all the processes have been performed, the process proceeds to S8. S4: Let P be an unprocessed pixel in the contour part pixel set B.

S5: A pixel set included in the same subject as the pixel P is searched for and set as E. This processing is performed by a known pixel expansion method. First, each pixel Q in the 3 × 3 neighborhood of the pixel P
Is evaluated by the evaluation function f (P, P, Q) for the similarity with the pixel Q. When it is determined that the pixel Q is similar to the pixel P, it is determined that the pixel Q is included in the same subject and the pixel Q is included in the pixel set E. Add.
Next, a 3 × 3 neighborhood centered on each of the pixels Q newly added to the pixel set E is taken, and the similarity between the pixels R inside the pixel R and the pixels P is determined by an evaluation function f (P, Q, R). Pixels evaluated in the same manner and determined to be similar are newly added to the pixel set E. This process is repeated until there are no more pixels to be newly added to the pixel set E. The evaluation function f (P1, P2, P3) is defined, for example, as follows. f (P1, P2, P3) = 1: | Hue (P1) -Hue (P3) | + | Sat (P
1) When −Sat (P3) | <Tc and | L (P3) −L (P2) | <Ty and | Dist (P3) −Dist (P2) | <Td f (P1, P2, P3) = 0 At other times, P1 is a pixel at which the pixel expansion method is started, P2 is a pixel already determined to be similar to P1, and P3 is a pixel near P2. The functions Hue, Sat, and Dist respectively represent the hue, saturation, and distance estimation value at a given pixel position. According to this evaluation function, P3 and P1 are small when comparing colors ignoring brightness, the rate of change in luminance from P1 to P3 is small, and the distance estimation value from P1 to P3. Are small, it is determined that P1 and P3 are similar.

S6: The pixel set E is added to the pixel set U. When adding, overlapping pixels are left as they are. S7: The pixel P is removed from the contour portion pixel set B, and the process returns to S3.

S8: As a result of processing all the pixels of the contour portion pixel set B, the pixel set U is a set of pixels determined to be similar to any of the contour pixels of P. In FIG. 12, extended regions determined to be similar for each of the outline pixels P1 to P4 are indicated by R1 to R4. Extended areas R1 to R4
Are included in the pixel set U, and the pixels included in the plurality of extended areas are included in the plurality U. For example, point Q
1 appears twice and Q2 appears once in U. The larger the number is, the more included it is in the extended area, and the more likely it is that it is included in the same subject as the area D. In this step, pixels that appear less than N times in the pixel set U are excluded as having low reliability, and only pixels having a sufficiently high possibility of being included in the same subject as the region D are left in the pixel set U.

S9: When the number of pixels included in the pixel set U is 0, it means that the area cannot be expanded any more, and the processing is terminated. S10: Add the pixel set U to the area D, and return to S2.

After performing this processing, the PC 208 sets the pixel values of the pixels included in the expanded area D to the pixel values in the image I12, and sets the values of the pixels not included to the specific color to the right side and the left side to the left side. Display the extension result. As a result, for example, FIG.
As shown in (1), the missing subject region is correctly selected only by the threshold processing based on the distance estimation value. If the extension result is not preferable, the parameter can be adjusted and re-executed, or the extension process can be performed again from the result of the extension once.

With the above operation, when the user presses the shutter, the image is taken by the image taking control section 206 under two conditions of strobe light emission and no light emission.
Image I having a pixel value corresponding to the distance to the subject
15 is generated. This image I15 is monitored by PC208
209, and the user can select a rough area of the subject only by threshold processing, so that the main area of the subject is selected as a combination of several sub-areas, such as when color information is used. No more hassle. After selection of a rough area, area expansion is performed, but since expansion is performed based on color information and subject distance information, there are few errors, and the threshold value for color information can be set loosely,
Here too, segmentation of the area can be prevented.

Various modifications and changes can be made to this embodiment. For example, even if a region including the subject region is roughly cut out by the threshold processing of the distance estimation image, and this region is divided into regions by a known method, the time required for the region division is reduced, and the cut-out region extends outside the background region. Errors are reduced. The present invention is also effective when the entire image is divided into regions, other than cutting out the region of the specific subject. If the pixel expansion method described in the description of S8 in the processing flow shown in FIG. 11 is appropriately selected and repeated, a division with less failure can be obtained as compared with a case where distance information is not used. For example, a method of selecting a start pixel from a pixel having a color with a large color histogram can be considered.

[0099]

As described above with reference to the embodiment, according to the first aspect of the present invention, two image areas are the same based on the feature amount of the subject in a plurality of images obtained by continuous shooting. Since it is configured to determine whether or not the image belongs to the subject, a more accurate discriminant function can be created than when only one image is used, the area corresponding to each subject is generated with high accuracy, and the The cutting can be stably performed without depending on the surface state such as the color. According to the second aspect of the present invention, a statistic relating to a distance estimation value distribution and a pixel value distribution with respect to two image regions based on a plurality of images obtained by continuous shooting is used as a feature amount so that the two image regions are the same subject. Is determined, so that the discriminant function can be generated with higher accuracy.

According to the third and fourth aspects of the present invention, since the reliable feature is preferentially used as the basis for the judgment by the discriminating means in accordance with the evaluation of the reliability of the feature. Even if there is an error in the feature value, the degree of the error is evaluated and the feature value that is likely to have an error is not used.
The erroneous determination of the determination means is reduced, and stable determination can be performed. According to the invention according to claims 5 and 6,
The two image regions can be classified based on whether or not the feature amounts belong to a specific range. If a real subject tends to have a feature range in a physically fixed range, the characteristics can be reflected, and more reliable This makes it possible to perform highly accurate determination. According to the seventh aspect of the present invention, the distance distribution histogram is approximated by the sum of a plurality of basic distributions, and the distance range is set based on the distance distribution histogram. In a non-existing scene, the distance range in which each subject exists can be reliably estimated, and as a result, it is possible to increase the cutting accuracy.

According to the invention of claim 8, the estimated values obtained by the plurality of distance estimating means are evaluated by the reliability evaluation means, and the distance estimation result determined to be most reliable is used. Therefore, the accuracy of the discriminant function can be improved. According to the ninth aspect of the present invention, continuous shooting is performed under the condition that the strobe emits light and the condition that the strobe does not emit light,
Since an image illuminated with only the strobe light is obtained from the difference between the two continuous shot images and corrected to obtain a distance estimation value, an extremely accurate distance estimation value can be obtained. According to the tenth aspect, the luminance of an image illuminated only with strobe light is corrected by multiplying by a correction coefficient inversely proportional to the luminance of an image obtained under a condition in which no strobe light is emitted to obtain a distance estimation value. Thus, the influence of the color of the clothes of the human subject can be reduced, for example, when estimating the distance.

According to the eleventh aspect of the present invention, the continuous shooting is performed while shifting the focus position, and the distance estimation value is obtained from the continuous shot images having different focus positions. Can be estimated, and unlike the case of using a strobe, it is possible to prevent the influence of uneven irradiation or the like. Further, according to the invention according to claim 12, for two overlapping image regions, the change of the luminance and the distance estimation value between the overlapping region and the other region is continuous, and the change amount of the hue and the saturation is small. Since it is configured to determine that two image regions belong to the same subject only when it is within a predetermined threshold, it is determined that two image regions of the same material in the same subject belong to another subject. Possibility can be reduced. According to the invention according to claim 13, a connected area composed of pixels determined to belong to the same subject by the distance determining means is generated as a basic area, and the same subject as a plurality of pixels on the outline of the connected area is generated. Pixels determined to belong to are extracted and added to the basic area to extend the area, and the expansion is performed based on high-precision determination. Can be dealt with when the object matches the contour of the subject. In addition, many contour pixels in the basic area are extended using only external pixels judged to be the same subject. The risk of occurrence can be reduced.

According to the fourteenth aspect of the present invention, the basic region is expanded by newly adding a pixel having the highest probability of belonging to the basic region among the circumscribed pixels of the basic region. Pixels that do not belong to the same subject are added to the region, so that the possibility of occurrence of a failure such as the basic region greatly protruding from the region of pixels belonging to the same subject can be reduced. According to the invention of claim 15,
Since it is determined whether or not the circumscribing pixel is included in the basic region and whether or not the circumscribing pixel is included outside the neighboring basic region, the probability that the circumscribing pixel is included in the basic region is calculated. In addition, it is possible to prevent the probability that a circumscribing pixel is included in the basic region from being overestimated in a portion where there is little difference in color and distance information inside and outside the basic region, and perform accurate determination. According to the invention according to claim 16, the closest range is specified from the range of the distance estimation value set by the subject feature estimating means, and a subject corresponding to the range is selected as a foreground subject. This makes it possible to automate the work of cutting out the foreground such as portraits. According to the invention according to claim 17, a mask image representing a region selected as a foreground subject is formed,
Since only images obtained under the standard imaging conditions are selected from a plurality of images obtained by continuous shooting and output together with the mask image, output of only necessary and sufficient data enables storage of an output destination. Waste of capacity and the like can be avoided.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a digital camera according to a first embodiment of an image processing apparatus according to the present invention.

FIG. 2 is a diagram illustrating a mode in which a subject is illuminated with a strobe light or a light source other than a strobe to capture an image.

FIG. 3 is a histogram showing a subject distance estimated value distribution in an image I6 obtained from a captured image.

FIG. 4 is a diagram showing a distance estimation value distribution of a foreground and a background.

FIG. 5 is a processing flowchart for explaining the operation of an area extension unit in the first embodiment shown in FIG. 1;

FIG. 6 is a diagram showing a pixel value distribution in an RGB space of an image I7 based on a distance estimation value of a captured image, a pixel set In included in a foreground candidate area, and a pixel set Out not included in a foreground candidate area.

FIG. 7 is a block diagram showing a second embodiment.

FIG. 8 is a diagram showing a mode in which an image I12 captured under normal conditions and an image I15 having a subject distance estimated value as a pixel value are shown on the monitor in the second embodiment shown in FIG. 7;

FIG. 9 shows an image I16 obtained by performing threshold processing on an image I15,
FIG. 14 is a diagram showing a mode of a monitor displaying an image I17 obtained by multiplying RGB components of an image I16 and an image I12.

FIG. 10 shows an image obtained by setting an expansion result of an image I16, a pixel value of a pixel included in a region D after the expansion processing to a pixel value of an image I12, and a pixel value of a pixel not included to a specific color. It is a figure which shows the aspect of a monitor.

FIG. 11 is a flowchart illustrating a processing operation of area expansion according to the second embodiment.

FIG. 12 is a diagram illustrating an extended area of each contour pixel.

[Explanation of symbols]

 100 Digital camera 101 Optical system 102 Strobe 103 Image capture controller 104 CCD 105 Interpolator 106 Image memory 107 Strobe image generator 108 Focus distance estimator 109 Reflectivity estimator 110 Strobe distance estimator 111 Foreground extractor 112 Area expander 113 Mask Memory 114 Recording unit 200 Digital camera 201 Optical system 202 Strobe 203 CCD 204 Interpolation unit 205 Image memory 206 Imaging control unit 207 Output terminal 208 PC 209 Monitor 210 Mouse 211 Parameter adjustment unit 211a to 211d Slider 211e button 301 Subject 302 Camera 303 Camera light Axis 304 Strobe light 305 Light source other than strobe light

 ──────────────────────────────────────────────────続 き Continued on the front page F-term (reference) 5B057 AA20 CA01 CA08 CA12 CB01 CB08 CB12 DA08 DB02 DB06 DB09 DC01 DC23 DC30 5C022 AA13 AB15 AB19 AB28 AB68 AC01 AC32 AC42 AC52 AC54 5C076 AA01 AA02 AA13 BA01 BA06 CA10 5C027 PP PP35 PP47 PP58 PQ08 PQ19 SS01 SS07 TT09

Claims (17)

[Claims] An imaging unit that obtains a plurality of images by performing continuous shooting while changing imaging conditions; and calculating a feature amount related to a subject from pixel values of the plurality of images in each of two image regions; Determining means for generating a determination function for determining whether two image areas belong to the same subject based on the feature amount; and generating an area including pixels belonging to the same subject based on the determination by the determination means. Region generating means;
An image processing apparatus comprising: an area selecting unit that selects a generation result of the area generating unit corresponding to a subject satisfying a predetermined condition. 2. An imaging unit for obtaining a plurality of images by performing continuous shooting while changing imaging conditions, and a distance for obtaining an estimated value related to a distance to a subject in each pixel based on the plurality of images. Estimating means for calculating, as a feature amount, a statistic regarding a distance estimation value distribution and a pixel value distribution inside each of the two image regions, and determining whether the two image regions belong to the same subject based on the feature amount. Discriminating means for generating a discriminant function for judging whether or not, an area generating means for generating an area consisting of pixels belonging to the same subject based on the judgment of the discriminating means, and an area corresponding to a subject satisfying a predetermined condition An image processing apparatus comprising: an area selection unit that selects a generation result of a generation unit. 3. The method according to claim 1, wherein the determining unit includes a reliability evaluation unit that evaluates the reliability of the feature amount, and determines whether or not the feature amount is the same subject according to the evaluation of the reliability evaluation unit. The image processing apparatus according to claim 1, wherein the image processing apparatus is configured to determine a priority. 4. The method according to claim 1, wherein the determining unit includes a reliability evaluation unit that evaluates the reliability of the feature amount, and determines whether or not the feature amount is the same subject according to the evaluation of the reliability evaluation unit. 3. The image processing apparatus according to claim 2, wherein the image processing apparatus is configured to determine a priority. 5. The image processing apparatus according to claim 1, wherein the determining unit includes a subject feature estimating unit that estimates a range in which the feature amount can be taken in an image region corresponding to the subject. The image processing apparatus according to claim 1, wherein the image processing apparatus is configured to determine that two image areas belong to the same subject. 6. The image processing apparatus according to claim 1, wherein the determining unit includes a subject feature estimating unit configured to estimate a range in which the feature amount can be taken in an image region corresponding to the subject. The image processing apparatus according to claim 2, wherein the image processing apparatus is configured to determine that two image areas belong to the same subject. 7. The object feature estimating means approximates a histogram of distance estimation values in a predetermined area of an image by a sum of a predetermined number of basic distributions, and determines a range in which each basic distribution has a value equal to or more than a predetermined threshold value. 7. The image processing apparatus according to claim 6, wherein the image processing apparatus is configured to set a range of a distance estimation value occupied by a subject. 8. A plurality of the distance estimating means, wherein the discriminating means gives priority to a distance estimation value determined to be most reliable by the reliability evaluation means among the distance estimation results of the plurality of distance estimating means in each pixel. The image processing apparatus according to claim 4, wherein the image processing apparatus is configured to be used. 9. The image capturing means is configured to perform continuous shooting under a condition where a strobe light is emitted and a condition where no flash light is emitted, and the distance estimating means is illuminated only with a strobe light based on a difference between two continuous shot images. 3. The image processing apparatus according to claim 2, further comprising: a difference unit that obtains an image; and a strobe distance estimation unit that obtains a distance estimation value by correcting an image illuminated only by the strobe light. 10. The strobe distance estimating means corrects the distance by multiplying a luminance of an image illuminated only by the strobe light by a correction coefficient inversely proportional to a luminance of an image obtained under a condition in which the strobe light is not emitted. The image processing apparatus according to claim 9, wherein the image processing apparatus is configured to obtain an estimated value. 11. The image capturing unit is configured to perform continuous shooting while shifting a focus position, and the distance estimating unit includes a focus distance estimating unit that obtains a distance estimated value from continuously shot images having different focal positions. The image processing apparatus according to claim 2, wherein: 12. The determining unit according to claim 2, wherein, for two overlapping image regions, a change in luminance and a distance estimation value between the overlap region and another region is continuous, and a change amount of hue and saturation is predetermined. The image processing apparatus according to claim 2, wherein the image processing apparatus is configured to determine that two image areas belong to the same subject only when the values are within the threshold value. 13. A basic region generating unit configured to generate, as a basic region, a connected region including pixels determined to belong to the same subject by the determining unit, and a region generating unit configured to generate each of the connected regions by the determining unit. 3. The image processing apparatus according to claim 2, further comprising an area expanding unit that extracts a pixel determined to belong to the same subject as a plurality of pixels on the contour and adds the extracted pixel to the basic area. 14. The circumscribing pixel extracting means for extracting circumscribing pixels adjacent to the outside of a basic area composed of pixels determined to belong to the same subject, and each of the circumscribing pixel extracting means The circumscribing pixel has inside and outside determining means for evaluating the probability that the pixel belongs to the basic region, and the circumscribing pixel evaluated as having the highest probability by the inside and outside determining means is configured to be newly added to the basic region. The image processing device according to claim 1. 15. The inside / outside determination unit includes a determination result of the determination unit for a pixel set belonging to the inside of the basic region near the circumscribed pixel and two regions of the circumscribed pixel itself, and a pixel set belonging to the outside of the basic region near the circumscribed pixel. 15. The image processing device according to claim 14, wherein the image processing device according to claim 14, wherein a probability that the circumscribing pixel belongs to the basic region is calculated based on both of the determination results of the determining unit for the two regions of the circumscribing pixel itself. 16. The region selecting unit, wherein the region corresponding to a subject corresponding to the closest range among the range of the distance estimation value set by the subject feature estimating unit among the regions generated by the region generating unit. The image processing apparatus according to claim 7, wherein the image processing apparatus is configured to select: 17. The image processing apparatus according to claim 1, wherein the area selection unit includes a mask image generation unit that forms a mask image representing the selected area, and is obtained under a standard imaging condition from a plurality of images obtained by the imaging unit. 8. The image processing apparatus according to claim 7, wherein the image processing apparatus is configured to select the selected image and output the selected image together with the mask image.