JP2021127998A - Distance information acquisition device and distance information acquisition method - Google Patents

Abstract

To provide a distance information acquisition device and a distance information acquisition method capable of detecting unreliable distance information among the acquired distance information.SOLUTION: The distance information acquisition device acquires distance information of a subject by a predetermined method. And, the distance information acquisition device generates a first shadow image representing a shadow area and an illumination area by using a first captured image taken by emitting light from an auxiliary light source. The distance information acquisition device further generates a second shadow image by using the distance information and the position information of the auxiliary light source. The distance information acquisition device detects unreliable distance information among the distance information based on the difference between the first shadow image and the second shadow image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a distance information acquisition device and a distance information acquisition method.

A method of acquiring distance information of a subject from an image of the subject is known. For example, distance information can be acquired for each pixel based on the amount of parallax between a set of images of the same subject taken from different viewpoints (Patent Document 1). An image in which the value of each pixel indicates distance information is also called a distance image, a depth map, a distance map, or the like.

Japanese Unexamined Patent Publication No. 2010-177741

If there is an error in the acquired distance information, the accuracy of processing using the distance information will decrease. For example, when adding a lighting effect by a virtual light source to an image of a subject, if there is an error in the distance information of the subject, the lighting effect may become unnatural. Therefore, it is desired to detect unreliable distance information in which an error should be taken into consideration.

An object of the present invention is to provide a distance information income device and a distance information acquisition method capable of detecting unreliable distance information among the acquired distance information.

The above-mentioned object is to generate a first shadow image representing a shadow area and an illumination area by using an acquisition means for acquiring distance information of a subject and a first captured image taken by emitting an auxiliary light source. The difference between the first shadow image and the second shadow image of the second generation means for generating the second shadow image by using the first generation means and the distance information and the position information of the auxiliary light source. Based on this, it is achieved by a distance information acquisition device characterized by having a detection means for detecting unreliable distance information among the distance information.

According to the present invention, it is possible to provide a distance information income device and a distance information acquisition method capable of detecting unreliable distance information among the acquired distance information.

(A) is a diagram showing a configuration example of an imaging device as a distance information acquisition device according to an embodiment, (b) is a diagram showing a configuration example of an auxiliary light source, and (c) is a schematic diagram showing an example of pattern light. Schematic diagram showing a structural example of pixels included in the photographing element of FIG. (A) is a flowchart regarding the operation of the distance information acquisition device according to the embodiment, and (b) is a schematic diagram regarding the distance information acquisition operation. Flow chart regarding the operation of the distance information acquisition device according to the modified example of the embodiment The figure for demonstrating another modification of embodiment Schematic diagram showing a configuration example when an external device of the image pickup device is used as a distance information acquisition device.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings based on an exemplary embodiment. The following embodiments do not limit the invention according to the claims. Further, although a plurality of features are described in the embodiment, not all of them are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

In the following embodiments, a case where the present invention is carried out by an imaging device will be described. However, the present invention is applicable to any electronic device capable of image processing. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, drive recorders and the like. These are examples, and the present invention can be applied to other electronic devices.

● (1st embodiment)
(Device configuration)
FIG. 1A is a block diagram showing a functional configuration example of an image pickup apparatus 100 as a shape information acquisition apparatus according to an embodiment of the present invention. FIG. 1A corresponds to a top view. Here, the image pickup apparatus 100 is a digital still camera capable of taking a picture using an auxiliary light source 101 such as a flash or a light. The auxiliary light source 101 may be provided in the image pickup apparatus 100, or may be an external type that can be attached to and detached from the image pickup apparatus 100. The auxiliary light source 101 has a configuration capable of illuminating a subject from different directions. Here, as an example, an auxiliary light source 101 in which a plurality of light emitting sources are arranged on the same circumference is used, but a plurality of general flash devices may be used. In the xyz coordinate system in FIG. 1, the z-axis is parallel to the optical axis 140 of the photographing optical system 104, the x-axis is vertically upward and the y-axis is perpendicular to each other, and the axis is perpendicular to the optical axis.

The image pickup apparatus 100 has a control unit 108. The control unit 108 is, for example, a CPU (also called an MPU or a microprocessor), and controls the operation of each part of the image pickup apparatus 100 by reading the program stored in the ROM 109 into the RAM 107 and executing the program, and performs the functions of the image pickup apparatus 100. Realize. When the photographing optical system 104 is an interchangeable type, the control unit 108 communicates with the controller of the photographing optical system 104 to control the operation of the photographing optical system 104.

The ROM 109 is a rewritable non-volatile memory, and stores a program executed by the control unit 108, various setting values of the image pickup apparatus 100, GUI data, and the like. The RAM 107 is used by the control unit 108 when executing a program, or is used by the image processing circuit 106 to hold image data. A part of RAM 107 may be used as a video memory.

The image processing circuit 106 applies predetermined image processing to the image signal obtained from the image sensor 105 to generate signals and image data, and acquire and / or generate various types of information. The image processing circuit 106 may be a dedicated hardware circuit such as an ASIC designed to realize a specific function, or a programmable processor such as a DSP may execute the software to perform the specific function. It may be a configuration to be realized.

Here, the image processing applied by the image processing circuit 106 includes preprocessing, color interpolation processing, correction processing, data processing processing, evaluation value calculation processing, special effect processing, and the like. Preprocessing includes signal amplification, reference level adjustment, defective pixel correction, and the like. The color interpolation process is a process of interpolating the values of color components not included in the image data read from the pixels, and is also called a demosaic process or a simultaneous process. The correction processing includes white balance adjustment, gradation correction (gamma processing), processing for correcting the influence of optical aberration and limb darkening of the photographing optical system 104, and processing for correcting color. The data processing process includes synthesis processing, scaling processing, coding and decoding processing, header information generation processing, and the like. The evaluation value calculation process includes generation of signals and evaluation values used for automatic focus detection (AF), calculation process of evaluation values used for automatic exposure control (AE), and the like. Special effects processing includes adding blur, changing color tones, rewriting processing, and the like. Note that these are examples of image processing to which the image processing circuit 106 can be applied, and do not limit the image processing to which the image processing circuit 106 can be applied.

The control unit 108 controls the setting and operation of the auxiliary light source 101. The control unit 108 may set the auxiliary light source 101 in consideration of the evaluation value generated by the image processing circuit 106. Further, the control unit 108 also controls the drive of the focus lens of the photographing optical system 104 and the aperture that also serves as the shutter.

The operation unit 102 is a general term for input devices such as buttons and switches for a user to input various instructions to the image pickup device 100. The input devices constituting the operation unit 102 have names according to the assigned functions. For example, the operation unit 102 includes a release switch for instructing the start of the shooting preparation operation and the start of shooting, a shooting mode selection switch for selecting a shooting mode, a menu button, a direction key, a decision key, and the like. Multiple functions may be assigned to the same input device. Further, the input device may be a software button / key.

The operation of the image pickup device 100 and the auxiliary light source 101 may be controlled by an external device such as a PC or a mobile device.

FIG. 1B is a front view of the auxiliary light source 101 as viewed from the optical axis direction. The auxiliary light source 101 has a plurality of light emitting units 110. The light emitting unit 110 is arranged so that the center is located on the circle 141 centered on the optical axis 140 of the photographing optical system 104 and is rotationally symmetrical. The positions of the plurality of light emitting units 110 when the auxiliary light source 101 is correctly attached to the lens barrel are constant, and the relative positional relationship with the image pickup apparatus 100 is also constant. The auxiliary light source 101 further includes one projection unit 111 and a mounting member 112 for mounting the auxiliary light source 101 on the lens barrel of the image pickup apparatus 100. The position of the light emitting unit 110 does not have to be rotationally symmetric. Further, the number and density of the light emitting units 110 are not particularly limited. The plurality of light emitting units 110 can selectively emit light, and the control unit 108 can control which light emitting unit 110 emits light.

The projection unit 111 projects pattern light having a spatial brightness change. The configuration of the projection unit 111 is not particularly limited, but a configuration in which an optical image obtained by irradiating a pattern mask having a pattern formed on frosted glass or a metal plate with a light source such as an LED (Light Emitting Diode) is projected onto an imaging lens. Can be. FIG. 1C shows an example of pattern light. The pattern light 130 is a line pattern light in which high-luminance regions and low-luminance regions are alternately repeated in the x-axis direction and has a long linear luminance region in the y-axis direction. The projection unit 111 does not have to be arranged between the light emitting units 110. Further, the position in the optical axis direction may be different from that of the light emitting unit 110. It is assumed that the position information of the light emitting unit 110 and the projection unit 111 included in the auxiliary light source 101 is stored in the ROM 109 or the memory in the auxiliary light source 101.

In the present embodiment, the image sensor 105 is configured to be able to acquire a parallax image pair. Specifically, each of the plurality of pixels two-dimensionally arranged on the image sensor 105 has two photoelectric conversion units that share one microlens, and each photoelectric conversion unit is configured to be able to read a signal. There is.

FIG. 2 schematically shows a configuration example of one of a plurality of pixels arranged on the image pickup device 105. FIG. 2A is a vertical cross-sectional view, and FIG. 2B is a top view. The pixel has a microlens 201, a color filter 202, and photoelectric conversion units 203A and 203B formed on the substrate 204. In FIG. 2, the wiring and the like of the pixel 200 are omitted. In the present embodiment, it is assumed that the image sensor 105 is provided with a color filter having a primary color Bayer array. In this case, the color filter 202 included in the pixel 200 has one of R (red), G (green), and B (blue). The color of the color filter 202 possessed by each pixel is determined by the type of the color filter provided in the image sensor 105 and the position of the pixel.

FIG. 2B is a diagram schematically showing a state in which the exit pupil 130 of the photographing optical system 104 is viewed from the intersection (center image height) of the optical axis 140 and the image sensor 105. The photoelectric conversion units 203A and 203B provided in the pixel 200 have a first luminous flux that has passed through the first pupil region 210, which is a different region of the exit pupil 130, and a second luminous flux that has passed through the second pupil region 220. Are incident on each. Therefore, for a plurality of pixels of the image sensor 105, an image composed of signals obtained by the photoelectric conversion unit 203A (referred to as an A image) and an image composed of signals obtained by the photoelectric conversion unit 203B (B). (Called an image) constitutes a parallax image pair having different viewpoints from each other.

The center of gravity position (first center of gravity position) 211 of the first pupil region 210 and the center of gravity position (second center of gravity position) 221 of the second pupil region 220 are the first axes from the center of the ejection pupil 130. They are eccentric (moving) in opposite directions along the 200. The direction along the first axis 200 is called the pupil division direction. Further, the distance 222 from the first and second center of gravity positions 211 and 221 corresponds to the baseline length of the parallax. In this embodiment, as shown in FIG. 2B, the pupil division direction is equal to the x-axis direction.

The deviation between the coordinates in the A image corresponding to one point (referred to as the point P) in the shooting range and the coordinates in the B image differs depending on the distance of the point P. Therefore, by detecting the magnitude of the positional deviation of the corresponding points between the A image and the B image, it is possible to obtain the distance information of the subject corresponding to each pixel. Further, since the magnitude of the misalignment corresponds to the magnitude of the defocus amount, it can be used for the automatic focus detection of the phase difference detection method.

By adding the A image and the B image, a normal photographed image can be generated. Therefore, the B image (A image) can be generated by subtracting the A image (B image) from the normal captured image. The normal captured image is an image obtained by acquiring an image signal corresponding to a light flux passing through the entire exit pupil 130 from each pixel.

The image processing circuit 106 reads, for example, a normal captured image and an A image from the image sensor 105 and stores them in the RAM 107. Then, the image processing circuit 106 generates a B image as a difference between the normal captured image and the A image. The image processing circuit 106 generates a distance image distance image by calculating the amount of misalignment for each pixel based on, for example, a correlation calculation, and further converting the amount of misalignment into distance information. The image processing circuit 106 stores the generated distance image in the RAM 107.

(Image generation operation with virtual lighting effect)
In the present specification, in an image of a subject, an area illuminated by light from a light source is referred to as an "illumination area", and an area in which the light from the light source is blocked and shadowed is referred to as a "shadow area". Further, information indicating whether each pixel of the captured image is a shadow area or an illumination area is called a “shadow image”. The shadow image may be, for example, a binary image in which the value of the pixel in the shadow region is 0 and the value of the pixel in the illumination region is 1. Alternatively, the shadow image may be a multi-valued image having a larger value as the pixels in the region having higher illuminance.

Hereinafter, an operation in which the image pickup apparatus 100 generates a shadow image and uses the shadow image to generate an image to which a virtual lighting effect is applied will be described with reference to the flowchart of FIG. 3A. This operation can be executed, for example, in response to an instruction from the operation unit 102 to generate an image to which the virtual lighting effect is applied. Since the exposure conditions (including the determination of the light emitting method of the auxiliary light source 101) and the focus adjustment can be performed by a known method, the details thereof will be omitted. At least the position of the image pickup apparatus 100 is fixed while the shadow image is generated.

In step 301 "image acquisition", the control unit 108 causes one of the plurality of light emitting units 110 included in the auxiliary light source 101 to emit light to perform imaging. Which light emitting unit 110 emits light is arbitrary, and may be determined in advance, for example. The control unit 108 reads out a set of image signals from the image sensor 105 so that an A image and a B image can be generated. The image processing circuit 106 applies predetermined image processing to the read image signal to generate data of A image and B image in a predetermined format. The image processing circuit 106 stores the generated image data in the RAM 107. The image data generated here represents an image when the subject is illuminated from a specific direction, and is therefore referred to as an "illuminated image" or a first captured image.

The control unit 108 continues to take a picture in a state where the light emitting unit 101 of the auxiliary light source 101 does not emit light and the pattern light 130 is projected from the projection unit 111. The control unit 108 reads out a set of image signals from the image sensor 105 so that an A image and a B image can be generated. The image processing circuit 106 applies predetermined image processing to the read image signal to generate data of A image and B image in the same format as the illumination image. The image processing circuit 106 stores the generated image data in the RAM 107. The image data generated here is called a "pattern light image". By taking an image in a state where the pattern light is projected, it is possible to improve the acquisition accuracy of the distance information for a subject having low contrast.

In step 302 “distance information acquisition”, the image processing circuit 106 (acquisition means) acquires the distance information of the subject by using the illumination image or the pattern light image. The image processing circuit 106 can acquire the distance information of the subject for each pixel from the A image and the B image, which are parallax image pairs, by using any known method. An example will be described below.

FIG. 3B schematically shows A image 320A and B image 320B. First, the image processing circuit 106 calculates the correlation value between the A image and the B image. The image processing circuit 106 sets a partial region including a pixel 321 (referred to as a pixel of interest) for acquiring distance information in the A image as a reference image 322. Next, the image processing circuit 106 sets, in the B image, a region having the same horizontal and vertical dimensions as the reference image as the reference image 323.

Then, the image processing circuit 106 detects the deviation amount of the reference image 322 by the block matching method using the reference image 322 as a template. Here, the position of the reference image in the B image in the vertical direction is the same as that of the reference image, the reference image is moved in the same x-axis direction as the pupil division direction, and the correlation value between the reference image and the reference image at each position is calculated. Get the data string of the correlation value for each position. The moving direction of the reference image is called the parallax calculation direction.

By setting the parallax calculation direction to the same direction as the pupil division direction, the amount of parallax between the A image and the B image generated according to the subject distance can be calculated correctly. The correlation value may be SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), or the like.

The image processing circuit 106 calculates the parallax amount of the pixel of interest based on the data string of the correlation values obtained for the reference image. The parallax amount can be calculated using an existing method. For example, the image processing circuit 106 has a movement amount having the highest correlation by a known interpolation method using a movement amount for which a correlation value representing the highest correlation is obtained and a correlation value corresponding to an adjacent movement amount. Can be obtained with sub-pixel accuracy and used as the amount of discrimination.

The calculated parallax amount can be converted into a defocus amount or a subject distance by a known method. The conversion from the parallax amount to the defocus amount can be calculated from the geometrical relationship using the baseline lengths of the A image and the B image. Further, by setting the distance at which the photographing optical system is in focus at the focus lens position determined according to the defocus amount of the photographing optical system 104 as the subject distance, the defocus amount can be converted into the subject distance. It may be converted into a subject distance by multiplying the parallax amount by a predetermined conversion coefficient. The amount of defocus and the subject distance are collectively called distance information.

In this way, the image processing circuit 106 can acquire the distance information about the pixel of interest. A distance image can be generated by acquiring distance information with each pixel of the A image as a pixel of interest. The image processing circuit 106 stores the generated distance image in the RAM 107. The image processing circuit 106 may generate a distance image for one of the illumination image and the pattern light image, for example, the pattern light image, but may generate a distance image for each of them.

In step 303 "first shadow image generation", the image processing circuit 106 (first generation means) generates a shadow image from the illumination image acquired in step 301. Specifically, the image processing circuit 106 is a first shadow in the illumination image, in which a region having a pixel value (for example, brightness) below a predetermined threshold value is a shadow region and a region having a pixel value larger than the threshold value is an illumination region. Generate an image. The shadow image may be a binary image in which the pixel value of the shadow region is 0 and the pixel value of the illumination region is 1. The illumination image used here may be either an A image, a B image, or a normal photographed image (an image obtained by adding an A image and a B image).

The image processing circuit 106 (second generation means) in step 304 “second shadow image calculation” includes the position information of the light emitting unit 110 that emits light when the illumination image is taken, and the distance image generated in step 302. To generate a shadow image using. The image processing circuit 106 can generate a shadow image by a known method such as a method using a distance image as a shadow map. For example, with the intersection of the optical axis 140 of the camera and the image sensor 105 as a viewpoint, a plane perpendicular to the optical axis (projection surface of the image sensor 105) existing at a predetermined distance from the viewpoint is assumed. When the distance from the point on the plane corresponding to each pixel of the image sensor 105 to the light emitting unit 110 is shorter than the value of the distance image corresponding to that pixel, that pixel is included in the shadow region. do. On the other hand, when the distance from the point on the plane corresponding to each pixel of the image sensor 105 to the light emitting unit 110 is equal to the value of the distance image corresponding to the pixel, there is nothing to block the light emitting unit 110 and its pixel. Therefore, it shall be included in the illumination area. The image processing circuit 106 stores the second shadow image thus generated in the RAM 107. The second shadow image is an image showing a shadowed portion and a non-shadowed portion in an image obtained when a virtual light source is arranged at the same position as the light emitting unit 110 that emits light when the illumination image is taken. Corresponds to.

In step 305 “Detection of shadow error region”, the image processing circuit 106 (detection means) detects an error region included in the second shadow image. The image processing circuit 106 compares the first shadow image and the second shadow image stored in the RAM 107, and detects a region in which the determination of the shadow region and the illumination region is different from each other as the shadow error region. When the shadow image is a binary image, the image processing circuit 106 can calculate a difference image between the first shadow image and the second shadow image, and detect a region whose value is not 0 as a shadow error region. When the shadow image is a multi-valued image, the image processing circuit 106 detects a region in which the absolute value of the value is equal to or greater than the threshold value as the shadow error region in the difference image between the first shadow image and the second shadow image. be able to.

In step 306 “Calculation of distance error area”, the image processing circuit 106 (detection means) detects the area in the distance image corresponding to the shadow error area obtained in step 305 as the distance error area. Further, the area of the distance image corresponding to the point between each point in the plane assumed in S304 and the light emitting unit 110 corresponding to the shadow error area is detected as the distance error area. In this way, an error region (a region where the reliability of the distance information is low) in the distance image can be detected. As described above, the image processing circuit 106 (detection means) detects the unreliable distance information among the distance information constituting the distance image based on the difference between the first shadow image and the second shadow image. do.

(Generation of shaded image)
In step 307 “Image generation”, the image processing circuit 106 generates an image to which a lighting effect (virtual lighting effect) by a virtual light source is added. A shadow image is generated in the same manner as in step 304 by using the preset position information of the virtual light source and the portion of the distance image generated in step 302 excluding the distance error region detected in step 306.

Here, it is assumed that a virtual lighting effect is added to the illumination image acquired in step 301. The image processing circuit 106 lowers the pixel value (luminance value) of the pixel in the shadow region in the shadow image of the virtual light source generated in step 307 of the illumination image, and reduces the pixel value (luminance value) of the pixel in the illumination region. Correct to raise. The image processing circuit 106 does not change the pixel value in the region corresponding to the error region in the distance image.

In this way, when applying the virtual lighting effect using the distance image, by not using the unreliable distance information included in the distance image, an unnatural virtual lighting effect can be applied. It can be suppressed. The image to which the virtual lighting effect is applied is not limited to the image acquired in step 301. A virtual lighting effect can be applied to any image in which the distance image in which the distance error region has been detected can be used.

● (Modification example 1)
FIG. 4 is a flowchart relating to the operation of the image pickup apparatus 100 according to the modified example of the first embodiment. The same reference numbers will be added to the steps that perform the same operations as the flowchart of FIG. 3A, and the description thereof will be omitted. In this modification, when the shadow error region is large, the shadow error region is reduced by correcting the distance information and regenerating the second shadow image.

Since steps 301 to 306 are as described above, the description thereof will be omitted.
In "Detecting the size of the shadow error region" in step 308, the image processing circuit 106 detects the size of the shadow error region detected in step 305. The size of the shadow error area is any of the horizontal length, the vertical length, the area obtained by multiplying the vertical length and the horizontal length in the difference image, the number of pixels included in the shadow error area, and the like. It may be.

In step 309 “Determination of the size of the shadow error region”, the image processing circuit 106 determines whether or not the size calculated in step 308 is equal to or greater than a predetermined threshold value. The threshold value used here may be a fixed value, or may be determined in consideration of the position of the virtual light source for imparting the virtual lighting effect and the distance image.

The effect of the shadow error on the virtual lighting effect is more pronounced on the flat surface of the subject. The larger the incident angle of the virtual light with respect to the surface normal of the plane portion (the smaller the angle formed by the plane and the virtual light), the larger the shadow error caused by the error of the distance information. Therefore, for example, the main plane portion of the subject can be detected using a distance image or luminance information, and the larger the angle formed by the surface normal of the plane portion and the incident direction of the virtual light, the smaller the threshold value can be determined. As a result, the unnaturalness of the image to which the virtual lighting effect is added, which is generated in step 307, can be further reduced.

If it is not determined in step 309 that the size of the shadow error region is equal to or greater than the threshold value, the image processing circuit 106 executes step 307 as described above. On the other hand, when it is determined that the size of the shadow error region is equal to or larger than the threshold value, step 306 is executed as described above to detect the distance error region.

In step 310 “distance correction”, the image processing circuit 106 corrects the distance information included in the distance error region in the distance image. As a correction method, the distance information in the distance error region can be interpolated from the distance information in the surrounding region, or replaced with the average value of the distance information in the surrounding region. Alternatively, the color information of the image may be used to replace the distance information in the distance error region with the average value of the distance information in the region having the same or similar color as the image in the distance error region in the peripheral region of the distance error region. .. There is a high probability that the periphery of the same or similar color existing in the periphery has the same distance or a similar distance to the distance error region. Therefore, the accuracy of correction can be improved by using the average value of the distance information of the surrounding areas of the same or similar colors.

When the distance information in the distance error region is corrected, the image processing circuit 106 executes step 304 again to regenerate the second shadow image using the corrected distance image. After that, the image processing circuit 106 rediscovers the shadow error region in step 305.

The image processing circuit 106 repeats the correction of the distance information, the generation of the second shadow image, the detection of the shadow error region, and the size determination until the size of the shadow error region is not determined to be equal to or larger than the threshold value in step 309. A limit may be set on the number of repetitions.

By correcting the distance information in this way, the distance error area can be reduced and the accuracy of the virtual lighting effect can be improved. The effect of the shadow error on the addition of the virtual lighting effect becomes less noticeable as the distance error region becomes smaller. Therefore, the processing time is reduced by discontinuing the correction of the distance information when the size of the distance error region is no longer determined to be equal to or larger than the threshold value.

● (Modification example 2)
(Error area detected in images from multiple lighting positions)
A plurality of illumination images may be acquired by making the light emitting units 110 to emit light different, and a shadow error region may be detected for each illumination image. In this case, the image processing circuit 106 integrates the detected plurality of shadow error regions to obtain the final shadow error region. Further, the distance error region may be detected for each shadow error region.

By detecting the shadow error region and the distance error region for each different illumination direction in this way, the detection accuracy of the shadow error region and the distance error region can be improved. Further, the distance information can be corrected for each distance image. As a result, the distance information can be corrected with higher accuracy.

● (Modification example 3)
(Calculate the first shadow image from the illuminated image and other illuminated or non-illuminated images)
Step 303 “First shadow image calculation” may be realized by another method. For example, the illumination image acquired in step 301 is compared with the illumination image acquired by illuminating from a position different from the illumination position in step 301, and the area where the pixel value of the illumination image acquired in step 301 is small (dark) is defined. It can be detected as a shadow area. Alternatively, the illumination image acquired in step 301 may be compared with the image acquired without illumination, and a region having a small pixel value of the image acquired in step 301 may be detected as a shadow region.

● (Modification example 4)
(Detection method for other distance error areas)
A method of detecting the distance error region more accurately in step 306 “Detection of the distance error region” will be described with reference to FIG. 5 (a) and 5 (b) schematically show a case where the distance error regions 502 and 503 exist on the plane 501, and the shadow error region 504 is generated by the distance error regions 502 and 503.

FIG. 5A shows a state in which there is an error in the distance information of the region 502 that blocks the illumination light 505, and there is no error in the distance information 506 of the shadow region 504. FIG. 5B shows a state in which there is no error in the distance information of the region 507 that blocks the illumination light 505, and there is an error in the distance information 503 of the shadow region 504.

In the first embodiment, both the region that blocks the illumination light and the region that is shaded are detected as the distance error region. However, one of these may be detected as a distance error region. In this case, the steps in the flowchart shown in FIG. 5C can be performed instead of step 305.

In step 510 “Candidate region detection”, the image processing circuit 106 detects the light-blocking region 502 or 503 and the shaded region 506 or 507 as candidate regions.
Next, in step 511 "Distance difference calculation", the image processing circuit 106 calculates the difference 520 and 521 or the difference 522 and 523 between the distance information in the candidate area and the surrounding distance information for each candidate area.

In step 512 “distance error region detection”, the image processing circuit 106 compares the differences 520 and 521, or the differences 522 and 523. Then, the image processing circuit 106 detects one having a large difference as a distance error region. In this way, the distance error region can be detected more accurately.

● (Modification example 5)
(Other configurations of auxiliary light source)
In the first embodiment, the auxiliary light source 101 has a configuration having a plurality of light emitting units 110 and a projection unit 111, but even if it has a configuration having one light emitting unit 110 and no projection unit 111. good. In this case, the distance information can be acquired by using an image taken with or without causing the light emitting unit 110 to emit light. Alternatively, the auxiliary light source 101 may be configured not to have a light emitting unit 110. In this case, the distance information can be acquired and the distance error region can be detected by using the image acquired by projecting the pattern light on the projection unit 111. The same effect can be obtained even when the configuration of the auxiliary light source 101 is different.

● (Modification example 6)
(Other distance information acquisition methods)
The distance information may be acquired by a method different from the method described above. For example, it is possible to acquire highly accurate normal information from a plurality of images acquired by differently emitting light emitting units 110 by using a known illuminance difference stereo method. The normal information is information representing a change in the local shape of the subject, and it is possible to obtain highly accurate distance information by correcting the acquired distance information using the normal information. When performing the illuminance difference stereo method, it is necessary to acquire at least three images having different irradiation directions, so that the auxiliary light source 101 needs to have three or more light emitting units 110.

In addition, the distance information can be acquired by using a device such as LiDAR (laser radar) that detects the distance information based on the arrival time or frequency change of the reflected light, or by using a triangular distance measurement with a stereo camera. can.

● (Modification 7)
In the first embodiment, the case where the invention is applied to the image pickup apparatus 100 has been described. However, the photographing function is not essential to the present invention, and an illumination image, a pattern light image, a distance image, a position information of a light emitting unit, and the like may be acquired from an external device. Therefore, the first embodiment and the configuration described as a modification thereof can be implemented in all electronic devices provided with a computer (CPU).

For example, as shown in FIG. 6, the image processing circuit 106 is also used in the information processing device 600 capable of acquiring an illumination image, a pattern light image, a distance image, a position information of a light emitting unit, and the like through a communication I / F 150 included in the image pickup device 100. Can perform the same operation as. The communication I / F 150 may be a wired or wireless communication I / F. Further, the illumination image, the pattern light image, the distance image, the position information of the light emitting unit, and the like may be acquired not from the image pickup apparatus 100 but from a server on the network, a recording medium, or the like. The CPU included in the information processing device 600 can realize the same processing as the image processing circuit 106 described above by loading and executing a program stored in a storage device such as a ROM into the RAM.

(Other embodiments)
The projection unit 111 may use an LD (Laser Diode) as a light source. Further, as the pattern mask, a reflective LCOS (Liquid Crystal On Silicon), a transmissive LCOS, or a DMD (Digital Micromirror Device) may be used. By dynamically changing the spatial frequency of the projection pattern 130 according to the size and distance of the subject, it is possible to acquire more accurate distance information according to the situation.

It is preferable that the wavelength of the light source of the projection unit 111 is white including the entire visible light region because the effect of reflectance correction can be obtained regardless of the spectral reflectance of the subject. When the light source of the projection unit 111 is composed of three colors of R, G, and B, it matches the color filter transmission band of the image sensor 105 and is suitable from the viewpoint of light utilization efficiency with respect to the energy used. The wavelength of the light projected by the projection unit 111 is in the IR (Infrared) light emitting region, and the image sensor is also configured to support infrared light, which makes it possible to simultaneously capture viewing images using the RGB band. It is suitable. In particular, when the wavelength band of IR is between 800 nm and 1100 nm, Si can be used for the photoelectric conversion unit, and by changing the arrangement of the color filters, RGB viewing images and IR ranging images can be acquired with one image sensor. It is suitable because it can be used.

The image pickup device 100 may be composed of a stereo camera composed of two or more optical systems and corresponding image pickup elements. In this case, the degree of freedom in designing the baseline length is improved, and the resolution of the distance information can be improved. Further, the auxiliary light source may be an external device of the imaging device.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 ... image pickup device, 101 ... auxiliary light source, 105 ... image sensor, 106 ... image processing circuit, 108 ... control unit, 120 ... subject

Claims (12)

An acquisition method for acquiring the distance information of the subject,
A first generation means for generating a first shadow image representing a shadow area and an illumination area by using a first captured image taken by emitting light from an auxiliary light source.
A second generation means for generating a second shadow image using the distance information and the position information of the auxiliary light source, and
A distance information acquisition device comprising a detection means for detecting unreliable distance information among the distance information based on the difference between the first shadow image and the second shadow image. .. The distance information acquisition device according to claim 1, wherein the acquisition means acquires the distance information based on a parallax image pair in which the subject is photographed. The distance information acquisition device according to claim 2, wherein the parallax image pair is an image taken by projecting a specific pattern light onto the subject. The distance information acquisition device according to any one of claims 1 to 3, wherein the distance information is a distance image having distance information corresponding to each pixel. Further, it has a correction means for correcting the unreliable distance information when the size of the region having the unreliable distance information is equal to or larger than the threshold value in the distance image.
When the correction means corrects the distance information, the second generation means regenerates the second shadow image using the corrected distance information.
The detection means is characterized in that it rediscovers unreliable distance information among the distance information based on the difference between the first shadow image and the regenerated second shadow image. The distance information acquisition device according to claim 4. The first generation means generates the first shadow image from each of the plurality of first captured images taken by emitting auxiliary light sources at different positions.
Any of claims 1 to 5, wherein the detection means detects the unreliable distance information based on the difference between each of the first shadow images and the second shadow image. The distance information acquisition device according to item 1. Among the distance information, the detection means detects the distance information of the region that blocks the light of the auxiliary light source and the distance information of the region that blocks the light of the auxiliary light source and becomes a shadow as a candidate region, and detects the candidate. The claim is characterized in that, among the regions, the distance information of the candidate region having a large difference between the distance information of the candidate region and the distance information of the peripheral region of the candidate region is detected as the unreliable distance information. The distance information acquisition device according to any one of 1 to 6. Further, it has an additional means for generating an image in which a virtual lighting effect by a virtual light source is added to a captured image of the subject by using the distance information.
The distance information according to any one of claims 1 to 7, wherein the additional means does not use the unreliable distance information detected by the detection means among the distance information. Acquisition device. Image sensor and
It has the distance information acquisition device according to any one of claims 1 to 8.
The distance information acquisition device is an image pickup device, characterized in that an image captured by using the image pickup element is used as the first image pickup image. Each of the pixels provided in the image sensor has a plurality of photoelectric conversion units, and has a plurality of photoelectric conversion units.
The image pickup apparatus according to claim 9, wherein the acquisition means acquires the distance information based on a parallax image pair obtained by using the image pickup device. It is a distance information acquisition method executed by the distance information acquisition device.
The acquisition process to acquire the distance information of the subject,
A first generation step of generating a first shadow image representing a shadow area and an illumination area by using a first captured image taken by emitting light from an auxiliary light source, and
A second generation step of generating a second shadow image using the distance information and the position information of the auxiliary light source, and
A distance information acquisition method comprising a detection step of detecting unreliable distance information among the distance information based on a difference between the first shadow image and the second shadow image. .. A program for causing a computer to function as each means included in the distance information acquisition device according to any one of claims 1 to 8.

Images