JP7005175B2 - Distance measuring device, distance measuring method and imaging device - Google Patents

Description

The present invention relates to a distance measuring device, a distance measuring method, and an imaging device.

Recently, a technique for acquiring distance information from an image obtained by shooting has been proposed. For example, it has been proposed to acquire a plurality of images from different viewpoints, obtain a parallax amount based on the correlation between the acquired plurality of images, and calculate a distance based on the obtained parallax amount. In Patent Document 1, the difference between the image taken by the left and right image capturing units with the feature pattern projected on the measurement object and the image taken without projecting the feature pattern on the measurement object is taken. , Extracting feature patterns is disclosed.

Japanese Unexamined Patent Publication No. 10-318732

However, with the conventional technique, it may not always be possible to measure the distance satisfactorily.
An object of the present invention is to provide a distance measuring device, a distance measuring method, and an imaging device capable of satisfactorily measuring a distance.

According to one aspect of the embodiment, the first image of the first viewpoint in which the first light having a pattern is applied to the subject and the first image in which the first light is applied to the subject. The second image of the second viewpoint different from the viewpoint of the above, the third image of the first viewpoint in which the subject is irradiated with the second light having no pattern, and the second light An acquisition unit that acquires a fourth image of the second viewpoint illuminated on the subject, a fifth image obtained by correcting the first image with the third image, and the first image. A distance measuring device is provided having a control unit for acquiring information on a distance using a sixth image obtained by correcting the image of 2 with the fourth image.

According to the present invention, it is possible to provide a distance measuring device, a distance measuring method and an imaging device capable of satisfactorily measuring a distance.

It is a schematic diagram which shows the distance measuring apparatus by 1st Embodiment. It is a schematic diagram which shows the image pickup part. It is a top view which shows the example of the arrangement of a pixel. It is a schematic diagram which shows the relationship between a light flux passing through an exit pupil of an image pickup optical system, and a pixel. It is a circuit diagram of a pixel. It is a figure which shows the example of the pattern projected on the subject. It is a figure which shows the example of the image and the signal intensity distribution obtained by photographing the subject which the light including a pattern is projected. It is a figure which shows the example of the image and the signal intensity distribution obtained by photographing the subject irradiated with the light which does not include a pattern. It is a figure which conceptually shows the correction process performed in the distance measuring apparatus by 1st Embodiment. It is a figure which shows the example of the signal intensity distribution obtained by performing a correction process. It is a flowchart which shows the operation of the distance measuring apparatus by 1st Embodiment. It is a flowchart which shows the operation of the distance measuring apparatus by 2nd Embodiment. It is a schematic diagram which shows the example of the distance measuring apparatus by a modification embodiment. It is a figure which conceptually shows the correction process performed in the distance measuring apparatus by a modification embodiment. It is a figure which shows the example of the pattern used in the modification embodiment.

As mentioned above, there are cases where the conventional technique cannot always measure the distance satisfactorily. For example, when the subject has a poor texture such as a pattern, the distance cannot always be measured satisfactorily. In addition, even when the reflectance of the subject was extremely high, the distance could not always be measured satisfactorily. Therefore, the inventor of the present application has conceived the following invention in order to enable better measurement of the distance.
Embodiments of the present invention will be described in detail below with reference to the drawings. The present invention is not limited to the following embodiments.

[First Embodiment]
The distance measuring device and the distance measuring method according to the first embodiment will be described with reference to the drawings. FIG. 1 is a schematic view showing a distance measuring device according to the present embodiment. As shown in FIG. 1, the distance measuring device 100 according to the present embodiment has an irradiation unit 101 that irradiates light, a plurality of images having different viewpoints, that is, an image pickup unit 103 that can acquire a parallax image, and the present embodiment. It includes a control unit 104 that controls the overall control of the distance measuring device 100.

The irradiation unit (photographing unit) 101 includes a light source (light source unit) 105 that emits light, and a spatial light modulator 106 that generates light having a pattern by modulating the light emitted from the light source 105, that is, pattern light. Is provided. Further, the irradiation unit 101 is provided with a projection lens 107 that magnifies the pattern light and projects it onto the subject 102, and an irradiation unit control unit 108 that controls each functional block provided in the irradiation unit 101. ing. From the viewpoint of performing the correction processing described later with high accuracy without being affected by the spectral reflectance of the subject 102, the light source 105 is a light source that emits white light including the entire visible light region, that is, a white light source. Is preferable. As the light source 105, for example, a discharge light emitting type light source such as a xenon lamp can be used. As the spatial light modulator 106, for example, a liquid crystal panel or the like can be used. Here, the case where the spatial light modulator 106 is a transmission type spatial light modulator will be described as an example, but the spatial light modulator 106 is not limited to this. The liquid crystal panel is provided with a liquid crystal cell in which a liquid crystal is enclosed, and a large number of pixel electrodes are arranged in a matrix in the liquid crystal cell. In the liquid crystal panel, the orientation direction of the liquid crystal molecules is appropriately changed by appropriately applying a voltage to the pixel electrodes arranged in a matrix, and the light passing portion and the light blocking portion are formed so as to form a desired pattern. To form. As a result, the light from the light source 105 is modulated by the spatial light modulator 106, and a desired pattern of light is obtained. For example, modulation is performed by the spatial light modulator 106 so that the light of the desired pattern 601 as described later with reference to FIG. 6 can be obtained. When modulation is performed by the spatial light modulator 106, light having a pattern, that is, patterned light is obtained. On the other hand, if modulation is not performed by the spatial light modulator 106, uniform light without a pattern, that is, non-patterned light can be obtained. Therefore, the irradiation unit 101 can also irradiate the subject 102 with uniform light having no pattern. That is, the irradiation unit 101 can selectively irradiate the subject 102 with the first light having a pattern and the second light having no pattern. Since the spatial light modulator 106 can operate at high speed, according to the present embodiment, it is possible to sequentially irradiate light with a pattern and light without a pattern at short time intervals. Further, in the present embodiment, since the presence or absence of the pattern is controlled by using the spatial light modulator 106, it is possible to irradiate the light with the pattern and the light without the pattern from the same position. Therefore, the first parallax image described later and the second parallax image described later can be acquired by using the light emitted from the same position. Since the correction process described later is performed using the first parallax image and the second parallax image acquired by using the light emitted from the same position, the correction process with high accuracy is performed according to the present embodiment. It will be possible to do.

FIG. 2 is a schematic view showing an image pickup unit. The image pickup unit (imaging device) 103 is, for example, a digital camera. The main body (body) 201 of the image pickup unit 103 is provided with an image pickup optical system (lens unit) 206 including an image pickup lens 202. The image pickup optical system 206 may or may not be detachable from the body 201. The body 201 includes an image sensor 203, a processing unit 204, a memory 205, and the like. The alternate long and short dash line in FIG. 2 indicates the optical axis 207. The optical image (subject image) formed by the image pickup optical system 206 is incident on the image pickup surface 208 of the image pickup element 203. The image pickup device 203 generates an image (image data) of the subject 102 by photoelectric conversion of the optical image formed by the image pickup optical system 206. The image data acquired by the image pickup device 203 is input to the processing unit 204. As described above, the image pickup device 203 can acquire a plurality of images having different viewpoints. The processing unit (control unit) 204 captures a first image of the first viewpoint and a second image of the second viewpoint different from the first viewpoint in a state where the subject 102 is irradiated with the pattern light. The first image and the second image acquired by using the element 203 are stored in the memory 205. That is, the processing unit 204 has a first image of the first viewpoint in which the first light having a pattern is applied to the subject and a second image of the second viewpoint in which the first light is applied to the subject. It can function as an acquisition unit to acquire the image of. Further, the processing unit 204 uses the image pickup element 203 to capture the third image of the first viewpoint and the fourth image of the second viewpoint in a state where the subject 102 is irradiated with uniform light having no pattern. The acquired third image and the acquired fourth image are stored in the memory 205. That is, the processing unit 204 has a third image of the first viewpoint in which the subject is irradiated with the second light having no pattern, and a fourth image of the second viewpoint in which the second light is applied to the subject. It can function as an acquisition unit to acquire the image of. The processing unit 204 correlates the fifth image obtained by correcting the first image with the third image and the sixth image obtained by correcting the second image with the fourth image. Get information about the distance based on. The information regarding such a distance may be, for example, a parallax amount or a distance value to the subject 102. Then, the processing unit 204 stores the information regarding the distance in the memory 205.

FIG. 3 is a plan view showing an example of pixel arrangement. A large number of pixels, that is, a large number of unit pixels 301 are arranged on the image pickup surface 208 of the image pickup element 203, but here, 48 unit pixels 301 out of the large number of unit pixels 301 are extracted and shown. There is. As shown in FIG. 3, a plurality of unit pixels 301 are arranged two-dimensionally, that is, in a matrix on the image pickup surface 208 of the image pickup element 203. R (Red), G (Green), and B (Blue) color filters are arranged on each unit pixel 301, for example, in a Bayer array. A divided pixel (sub-pixel) a and a divided pixel (sub-pixel) b are arranged in each unit pixel 301. The divided pixels a and b are provided with photodiodes (photoelectric conversion units) 302a and 302b, respectively. The output signals from the divided pixels a and b have parallax with each other as described later. The value of the output signal of the divided pixel b can be calculated, for example, by subtracting the value of the output signal of the divided pixel a from the added value of the output signals of the divided pixels a and b. Here, a case where one unit pixel 301 is composed of two divided pixels a and b, that is, a case where one unit pixel 301 is divided into two has been described as an example. Not limited. One unit pixel 301 may be composed of, for example, three or more divided pixels. Further, here, the case where all the unit pixels 301 provided on the imaging surface 208 are divided is described as an example, but the present invention is not limited to this, and only a part of the unit pixels 301 is divided. May be.

FIG. 4 is a schematic diagram showing an example of the relationship between the luminous flux passing through the exit pupil of the imaging optical system and the unit pixel. Note that FIG. 4 shows the relationship between the unit pixel located at the center of the angle of view and the luminous flux. As shown in FIG. 4, the color filter 402 and the microlens 403 are formed on the unit pixel 301. The center of the luminous flux passing through the exit pupil 401 of the imaging optical system 206 (see FIG. 2) coincides with the optical axis 207. The luminous flux that has passed through the exit pupil 401 is incident on the unit pixel 301 about the optical axis 207. The exit pupil 401 of the imaging optical system 206 has a first pupil region 405 and a second pupil region 406 different from the first pupil region 405. The luminous flux passing through the first pupil region 405 is received by the photodiode 302a provided in the divided pixel a via the microlens 403. On the other hand, the luminous flux passing through the second pupil region 406 is received by the photodiode 302b provided in the divided pixel b via the microlens 403. In this way, the divided pixels a and b receive light from the separate pupil regions 405 and 406 of the exit pupil 401 of the imaging optical system 206, respectively. Therefore, there is a parallax between the signal acquired by the divided pixel a and the signal acquired by the divided pixel b.

FIG. 5 is a circuit diagram of pixels. The photodiodes 302a and 302b provided in the divided pixels a and b respectively perform photoelectric conversion of the light incident on the divided pixels a and b, that is, the optical image, and accumulate the electric charge according to the exposure amount. By setting the signals txa and txb applied to the gates of the transfer switches 501a and 501b to the High level, the respective charges stored in the photodiodes 302a and 302b are transferred to the floating diffusion unit 502. The floating diffusion unit 502 is connected to the gate of the amplification transistor 503, and the potential of the gate of the amplification transistor 503 becomes a potential corresponding to the amount of electric charge transferred from the photodiodes 302a and 302b. The drain of the amplification transistor 503 is connected to the power supply potential Vdd. The output of the amplification transistor 503 is connected to a current source (not shown) via a vertical output line (not shown). The source follower circuit is composed of the amplification transistor 503 and the current source. The drain of the reset switch (reset transistor) 504 for resetting the floating diffusion unit 502 is connected to the power supply potential Vdd. By setting the signal res applied to the gate of the reset switch 504 to the High level, the floating diffusion unit 502 is reset. When resetting the charges of the photodiodes 302a and 302b, by setting the signal res and the signals txa and txb to the high level at the same time, both the transfer switches (transfer gates) 501a and 501b and the reset switch 504 are turned on. do. Then, the photodiodes 302a and 302b are reset via the floating diffusion unit 502. By setting the signal sel applied to the gate of the selection switch (selection transistor) 505 to the High level, a pixel signal corresponding to the potential of the gate of the amplification transistor 503 is output to the output vout of the unit pixel 301. The pixel circuit is not limited to the circuit shown in FIG.

The A image, which is one of the parallax images, is composed of the divided pixels a of the unit pixels 301 arranged two-dimensionally on the image pickup element 203, that is, the aggregate of the output signals of the first divided pixels. Will be done. Further, the B image, which is the other image of the disparity image, is formed by the divided pixels b of the unit pixels 301 arranged two-dimensionally on the image sensor 203, that is, the aggregate of the output signals of the second divided pixels. Is configured. The B image may be acquired by subtracting the A image from the A + B image. The A image and the B image acquired by the image pickup device 203 are transmitted to the processing unit 204. The processing unit 204 performs a correction process as described later on the A image and the B image, and determines the distance to the subject 102 based on the correlation between the A image and the B image obtained by performing the correction process. Calculate the indicated distance value. The processing unit 204 stores the calculated distance value in the memory 205. The distance measurement can be calculated by a known method. For example, a distance value can be obtained by calculating a correlation value by SSD (Sum of Squared Difference), obtaining a parallax amount from the calculated correlation value, and converting the obtained parallax amount into a distance.

When the subject 102 has a poor pattern or the like, that is, when the texture is poor, a remarkable peak is unlikely to appear in the correlation value between the A image and the B image, and the distance value cannot be obtained with high accuracy. Therefore, in the present embodiment, the surface of the subject 102 is textured by projecting the pattern light onto the subject 102 by the irradiation unit 101, whereby a remarkable peak appears in the correlation value between the A image and the B image. I am doing it.

FIG. 6 is a diagram showing an example of a pattern projected on a subject. As shown in FIG. 6, in the pattern 601 the dots are arranged so that the period and the size are random in any direction. If such a pattern 601 in which dots are randomly arranged in any direction is projected onto the subject 102, the shape of the window 705, which will be described later, is not limited, and the division direction of the unit pixel 301 is limited. It does not occur. Therefore, it is preferable to use the random dot pattern 601 as shown in FIG. The pattern as shown in FIG. 6 can be generated by using the spatial light modulator 106 provided in the irradiation unit 101. The pattern projected on the subject 102 is not limited to the pattern shown in FIG. 6, and can be appropriately set.

Regions corresponding to each other are set for the A image and the B image, respectively, and based on the peak of the correlation value between the A image signal and the B image signal acquired while gradually shifting the relative positions of these regions. It is possible to calculate the amount of parallax between the A image and the B image. However, when both the portion having a relatively low reflectance and the portion having a relatively high reflectance are located in such a region, the signal corresponding to the portion having a relatively high reflectance becomes dominant, and A The amount of parallax between the image and the B image cannot always be calculated accurately. In such a case, the amount of parallax between the A image and the B image can always be calculated accurately even when the image obtained by photographing the subject 102 on which the pattern 601 is projected is used. Therefore, the distance value to the subject 102 cannot always be calculated accurately. Therefore, in the present embodiment, the image obtained by photographing the subject 102 on which the pattern 601 is not projected is also used, and the distance measurement accuracy is improved by performing the correction processing as described later.

By photographing the subject 102 in a state where the pattern 601 is projected, that is, in a state where the pattern light is irradiated, the first A image 701A, that is, the first image and the first B image 701B, that is, the second And the image of. FIG. 7A is a diagram showing an example of an image obtained by photographing the subject 102 on which the pattern 601 is projected. As shown in FIG. 7A, the surface of the subject 102 is textured by the projection of the pattern 601. Parallax occurs between the first A image 701A and the first B image 701B, but here, for convenience of explanation, the first A image 701A and the first B are used in the same drawing. The image 701B is conceptually shown.

FIG. 7 (b) shows an example of the signal intensity distribution along the line segment 702 shown in FIG. 7 (a). The horizontal axis shows the position, and the vertical axis shows the signal strength I. As can be seen from FIG. 7B, the subject 102 has a portion 703 having a relatively low reflectance and a portion 704 having a relatively high reflectance. The area to be compared when the correlation value between the A image and the B image is obtained is defined by the windows 705A and 705B as shown in FIG. 7B. Reference numeral 705 is used when describing a window in general, reference numeral 705A is used when describing a specific window for an A image, and reference numeral 705B is used when describing a specific window for a B image. do.

By photographing the subject 102 in a state of irradiating a uniform light without a pattern, a second A image 801A, that is, a third image, and a second B image 801B, that is, a fourth image can be obtained. Is obtained. FIG. 8A is a diagram showing an example of an image obtained by photographing the subject 102 in a state of irradiating a uniform light without a pattern. As shown in FIG. 8A, the pattern 601 is not projected on the surface of the subject 102. There is parallax between the second A image 801A and the second B image 801B, but here, for convenience of explanation, the second A image 801A and the second B are used in the same drawing. The image 801B is shown.

FIG. 8B shows an example of a signal intensity distribution along the line segment 802 shown in FIG. 8A. The horizontal axis shows the position, and the vertical axis shows the signal strength I. As can be seen from FIG. 8B, the subject 102 has a portion 803 having a relatively low reflectance and a portion 804 having a relatively high reflectance.

As described above, when both the portion having a relatively low reflectance and the portion having a relatively high reflectance are located in the windows 705A and 705B, the signal corresponding to the portion having a relatively high reflectance is obtained. It becomes dominant and the amount of parallax between the A image and the B image cannot always be calculated accurately. If the parallax amount cannot be calculated accurately, the distance value cannot be calculated accurately. Therefore, in the present embodiment, the following correction processing is performed. That is, as shown in FIG. 9A, the first A image 701A obtained by taking a picture in a state of being irradiated with light having a pattern is taken by taking a picture in a state of being irradiated with light having no pattern. Divide by the second A image 801A obtained. At this time, each pixel value of the first A image 701A is divided by the pixel value of the second A image 801A corresponding to each pixel of the first A image 701A. By performing such a correction process, a third A image 901A, that is, a fifth image in which the influence of the reflectance is eliminated can be obtained. Further, as shown in FIG. 9B, by projecting the first B image 701B obtained by taking a picture in a state of being irradiated with light having a pattern, by projecting in a state of being irradiated with light having no pattern. Divide by the second B image 801B obtained. At this time, each pixel value of the first B image 701B is divided by the pixel value of the second B image 801B corresponding to each pixel of the first B image 701B. By performing such a correction process, a third B image 901B, that is, a sixth image in which the influence of the reflectance is eliminated can be obtained. FIG. 10 is a diagram showing an example of the signal intensity distribution obtained by performing the correction process. FIG. 10 shows an example of the signal intensity distribution of the third A image 901A and the third B image 901B along the line segment corresponding to the line segment 702 and 802. Although there is a parallax between the third A image 901A and the third B image 901B, for convenience of explanation, the same drawing is used here for the signal intensity distribution in the third A image 901A and the third B. The signal intensity distribution in the image 901B is shown. Then, the parallax amount is calculated based on the correlation value between the third A image 901A and the third B image 901B thus obtained. Since the influence of the reflectance is eliminated in both the third A image 901A and the third B image 901B, if the parallax amount is calculated based on these correlation values, a highly accurate parallax amount can be obtained. Therefore, according to the present embodiment, it is possible to obtain the distance value with high accuracy even when the subject 102 has locations having different reflectances.

FIG. 11 is a flowchart showing the operation of the distance measuring device according to the present embodiment.
In step S1101, the control unit 104 controls the irradiation unit 101 and the image pickup unit 103 so that the image of the subject 102 is acquired in a state where the subject 102 is irradiated with light having a pattern. As a result, a first parallax image obtained by irradiating the subject 102 with light having a pattern, that is, a first parallax image 701A and a first B image 701B can be obtained. The processing unit (control unit) 204 provided in the image pickup unit 103 stores the first A image 701A and the first B image 701B in the memory 205 provided in the image pickup unit 103. After that, the process proceeds to step S1102.

In step S1102, the control unit 104 controls the irradiation unit 101 and the image pickup unit 103 so that the image of the subject 102 is acquired in a state where the subject 102 is irradiated with uniform light having no pattern. As a result, a second parallax image obtained by irradiating the subject 102 with light having no pattern, that is, a second parallax image 801A and a second B image 801B can be obtained. The processing unit 204 stores the second A image 801A and the second B image 801B in the memory 205. After that, the process proceeds to step S1103.

In step S1103, the processing unit 204 provided in the imaging unit 103 performs correction processing for eliminating the influence of the reflectance as follows. The processing unit 204 takes a first A image 701A obtained by taking a picture in a state of being irradiated with light having a pattern, and a second image obtained by taking a picture of the first A image 701A obtained by taking a picture in a state of being irradiated with light having no pattern. Divide by the A image 801A of. As a result, the processing unit 204 acquires the third A image 901A. Further, the processing unit 204 was obtained by taking a picture of the first B image 701B obtained by taking a picture in a state of being irradiated with light having a pattern, by taking a picture of a first B image 701B obtained by taking a picture in a state of being irradiated with light having no pattern. Divide by the second B image 801B. As a result, the processing unit 204 acquires the third B image 901B. The processing unit 204 stores the third A image 901A and the third B image 901B thus obtained in the memory 205. After that, the process proceeds to step S1104.

In step S1104, the processing unit 204 calculates the correlation value between the third A image 901A and the third B image 901B, and calculates the parallax amount by, for example, subpixel estimation which is a known method. After that, the process proceeds to step S1105.

In step S1105, the processing unit 204 provided in the imaging unit 103 converts the parallax amount into a distance value by, for example, a known method. For example, the parallax amount is converted into a distance value based on the baseline length and the geometric relationship. For example, the distance on the exit pupil 401 between the center of gravity of the light flux passing through the first pupil region 405 and the center of gravity of the light flux passing through the second pupil region 406 corresponds to the baseline length. In this way, the distance value, that is, the distance to the subject 102 is obtained.

As described above, according to the present embodiment, the first A image 701A obtained by taking a picture in a state of being irradiated with light having a pattern is taken by taking a picture in a state of being irradiated with light having no pattern. It is divided by the second A image 801A obtained. As a result, a third A image 901A in which the influence of the reflectance is eliminated is obtained. Further, in the present embodiment, the first B image 701B obtained by taking a picture in a state of being irradiated with light having a pattern is a first obtained by taking a picture in a state of being irradiated with light having no pattern. It is divided by the B image 801B of 2. As a result, a third B image 901B in which the influence of the reflectance is eliminated is obtained. Then, the parallax amount is calculated based on the correlation value between the third A image 901A and the third B image 901B thus obtained. A correlation value is calculated using the third A image 901A and the third B image in which the influence of the reflectance is eliminated, and the parallax amount is calculated based on the correlation value. Even when the subject 102 is present, a highly accurate parallax amount can be obtained. Since the distance to the subject can be obtained based on the amount of parallax obtained in this way, the distance to the subject can be obtained with high accuracy. As described above, according to the present embodiment, since the influence of the reflectance of the subject 102 can be eliminated, it is possible to provide a distance measuring device and a distance measuring method that can obtain a good distance.

[Second Embodiment]
The distance measuring device and the distance measuring method according to the second embodiment will be described with reference to the drawings. FIG. 12 is a flowchart showing the operation of the distance measuring device according to the present embodiment. The distance measuring device according to the present embodiment determines whether or not to perform a correction process for eliminating the influence of the reflectance. The processing unit 204 determines whether or not to acquire the third image (second A image 801A) and the fourth image (second B image 801B) used for the correction processing in the first image (first image (first)). The determination is made based on the A image 701A) of 1 or the second image (1st B image 701B).

In step S1201, the control unit 104 takes an image with the irradiation unit 101 so that the image of the subject 102 is acquired while the subject 102 is irradiated with light having a pattern, as in step S1101 described above in the first embodiment. It controls the unit 103. As a result, a first parallax image obtained by taking a picture in a state of being irradiated with light having a pattern, that is, a first A image 701A and a first B image 701B can be obtained. The processing unit 204 provided in the image pickup unit 103 stores the first A image 701A and the first B image 701B in the memory 205 provided in the image pickup unit 103. After that, the process proceeds to step S1202.

In step S1202, whether or not the processing unit 204 provided in the imaging unit 103 performs correction processing for eliminating the influence of the reflectance on the first parallax image acquired in step S1201 is as follows. Judgment is made in this way. That is, the processing unit 204 sets a window of a predetermined size for the image acquired in step S1201, and the portion concerned is based on the difference between the maximum value and the minimum value of the brightness of the portion located in the window. Calculate the contrast value of. The processing unit 204 sequentially calculates the contrast values in the window at each position while sequentially changing the position of the window, and generates a histogram of a plurality of calculated contrast values, that is, a histogram of the contrast. When the contrast value corresponding to the peak existing in the contrast histogram is less than a preset threshold value, it is necessary to perform a correction process for eliminating the influence of the reflectance. That is, when the reflectance of the subject 102 is low, it is necessary to perform a correction process for eliminating the influence of the reflectance. Further, when a plurality of peaks exist in the contrast histogram and the contrast value corresponding to any of the plurality of peaks is less than a preset threshold value, the influence of the reflectance is eliminated. It is necessary to perform correction processing. That is, when the subject 102 in which a portion having a high reflectance and a portion having a low reflectance are photographed, it is necessary to perform a correction process for eliminating the influence of the reflectance. If it is necessary to correct the reflectance (YES in step S1202), the process proceeds to step S1204. In cases other than the above cases, it is not necessary to perform correction processing for eliminating the influence of the reflectance. If the correction process for eliminating the influence of the reflectance is not performed (NO in step S1202), the process proceeds to step S1203.

In step S1203, the processing unit 204 provided in the imaging unit 103 calculates the correlation value between the first A image 701A and the first B image 701B, and calculates the parallax amount by, for example, subpixel estimation, which is a known method. calculate. After that, the process proceeds to step S1207.
In step S1204, the control unit 104 irradiates the subject 102 so that the image of the subject 102 is acquired in a state where the subject 102 is irradiated with uniform light having no pattern, similarly to the step S1102 described above in the first embodiment. The unit 101 and the image pickup unit 103 are controlled. As a result, a second parallax image obtained by irradiating the subject 102 with light having no pattern, that is, a second parallax image 801A and a second B image 801B can be obtained. The processing unit 204 stores the second A image 801A and the second B image 801B in the memory 205. After that, the process proceeds to step S1205.

In step S1205, the processing unit 204 provided in the imaging unit 103 performs the correction processing for eliminating the influence of the reflectance as follows, as in the above-mentioned step S1103 in the first embodiment. The processing unit 204 takes a first A image 701A obtained by taking a picture in a state of being irradiated with light having a pattern, and a second image obtained by taking a picture of the first A image 701A obtained by taking a picture in a state of being irradiated with light having no pattern. Divide by the A image 801A of. As a result, the processing unit 204 acquires the third A image 901A. Further, the processing unit 204 was obtained by taking a picture of the first B image 701B obtained by taking a picture in a state of being irradiated with light having a pattern, by taking a picture of a first B image 701B obtained by taking a picture in a state of being irradiated with light having no pattern. Divide by the second B image 801B. As a result, the processing unit 204 acquires the third B image 901B. The processing unit 204 stores the third A image 901A and the third B image 901B thus obtained in the memory 205. After that, the process proceeds to step S1206.

In step S1206, the processing unit 204 calculates the correlation value between the third A image 901A and the third B image 901B in the same manner as in step S1104 described above in the first embodiment, and for example, a sub that is a known method. The amount of parallax is calculated by pixel estimation. After that, the process proceeds to step S1207.
In step S1207, the processing unit 204 provided in the imaging unit 103 converts the parallax amount into a distance value by a known method in the same manner as in step S1105 described above in the first embodiment. In this way, the distance value, that is, the distance to the subject 102 is obtained.

As described above, according to the present embodiment, it is determined whether or not the correction process for eliminating the influence of the reflectance is performed, and when the correction process is unnecessary, the light having no pattern is emitted to the subject 102. No shooting is performed in the state of being illuminated by the light, and no such correction processing is performed. Therefore, according to the present embodiment, it is possible to speed up the measurement of the distance.

[Modification Embodiment]
Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various embodiments within the range not deviating from the gist of the present invention are also included in the present invention. included.

For example, in the above embodiment, the case where the transmissive spatial light modulator 106 using the liquid crystal panel is provided in the irradiation unit 101 has been described as an example, but the present invention is not limited thereto. For example, a reflection type spatial light modulator 106 equipped with LCOS (Liquid Crystal On Silicon) or the like may be provided in the irradiation unit 101. Further, the spatial light modulator 106 may be configured by using a DMD (Digital Micromirror Device).

Further, in the above embodiment, the case where a discharge light emitting type light source such as a xenon lamp is used as the light source 105 has been described as an example, but the present invention is not limited to this. For example, a semiconductor light emitting device such as an LED, a laser, or an organic EL (Organic Electro-Luminescence) can be used as the light source 105.

Further, in the above embodiment, the case where the pattern is controlled by using the spatial light modulator 106 has been described as an example, but the present invention is not limited to this. For example, light with a pattern may be generated by locating a mask on which the pattern is formed on the optical path. Examples of such a mask include a mask made of a glass plate on which a pattern is formed, a mask made of a metal plate on which a pattern is formed, and the like. FIG. 13 is a schematic view showing an example of a distance measuring device according to a modified embodiment. As shown in FIG. 13, the irradiation unit 101 is provided with a mask 1301. The light emitted from the light source 105 passes through the mask 1301 to obtain light with a pattern. A pattern 601 as shown in FIG. 6, for example, is formed on the mask 1301. The mask 1301 may be driven by an actuator (not shown). If the mask 1301 is not located on the optical path, it is possible to obtain uniform light without a pattern. On the other hand, if the mask 1301 is placed on the optical path, it is possible to obtain light having a pattern. In this way, the mask 1301 may be used to control the presence or absence of the pattern.

Further, in the above embodiment, the case where the pattern is controlled by using the spatial light modulator 106 has been described as an example, but the pattern may be controlled by switching the light source. For example, when a laser diode (LD: Laser Diode) is used as a light source, speckle can be generated. Therefore, it is possible to use a laser diode as a light source to obtain light having a speckle pattern. On the other hand, when a light emitting diode (LED: Light Emitting Diode) is used as a light source, no speckle is generated, so that uniform light can be obtained. In this way, it is possible to control the presence or absence of a pattern by switching the light source. When controlling the presence or absence of a pattern by such a method, the spatial light modulator 106 or the like becomes unnecessary, which can contribute to miniaturization and cost reduction.

Further, in the above embodiment, the case where the light source 105 is a white light source has been described as an example, but the present invention is not limited to this. For example, the light source 105 may be composed of a red (R) light source, a green (G) light source, and a blue (B) light source. The band of light emitted from each of these three color light sources matches the transmission band of the color filter 402 (see FIG. 4) provided in the image pickup element 203 (see FIG. 2). Therefore, if the light source 105 is composed of light sources of three colors R, G, and B, it can contribute to the improvement of light utilization efficiency.

Further, in the above embodiment, the case where the light source 105 is a light source that emits visible light has been described as an example, but the present invention is not limited to this. For example, the light source 105 may be a light source that emits infrared light (IR: Infrared), that is, an infrared light source. Then, the image pickup device 203 may be provided with a color filter having a transmission band corresponding to infrared light and a pixel having a light receiving sensitivity to infrared light. This makes it possible to obtain image data for viewing RGB and image data for distance measurement using infrared light. When the wavelength band of infrared light is between 800 nm and 1100 nm, it is possible to perform photoelectric conversion by a photoelectric conversion unit formed on a silicon substrate. Therefore, it is possible to acquire the image data for viewing and the image data for distance measurement only by changing the arrangement of the color filters.

Further, in the above embodiment, the parallax image is acquired by receiving the light flux passing through the plurality of pupil regions different from each other by the plurality of divided pixels, but the present invention is not limited to this. For example, a parallax image may be acquired by capturing an optical image formed by each of a plurality of imaging optical systems with a separate imaging element. That is, a parallax image may be acquired by a stereo camera. In this case, since the baseline length can be increased, it is possible to improve the distance measurement accuracy.

Further, in the above embodiment, the case where the irradiation unit 101 and the image pickup unit 103 are separately provided has been described as an example, but the present invention is not limited to this. For example, the irradiation unit 101 and the image pickup unit 103 may be integrated. If the irradiation unit 101 and the image pickup unit 103 can be integrated, the positional relationship between the irradiation unit 101 and the image pickup unit 103 will be fixed, which is preferable from the viewpoint of measurement accuracy. For example, the irradiation unit 101 may be configured by a strobe device attached to the image pickup unit 103, and the strobe device which is the irradiation unit 101 may be provided with a spatial light modulator 106 or the like.

Further, in the above embodiment, the case where the control unit 104 is provided separately from the image pickup unit 103 has been described as an example, but the present invention is not limited to this. For example, the processing unit 204 provided in the image pickup unit 103 may also serve as the control unit 104. This makes it possible to reduce the size and cost. In this case, it can be considered that the distance measuring device 100 according to the present embodiment is provided in the image pickup unit 103. That is, it can be considered that the distance measuring device 100 according to the present embodiment is provided in the image pickup device.

Further, in the above embodiment, the first A image 701A is divided by the second A image 801A to generate the third A image 901A, and the first B image 701B is divided by the second B image 801B. The case where the third B image 901B is generated by the above is described as an example. However, it is not limited to this. FIG. 14 is a diagram conceptually showing a correction process performed in a distance measuring device according to a modified embodiment. For example, as shown in FIG. 14A, the third A image 1301A may be generated by subtracting the second A image 801A from the first A image 701A. Further, as shown in FIG. 14B, the third B image 1301B may be generated by subtracting the second B image 801B from the first B image 701B. Thus, the fifth image (third A image 1301A) is generated by subtracting the third image (second A image 801A) from the first image (first A image 701A). You may. Further, even if the sixth image (third B image 1301B) is generated by subtracting the fourth image (second B image 801B) from the second image (first B image 701B). good. That is, the third A image 1301A may be the difference between the first A image 701A and the second A image 801A, and the third B image 1301B may be the first B image 701B and the second. It may be the difference from the B image 801B of. In this case, if subtraction is performed instead of division, the load during calculation can be reduced.

Further, in the above embodiment, light having a pattern is applied to the entire subject 102, uniform light having no pattern is applied to the entire subject 102, and correction processing is performed on the entire image. The case of doing this has been described as an example, but the present invention is not limited to this. For example, light with a pattern is applied to a part of the subject 102, uniform light without a pattern is applied to a part of the subject 102, and a part of the image is corrected. You may do so. If the correction process is performed on a part of the image instead of the entire image, the load on the correction process can be reduced. In this case, for example, a preset attention subject area may be used as a part of the subject area. In addition, a part thereof may be determined based on the result of face determination or the like. Further, the part may be determined based on the result of the main subject determination or the like based on the brightness, the contrast, or the like.

Further, in the above embodiment, the image sensor 203 is used to capture the third image of the first viewpoint and the fourth image of the second viewpoint in a state where the subject 102 is irradiated with the second light having no pattern. The case of acquiring the image has been described as an example, but the present invention is not limited to this. For example, in a state where the subject 102 is irradiated with a third light having an inversion pattern in which the light and darkness is inverted with respect to the first light pattern, the seventh image of the first viewpoint and the second of the second viewpoint. The image of 8 may be acquired by using the image pickup element 203. Then, by adding the first image and the seventh image, a third image in which the subject 102 is irradiated with uniform light having no pattern may be generated. Further, by adding the second image and the eighth image, a fourth image in which the subject 102 is irradiated with uniform light having no pattern may be generated. FIG. 15 is a diagram showing an example of a pattern used in the modified embodiment. FIG. 15A shows an example of the non-inverted pattern 1501, and FIG. 15B shows an example of the inverted pattern 1502 in which the light and darkness of the pattern 1501 shown in FIG. 15A is inverted. There is. For example, as the first light, the pattern 1501 as shown in FIG. 15A can be used. Further, as the third light, the pattern 1502 as shown in FIG. 15B can be used. The processing unit 204 generates a third image by adding the first image and the seventh image, and generates a fourth image by adding the second image and the eighth image. Can function as a generator.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 ... Irradiation unit 102 ... Subject 103 ... Image pickup unit 104 ... Processing unit 105 ... Light source unit 106 ... Spatial light modulator 107 ... Projection lens 108 ... Irradiation unit control unit 203 ... Image pickup element 204 ... Processing unit 205 ... Memory 206 ... Image pickup Optical system

Claims (21)

A first image of a first viewpoint in which a first light having a pattern is applied to a subject, and a second viewpoint different from the first viewpoint in which the first light is applied to the subject. The second image, the third image of the first viewpoint in which the subject is irradiated with the second light without the pattern, and the second image in which the second light is applied to the subject. The acquisition unit that acquires the fourth image of the viewpoint of 2 and
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. , A control unit that acquires information about the distance,
Have,
The distance control unit controls to determine whether or not to acquire the third image and the fourth image based on the first image or the second image. measuring device. A first image of a first viewpoint in which a first infrared light as a first light having a pattern is applied to a subject, and a first image in which the first infrared light is applied to the subject. The third image of the first viewpoint in which the subject is irradiated with the second image of the second viewpoint different from the first viewpoint and the second infrared light as the second light not provided with the pattern. And the acquisition unit that acquires the image of the second viewpoint and the fourth image of the second viewpoint in which the second infrared light is applied to the subject.
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. , A distance measuring device characterized by having a control unit for acquiring information about a distance. The fifth image is generated by dividing the first image by the third image.
The distance measuring device according to claim 1 or 2, wherein the sixth image is generated by dividing the second image by the fourth image. The fifth image is generated by subtracting the third image from the first image.
The distance measuring device according to claim 1 or 2, wherein the sixth image is generated by subtracting the fourth image from the second image. A first image of a first viewpoint in which a first light having a pattern is applied to a subject, and a second viewpoint different from the first viewpoint in which the first light is applied to the subject. The second image of the above, the seventh image of the first viewpoint in which the subject is irradiated with the second light having the inversion pattern in which the light and darkness is inverted with respect to the pattern, and the second light. Acquires the eighth image of the second viewpoint illuminated by the subject, and
Generation of generating a third image by adding the first image and the seventh image, and generating a fourth image by adding the second image and the eighth image. Department and
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. , A distance measuring device characterized by having a control unit for acquiring information about a distance. The distance measuring device according to any one of claims 1 to 5, wherein the information regarding the distance is a parallax amount. The distance measuring device according to any one of claims 1 to 5 , wherein the information regarding the distance is a distance to the subject. A light source unit and a spatial light modulator that generates the first light by modulating the light emitted from the light source unit are provided, and the first light and the second light are selectively selected for the subject. The distance measuring device according to any one of claims 1 to 7, further comprising an irradiation unit capable of irradiating the light source. A light source unit and a mask that generates the first light from the light emitted from the light source unit are provided.
The distance measuring device according to any one of claims 1 to 7 , further comprising an irradiation unit capable of selectively irradiating the subject with the first light and the second light. A light source unit, a mask that generates the first light from the light emitted from the light source unit, and a mask.
An irradiation unit capable of selectively irradiating the subject with the first light and the second light, and an irradiation unit.
Equipped with
The distance measuring device according to claim 1 or 5 , wherein the light source unit includes a white light source. A light source unit, a mask that generates the first light from the light emitted from the light source unit, and a mask.
An irradiation unit capable of selectively irradiating the subject with the first light and the second light, and an irradiation unit.
Equipped with
The distance measuring device according to claim 1 or 5 , wherein the light source unit includes a red light source, a green light source, and a blue light source. The distance measuring device according to claim 8 or 9, wherein the light source unit includes an infrared light source. By imaging the subject, the first image of the first viewpoint in which the first light having a pattern is applied to the subject and the first image in which the first light is applied to the subject. The second image of the second viewpoint different from the viewpoint, the third image of the first viewpoint in which the subject is irradiated with the second light without the pattern, and the second light An image pickup unit that acquires a fourth image of the second viewpoint illuminated on the subject, and an image pickup unit.
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. Has a control unit, which acquires information about the distance,
The control unit controls to determine whether or not to acquire the third image and the fourth image based on the first image or the second image. Device. The first image of the first viewpoint in which the subject is irradiated with the first infrared light having a pattern is different from the first viewpoint in which the first infrared light is applied to the subject. The second image of the second viewpoint, the third image of the first viewpoint in which the subject is irradiated with the second infrared light not provided with the pattern, and the second infrared light An acquisition unit that acquires a fourth image of the second viewpoint illuminated on the subject, and
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. An image pickup apparatus characterized by having a control unit for acquiring information about a distance. By imaging the subject, the first image of the first viewpoint in which the first light having a pattern is applied to the subject and the first image in which the first light is applied to the subject. The seventh image of the first viewpoint, in which the subject is irradiated with the second image of the second viewpoint different from the viewpoint and the third light having the inversion pattern in which the light and shade are inverted with respect to the pattern. An image pickup unit that acquires an image and an eighth image of the second viewpoint in which the third light is applied to the subject.
Generation of generating a third image by adding the first image and the seventh image, and generating a fourth image by adding the second image and the eighth image. Department and
A fifth image obtained by correcting the first image with the third image, and
An image pickup apparatus comprising a control unit for acquiring information regarding a distance by using a sixth image obtained by correcting the second image with the fourth image. A first image of a first viewpoint in which a first light having a pattern is applied to a subject, and a second viewpoint different from the first viewpoint in which the first light is applied to the subject. The second image of
A third image of the first viewpoint in which the second light without the pattern is applied to the subject, and a fourth image of the second viewpoint in which the second light is applied to the subject. Steps to get the image and
A fifth image obtained by correcting the first image with the third image, and
A step of acquiring information on a distance using a sixth image obtained by correcting the second image with the fourth image, and
A step of determining whether or not to acquire the third image and the fourth image based on the first image or the second image, and
A distance measuring method characterized by having. The first image of the first viewpoint in which the subject is irradiated with the first infrared light having a pattern is different from the first viewpoint in which the first infrared light is applied to the subject. The second image of the second viewpoint, the third image of the first viewpoint in which the subject is irradiated with the second infrared light not provided with the pattern, and the second infrared light The step of acquiring the fourth image of the second viewpoint illuminated on the subject, and
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. , Steps to get information about distances,
A distance measuring method characterized by having. A first image of a first viewpoint in which a first light having a pattern is applied to a subject, and a second viewpoint different from the first viewpoint in which the first light is applied to the subject. The second image of the above, the seventh image of the first viewpoint in which the subject is irradiated with the third light having the inversion pattern in which the light and darkness is inverted with respect to the pattern, and the third light. And the step of acquiring the eighth image of the second viewpoint illuminated on the subject.
A step of generating a third image by adding the first image and the seventh image, and generating a fourth image by adding the second image and the eighth image. When,
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. , Steps to get information about distances,
A distance measuring method characterized by having. On the computer
A first image of a first viewpoint in which a first light having a pattern is applied to a subject, and a second viewpoint different from the first viewpoint in which the first light is applied to the subject. The second image of
A third image of the first viewpoint in which the second light without the pattern is applied to the subject, and a fourth image of the second viewpoint in which the second light is applied to the subject. Steps to get the image and
A fifth image obtained by correcting the first image with the third image, and
A step of acquiring information on a distance using a sixth image obtained by correcting the second image with the fourth image, and
A step of determining whether or not to acquire the third image and the fourth image based on the first image or the second image, and
A program to execute. On the computer
The first image of the first viewpoint in which the subject is irradiated with the first infrared light having a pattern is different from the first viewpoint in which the first infrared light is applied to the subject. The second image of the second viewpoint, the third image of the first viewpoint in which the subject is irradiated with the second infrared light not provided with the pattern, and the second infrared light The step of acquiring the fourth image of the second viewpoint illuminated on the subject, and
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. , Steps to get information about distances,
A program to execute. On the computer
A first image of a first viewpoint in which a first light having a pattern is applied to a subject, and a second viewpoint different from the first viewpoint in which the first light is applied to the subject. The second image of the above, the seventh image of the first viewpoint in which the subject is irradiated with the third light having the inversion pattern in which the light and darkness is inverted with respect to the pattern, and the third light. And the step of acquiring the eighth image of the second viewpoint illuminated on the subject.
A step of generating a third image by adding the first image and the seventh image, and generating a fourth image by adding the second image and the eighth image. When,
Using a fifth image obtained by correcting the first image with the third image and a sixth image obtained by correcting the second image with the fourth image. , Steps to get information about distances,
A program to execute.

Images