WO2023058666A1

WO2023058666A1 - Distance measuring device

Info

Publication number: WO2023058666A1
Application number: PCT/JP2022/037197
Authority: WO
Inventors: 孝美長谷川
Original assignee: 株式会社ブルービジョン
Priority date: 2021-10-04
Filing date: 2022-10-04
Publication date: 2023-04-13
Also published as: JP7441380B2; JPWO2023058666A1

Abstract

This distance measuring device (1) comprises: a light emitting element (LED light source 18) that emits near infrared light and irradiates a photographic subject with said near infrared light; a light splitting unit (beam splitter 12) that splits reflected light traveling along the same optical axis from the photographic subject into visible light and near infrared light, and outputs the visible light and the near infrared light; a distance image sensor (TOF sensor 15) that receives the near infrared light outputted from the light splitting unit, and generates a distance image; an imaging sensor (Bayer sensor 13) that receives the visible light outputted from the light splitting unit, and generates a photographic subject image; a correction signal calculation unit (measuring unit 17) that calculates a correction signal for each prescribed block on the basis of the photographic subject image generated by the imaging sensor; and a distance measuring unit (measuring unit 17) that measures the distance to the photographic subject for each prescribed block, using the correction signals calculated by the correction signal generation unit and the distance image generated by the distance image sensor.　The present invention can measure the distance to a photographic subject with high accuracy, without being affected by the photographic subject or the external environment.

Description

distance measuring device

The present invention relates to a distance measuring device.

Conventionally, a camera equipped with an image sensor that has sensitivity in the near-infrared wavelength region in addition to visible light and an image signal processing unit that processes image information obtained by the image sensor, and near-infrared light is captured by the camera. A distance measurement camera system is disclosed that includes near-infrared irradiation means for irradiating an area and a monitor for displaying an output image of the camera (see Patent Document 1).

In Patent Document 1, the image sensor acquires, for each pixel, the time difference information of the light emitted from the near-infrared irradiating means until it reaches the image sensor after being reflected by the object. The image signal processing unit calculates the distance for each pixel based on the light time difference information acquired by the image sensor, and outputs color or brightness information according to the distance. As a result, the monitor displays an image whose color or brightness is changed according to the distance.

Japanese Patent Application Laid-Open No. 2021-092420

However, even if the distance to the subject is the same, if the material, color scheme, or orientation of the subject is different, the light reflectance of the subject will be different, so there is a problem that it cannot be measured accurately. Moreover, even if the object is the same, if the inclination of the object changes, the light reflectance of the object also changes. Furthermore, if the amount of unexpected scattered light or ambient light is large, the ambient light or the like may enter the image sensor, making accurate measurement impossible.

The present invention has been proposed in view of such circumstances.

A distance measuring device according to the present invention includes a light-emitting element that emits near-infrared light to irradiate a subject with the near-infrared light, and a light-emitting element that emits near-infrared light to irradiate the subject with the near-infrared light. A light separation unit that separates external light and outputs the visible light and the near-infrared light, respectively; and receives the near-infrared light output from the light separation unit to generate a distance image. a distance image sensor, an imaging sensor that receives the visible light output from the light separation unit and generates a subject image, and based on the subject image generated by the imaging sensor, for each predetermined block, Using a correction signal calculation unit that calculates a correction signal, the correction signal calculated by the correction signal generation unit for each of the predetermined blocks, and the distance image generated by the distance image sensor, the subject is detected. and a distance measuring unit that measures the distance of

The present invention can measure the distance to a subject with high accuracy without being affected by the subject or the external environment.

FIG. 1 is a block diagram showing the configuration of the distance measuring device. FIG. 2 is a diagram showing an anti-seven ratio table. FIG. 3 is a schematic diagram showing a yellow cubic corrugated cardboard box facing the distance measuring device. FIG. 4 is a schematic diagram showing a cardboard box when tilted. FIG. 5 is a schematic diagram showing the inclination θ of the cardboard box. FIG. 6 is a diagram showing the signal output level for each frame of the TOF sensor. FIG. 7 is a diagram showing spectral characteristics of sunlight.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing the configuration of the distance measuring device 1. As shown in FIG. The distance measuring device 1 includes a lens 11 that collects light reflected by an object, a beam splitter 12 that separates the light from the lens 11 into visible light and near-infrared light, and the visible light from the beam splitter 12. A Bayer sensor 13 that generates an image signal (object image) corresponding to the image, and a first signal processing unit 14 that performs predetermined signal processing on the object image generated by the Bayer sensor 13 .

Furthermore, the distance measuring device 1 includes a TOF (Time Of Flight) sensor 15 that generates an image signal (distance image) corresponding to the near-infrared light from the beam splitter 12, and a predetermined distance image generated by the TOF sensor 15. a second signal processing unit 16 that performs signal processing, a measurement unit 17 that calculates distance information from an object, an LED light source 18 that outputs pulsed near-infrared light of a predetermined wavelength, and a light source control unit 19 for controlling.

The subject is irradiated not only with natural light or illumination light but also with near-infrared light (for example, wavelength 850 nm) from the LED light source 18 . The light reflected by the subject passes through the lens 11 and enters the beam splitter 12 .

The beam splitter 12 is composed of two rectangular prisms. The oblique faces of the right-angle prisms are bonded together so as to face each other. Visible light (wavelength 450 to 700 nm, for example) out of the light incident on the beam splitter 12 is reflected by the bonding surface (reflecting surface) and then travels straight to enter the Bayer sensor 13 . On the other hand, near-infrared light (for example, wavelength 750 to 1000 nm) of the light incident on the beam splitter 12 is transmitted through the reflecting surface and is incident on the TOF sensor 15 .

That is, the beam splitter 12 separates the light incident from the subject through the same optical axis into visible light and near-infrared light. In this embodiment, the near-infrared light is mainly output from the LED light source 18 . Note that each band of visible light and near-infrared light is an example, and is not limited to the numerical values of the present embodiment. Also, instead of the LED light source 18, a laser light source may be used.

The Bayer sensor 13 has, for example, a plurality of light receiving elements arranged in a matrix. That is, the Bayer sensor 13 is an image sensor in which color filters are arranged in a Bayer array on each light receiving element. The Bayer sensor 13 generates a subject image according to visible light from the beam splitter 12 . The first signal processing unit 14 performs predetermined signal processing on the subject image generated by the Bayer sensor 13 .

The TOF sensor 15 is, for example, a range image sensor having a plurality of light receiving elements (pixels) arranged in a matrix. Note that the pixels of the Bayer sensor 13 and the TOF sensor 15 are not the same in size and number. However, each pixel of the Bayer sensor 13 and the TOF sensor 15 correspond to each other by scaling processing. Therefore, the signal value of each pixel of the TOF sensor 15 is corrected by the signal value of each corresponding pixel of the Bayer sensor 13, as will be described later. Also, the distance from the beam splitter 12 to the Bayer sensor 13 and the distance from the beam splitter 12 to the TOF sensor 15 are substantially equal.

The TOF sensor 15 generates a range image according to the near-infrared light from the beam splitter 12. The second signal processing unit 16 performs predetermined signal processing on the distance image generated by the TOF sensor 15 .

The measurement unit 17 is a computer having a data storage unit, an arithmetic processing unit, and the like. The measurement unit 17 corrects the distance image output from the second signal processing unit 16 using the subject image output from the first signal processing unit 14, thereby producing a distance image that shows the distance to the subject with high accuracy. to generate

The measurement unit 17 first detects the phase difference between the near-infrared light (reference signal) output from the LED light source 18 and the near-infrared light (reflection signal) received by the TOF sensor 15 for each pixel. do. This phase difference corresponds to distance information. Based on the phase difference, the measurement unit 17 calculates a distance image representing distance information to the subject for each pixel.

The measurement unit 17 then uses the subject image based on the visible light received by the Bayer sensor 13 to calculate correction information for each pixel. Finally, the measurement unit 17 corrects the distance image for each pixel using the correction information, and calculates a distance image that shows the distance to the subject with high accuracy.

The correction information will be described in detail below.
(Correction related to the material and color scheme of the subject)
In general, the reflectance of light differs for each object (subject) on which light is incident or reflected. Moreover, even for the same object, the reflectance of light differs for each color of the object. Therefore, even if the distance is actually the same, the distance image to be measured will be different if the object or color of the subject is different. For example, when the subject is a blank sheet of paper and a black sheet of paper, even if the respective distances from the distance measuring device 1 to the blank sheet and the black sheet of paper are the same, the distance images to be measured may differ. Therefore, in this embodiment, the measurement unit 17 stores a reflectance table in advance.

FIG. 2 is a diagram showing a reflectance table. The reflectance table shows, for each object, the corresponding reflectance for different colors of that object. For example, when the color of wood is yellow, ocher, and dark brown, the reflectances are r1, r2, and r3. It should be noted that these reflectances are values when the distance measuring device 1 is directly facing.

The measurement unit 17 creates the above reflectance table by performing learning processing in advance using an image of an object assumed as a subject. Note that the reflectance table is not limited to one created by learning processing, and may be set in advance as known data.

Based on the subject image obtained from the Bayer sensor 13, the measurement unit 17 determines the subject material and color scheme for each pixel, refers to the reflectance table, and calculates the reflectance corresponding to the subject material and color scheme. to select. Thereby, the reflectance of the object can be obtained accurately.

It should be noted that this reflectance is obtained when the subject faces the distance measuring device 1 directly. Hereafter, the reflectance at the time of direct confrontation is R1. However, since the subject does not always face the subject, it is necessary to consider the following.

(Correction for subject tilt)
Even with the same object and color, the reflectance R differs depending on whether the object faces the distance measuring device 1 directly or at an angle. Normally, the reflectance Ra when the object faces straight is higher than the reflectance Rb when the object is tilted (Ra>Rb).

Here, when the radiant energy to the subject is Lout, the incident energy to the TOF sensor 15 is Lin, and the reflectance is R, the following formula holds.
Lin = Lout x R

Therefore, when the subject is tilted compared to when it is facing directly, the reflectances R1 and R2 are different, so the accuracy of the distance image obtained by the TOF sensor 15 is reduced.

Now consider the case where the subject is a yellow cardboard box. At this time, assuming that the signal of one pixel of the subject image when the subject faces the distance measuring device 1 is Vsy, the following equation holds.
Vsy=aR+bG+cB (1)

R, G, and B indicate red, green, and blue signals detected by the Bayer sensor 13, and a, b, and c are coefficients of 0-1. When the yellow cardboard box is tilted with respect to the distance measuring device 1, the color ratio (a:b:c) of the signal of one pixel does not change compared to when it faces the distance measuring device 1. However, the overall brightness (luminance signal) changes. Therefore, the following formula (2) holds.
Vty=Vsy×(Rsy/Rty) (2)
Note that Vty is a signal detected by the Bayer sensor 13 . Rty is the reflectance R1 when the yellow cardboard box faces straight. Rsy is the reflectance when the yellow cardboard box is tilted.

Therefore, the signal when the object is facing directly can be obtained by equation (3).
Vsy=Vty×(Rty/Rsy) (3)
Vty is a signal actually detected by the sensors (Bayer sensor 13 and TOF sensor 15). Rty is the reflectance R1 determined by the material and color scheme of the subject as described above. Rsy is the reflectance determined by the tilt of the subject.

Therefore, it is necessary not only to determine the material and coloration of the subject as described above, but also to obtain the inclination θ of the subject and to obtain the reflectance that takes into account the material, coloration, and inclination of the subject.

FIG. 3 shows a yellow cubic cardboard box facing the distance measuring device 1. FIG. 4 is a cardboard box when tilted. FIG. 5 is a schematic diagram showing the inclination θ of the cardboard box.

As shown in FIGS. 3 and 4, consider the case where the shape of the detected edge of the subject is an isosceles trapezoid even though the front shape of the subject (the shape of the edge when facing the subject) is square. . It should be noted that the length of one side of the shape (square) when the object is directly facing is a. Also, the length of the bottom of the shape (isosceles trapezoid) when the object is tilted is a, which is the same as the length of one side of the square. However, the height (the length between the upper base and the lower base) shall be b.

If the aspect ratio of the object is known, the inclination of the object can also be known. In the case of direct confrontation, the aspect ratio is a:a (=1:1) and the inclination is zero. On the other hand, when the subject is tilted, the aspect ratio is b:a. Therefore, by using the inclination .theta. with respect to when the object is directly facing, the following equation holds.
cos θ=b/a

Therefore, when the object is a cube, the measurement unit 17 can obtain the inclination θ by performing the following calculation.
θ=cos ^-1 (b/a)

The method of calculating the inclination of the subject is not particularly limited, and other methods may be used. For example, the measuring unit 17 detects the shape of the edge of the subject using the subject image from the Bayer sensor 13, and compares the shape of the edge when the subject faces the subject, how much the shape of the detected edge is distorted. The inclination of the subject may be detected by determining .

Further, for example, the measurement unit 17 may track frame images continuously generated at 60 frames per second by the Bayer sensor 13 to determine whether the target subject is facing the front or is tilted. . Then, when the subject is tilted, the measurement unit 17 detects the tilt θ. In this case, if the measurement unit 17 learns the shape of the subject in advance, it can easily detect the inclination θ of the subject. Using the detected tilt θ, the measurement unit 17 obtains the reflectance Rsy when the subject is tilted.

Here, the signal of each pixel of the distance image generated from the TOF sensor 15 corresponds to Vty in Equation (3). Therefore, the measurement unit 17 uses the distance image Vty generated from the TOF sensor 15 and the reflectances Rsy and Rty obtained by the above-described processing according to equation (3) to obtain the distance image when facing straight ahead for each pixel. Vsy can be determined. This distance image shows accurate distance information, taking into consideration the material, coloration, and inclination of the subject.

(Correction considering ambient light and scattered light)
Unexpected disturbance light or scattered light (hereinafter referred to as "disturbance light or the like") exists in the imaging environment. If part of the ambient light or the like enters the TOF sensor 15, the distance image generated by the TOF sensor 15 will have abnormal values.

FIG. 6 is a diagram showing signal output levels of the TOF sensor 15 for each frame. The upper row shows the frame period, the middle row shows the signal output level when there is no disturbance light, and the lower row shows the signal output level when there is disturbance light.

As shown in FIG. 6, when affected by ambient light, the signal output level changes accordingly. As a result, correct distance information cannot be obtained. On the other hand, if the signal output level has already reached the saturation level when the disturbance light or the like is zero, the signal output level remains at the saturation level and does not change under the influence of the disturbance light or the like. However, the signal contains many components such as ambient light.

Therefore, the measurement unit 17 can use the spectral characteristics of sunlight to calculate correction information for removing the influence of disturbance light and the like.

(principle)
First, the measurement unit 17 performs white balance adjustment.
Specifically, the white balance adjustment of the Bayer sensor 13 is performed in the absence of ambient light including near-infrared light (NIR: 850 nm). At this time, a white LED is used as the LED light source 18 . Also, the color of the subject is white. That is, a subject defined as white is used.

Expression (4) is established by the white balance adjustment.
R=G=B=1 (4)
Here, R, G, and B are respective signal values of red light, green light, and blue light generated from the Bayer sensor 13 that has captured a white subject. After the white balance adjustment, the coefficients a, b, and c of the above equation (1) are all 1. Furthermore, if Ts is the signal value of near-infrared light generated from the TOF sensor 15 photographing a white subject, adjustment is made so that Ts=1.

Next, the case of photographing an object in an environment where sunlight is incident, that is, in an environment with ambient light will be described.
FIG. 7 is a diagram showing spectral characteristics of sunlight. The spectral characteristics (curves) of sunlight show the normalized radiant intensity spectrum (maximum intensity is 1) for each wavelength of sunlight. The measurement unit 17 stores in advance the spectral characteristics of sunlight.

If a white subject is photographed under the influence of ambient light, the formula (4) is no longer valid. That is, the signal values of red light, green light, blue light, and near-infrared light change due to the influence of ambient light.
Therefore, the measurement unit 17 refers to the spectral characteristics of sunlight and estimates and calculates the signal value of near-infrared light from the ratio of the signal values of red light, green light, and blue light for each pixel. As a result, the ambient light included in the signal value of each pixel of the distance image generated by the TOF sensor 15 can be obtained.

(Correction example)
Consider how much ambient light there is when the LED light source 18 is off. In the case of FIG. 7, the ratios of the signal values Bs, Gs, and Rs of blue light, green light, and red light detected by the Bayer sensor 13 due to ambient light are, for example, as follows.
Bs:Gs:Rs=0.96:0.98:0.98
Therefore, the signal value of disturbance light detected by the Bayer sensor 13 is given by equation (5).
(Disturbance light signal detected by Bayer sensor 13)
=0.96Bs+0.98Gs+0.98Rs (5)

The measurement unit 17 refers to the spectral characteristics of the sun, and based on the ratio of the signal values Bs, Gs, and Rs of the blue, green, and red light detected by the Bayer sensor 13, determines the near-infrared light. Estimate the signal value. As a result, the ratio of near-infrared light was 0.95, for example.
Therefore, the signal value of ambient light detected by the TOF sensor 15 is given by equation (6).
(Ambient light signal detected by TOF sensor 15)=0.95Ts (6)

Then, the measurement unit 17 turns on the LED light source 18 and calculates the distance to the subject using the signal value of the disturbance light obtained as described above. Note that the method of calculating the distance to the subject is not particularly limited.

For example, the measurement unit 17 may subtract the signal value of disturbance light from the signal value of the distance image when the LED light source 18 is on, and obtain a corrected distance image free from the influence of disturbance light. Then, the distance to the subject may be calculated based on the corrected distance image.

The measuring unit 17 also calculates a corrected distance obtained by converting the signal value of the disturbance light into a distance, calculates the distance to the subject based on the distance image when the LED light source 18 is on, and calculates the distance and the corrected distance. may be used to calculate the exact distance to the subject.

In this way, the measurement unit 17 refers to the spectral characteristics of sunlight and uses the signal values of the blue light, green light, and red light of the subject generated by the Bayer sensor 13 when the LED light source 18 is off. Then, the signal value of disturbance light in the wavelength band of near-infrared light is estimated and calculated. Then, the measurement unit 17 can use the signal value of the disturbance light obtained in this manner to remove the influence of the disturbance and the like, and obtain a highly accurate distance to the object.

As described above, the distance measuring device 1 of the present embodiment generates a distance image and a subject image from light that has passed through the same optical axis via the beam splitter 12, and uses the subject image to generate a distance image for each pixel. can be corrected.

Specifically, the distance measuring device 1 uses the subject image to determine the material and color scheme of the subject for each pixel, and further detects the inclination of the subject with respect to the front to determine the material, color scheme, and inclination of the subject. can be obtained. By correcting the distance image using the reflectance obtained in this way, the distance measuring device 1 can obtain accurate distance information without being affected by the material, color scheme, or inclination of the subject. .

In addition, when disturbance light is incident, the distance measuring device 1 refers to the spectral characteristics of the sun for each pixel, and from the ratio of the signal values of each color of the subject image when the LED light source 18 is turned off, A signal value of ambient light in the wavelength band of near-infrared light is estimated. The distance measuring device 1 uses the estimated signal value of ambient light and the signal value of near-infrared light detected by the TOF sensor 15 for each pixel, thereby removing the influence of ambient light and the like. , it is possible to obtain accurate distance information to the subject.

It should be noted that the present invention is not limited to the above-described embodiments, and can also be applied to designs modified within the scope of the matters described in the claims. For example, the distance measurement device 1 may correct the distance image for each block composed of a plurality of pixels instead of correcting the distance image for each pixel. Moreover, each wavelength of visible light and near-infrared light is an example, and other wavelengths may be used.

1 distance measuring device 11 lens 12 beam splitter 13 bayer sensor,
14 First signal processing unit 15 TOF sensor 16 Second signal processing unit 17 Measurement unit 18 LED light source 19 Light source control unit

Claims

a light-emitting element that emits near-infrared light and irradiates a subject with the near-infrared light;
a light separation unit that separates reflected light from a subject that passes through the same optical axis into visible light and the near-infrared light, and outputs the visible light and the near-infrared light, respectively;
a distance image sensor that receives the near-infrared light output from the light separation unit and generates a distance image;
an imaging sensor that receives the visible light output from the light separation unit and generates a subject image;
a correction signal calculation unit that calculates a correction signal for each predetermined block based on the subject image generated by the imaging sensor;
a distance measuring unit that measures the distance to the subject using the correction signal calculated by the correction signal generating unit and the distance image generated by the distance image sensor for each predetermined block;
distance measuring device with
The distance measuring device according to claim 1, wherein the predetermined block is an area composed of one pixel or a plurality of pixels.
The correction signal calculation unit detects the material and color of the subject based on the subject image generated by the imaging sensor for each predetermined block, and detects the light reflectance of the subject corresponding to the detected material and color. 2. The distance measuring device according to claim 1, wherein said correction signal is calculated based on:
2. The correction signal calculation unit detects a tilt of the subject based on the subject image generated by the imaging sensor for each of the predetermined blocks, and calculates the correction signal based on the detected tilt. A distance measuring device as described.
The correction signal calculation unit calculates disturbance light in a wavelength band of near-infrared light based on the subject image generated by the imaging sensor and prestored spectral characteristics of sunlight for each predetermined block. 2. The distance measuring device according to claim 1, wherein the correction signal is indicative of a signal value of .