JP2020051991A - Depth acquisition device, depth acquisition method, and program - Google Patents

Abstract

To provide a depth acquisition device with which it is possible to accurately acquire the depth to a subject.SOLUTION: This depth acquisition device 1 comprises a memory 200 and a processor 110a. The processor 110a acquires a visible light image generated by imaging a subject based on visible light and preserved in the memory 200, acquires the intensity of infrared light measured by receiving infrared light reflected by the subject and preserved in the memory 200, calculates a first depth that is the distance to the subject on the basis of the visible light image, calculates a second depth that is the distance to the subject on the basis of the intensity of the infrared light, calculates a first reliability mask that represents the reliability of the first depth by a multi-value, calculates a second reliability mask that represents the reliability of the second depth by a multi-value, and calculates an integrated depth by integrating the first and second depths on the basis of the first and second reliability masks.SELECTED DRAWING: Figure 4

Description

  The present disclosure relates to a depth acquisition device that acquires a distance to a subject as depth.

  2. Description of the Related Art Conventionally, a distance measuring device for measuring a distance to a subject has been proposed (for example, see Patent Document 1). This distance measuring device includes a light source and an imaging unit. The light source emits light to the subject. The imaging unit captures the light reflected by the subject. Then, the distance measuring device measures the distance to the subject by converting each pixel value of the image obtained by the imaging into a distance to the subject. That is, the distance measuring device acquires the depth of the image obtained by the imaging unit.

JP 2011-64498 A

  However, the distance measuring device disclosed in Patent Document 1 has a problem that the depth cannot be accurately obtained.

  Thus, the present disclosure provides a depth acquisition device that can accurately acquire the depth that is the distance to a subject.

  A depth acquisition device according to an aspect of the present disclosure includes a memory, and a processor, wherein the processor acquires a visible light image generated by imaging a subject based on visible light and stored in the memory, It is measured by receiving the infrared light radiated from and reflected by the subject, acquires the intensity of the infrared light stored in the memory, and, based on the visible light image, Calculating a first depth that is a distance, calculating a second depth that is a distance to the subject based on the intensity of the infrared light, and expressing the reliability of the first depth in multiple values. Calculating a mask, calculating a second reliability mask representing the reliability of the second depth in multiple values, and calculating the first depth and the second depth mask based on the first reliability mask and the second reliability mask. 2 depth Calculating the integration depth by integrating.

  Note that these comprehensive or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, and the computer program. And any combination of recording media. Further, the recording medium may be a non-transitory recording medium.

  The depth acquisition device according to the present disclosure can accurately acquire the depth that is the distance to the subject. Further advantages and advantages of one aspect of the present disclosure will be apparent from the description and drawings. Such advantages and / or advantages are each provided by some embodiments and by the features described in the specification and drawings, but are not necessarily all provided to obtain one or more of the same features. No need.

FIG. 1 is a block diagram illustrating a hardware configuration of the depth acquisition device according to the embodiment. FIG. 2 is a schematic diagram illustrating a pixel array included in the solid-state imaging device according to the embodiment. FIG. 3 is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source and the exposure timing of the first pixel of the solid-state imaging device in the embodiment. FIG. 4 is a block diagram illustrating an example of a functional configuration of the depth acquisition device according to the embodiment. FIG. 5 is a diagram illustrating an example of the intensity of infrared light acquired by the IR intensity acquisition unit according to the embodiment. FIG. 6 is a diagram illustrating another example of the intensity of infrared light acquired by the IR intensity acquisition unit according to the embodiment. FIG. 7 is a diagram illustrating an example of the intensity of infrared light acquired by the IR intensity acquisition unit and background noise according to the embodiment. FIG. 8 is a diagram illustrating a pattern of the intensity of infrared light having low reliability. FIG. 9 is a diagram illustrating a pattern of infrared light intensity with high reliability. FIG. 10 is a diagram illustrating an example of a method of calculating an IR reliability mask according to the embodiment. FIG. 11 is a flowchart illustrating a processing operation of the depth acquisition device according to the embodiment. FIG. 12 is a flowchart illustrating a detailed processing operation of calculating a reliability mask by the depth acquisition device according to the embodiment. FIG. 13 is a block diagram illustrating an example of a functional configuration of a depth acquisition device according to a modification of the embodiment.

(Knowledge underlying the present disclosure)
The present inventor has found that the following problems occur with respect to the distance measuring device of Patent Document 1 described in the section of “Background Art”.

  As described above, the distance measuring device of Patent Literature 1 irradiates a subject with light from a light source, captures an image of the subject irradiated with the light, acquires an image, and measures the depth of the image. For measuring the depth, a time of flight (ToF) is used. In such a distance measuring device, imaging under different imaging conditions is performed in order to improve ranging accuracy. That is, the distance measuring device performs imaging according to the predetermined imaging condition, and sets an imaging condition different from the predetermined imaging condition according to the imaging result. Then, the distance measuring device performs imaging again according to the set imaging conditions.

  However, it is difficult for the distance measuring device of Patent Document 1 to correctly measure the depth, which is the distance to a subject having low reflectance with respect to irradiation light, even when imaging conditions are changed.

  In order to solve such a problem, a depth acquisition device according to an aspect of the present disclosure includes a memory and a processor, wherein the processor is generated by imaging a subject based on visible light, and is stored in the memory. Acquiring the visible light image, measuring the intensity of the infrared light emitted from the light source and receiving the infrared light reflected by the object, and acquiring the intensity of the infrared light stored in the memory. A first depth that is a distance to the subject is calculated based on the light image, a second depth that is a distance to the subject is calculated based on the intensity of the infrared light, and the reliability of the first depth is calculated. A first reliability mask expressing multiplicity is calculated, a second reliability mask expressing multiplicity of the second depth reliability is calculated, and the first reliability mask and the second reliability mask are calculated. On the basis of, Calculating the integration depth by integrating serial first depth and said second depth. For example, each of the first reliability mask and the second reliability mask is a coefficient of 0 or more and 1 or less. In the calculation of the integrated depth, the first reliability mask and the second reliability mask The integrated depth may be calculated by weighted addition of the first depth and the second depth, which are used as a weighting factor for the first depth and a weighting factor for the second depth, respectively.

  Thereby, not only is the second depth calculated as the distance to the subject based on the intensity of the infrared light reflected by the subject, but also the first depth based on the visible light image is calculated as the distance to the subject. Is calculated. Then, the first depth and the second depth are integrated based on the first reliability mask and the second reliability mask. Therefore, even if the reliability of the second depth calculated based on the intensity of the infrared light is low because the reflectance of the infrared light on the subject is low, the second depth is calculated using the first depth. It can be corrected and calculated as an integrated depth. In other words, even if the second depth is missing, the first depth can compensate for the missing. As a result, the reliability or accuracy of the finally calculated depth can be improved.

  Further, since the first reliability mask and the second reliability mask are each multi-valued, it is possible to precisely integrate the first depth and the second depth, and to calculate an appropriate integrated depth.

  Further, the reception of the infrared light is performed by imaging based on infrared light. In the imaging based on the visible light and the imaging based on the infrared light, a scene to be imaged, a viewpoint and time of imaging, and May be substantially the same.

  This makes it possible to appropriately associate each pixel of the visible light image obtained by imaging based on visible light with each pixel of the infrared light image obtained by imaging based on infrared light. Therefore, the integrated depth can be calculated with high accuracy using the first depth and the second depth corresponding to each other.

  Here, an example of an image of substantially the same scene captured at substantially the same viewpoint and time is an image captured by different pixels of the same image sensor. Such an image is similar to each of the red, green, and blue channel images of a color image captured by the color filter of the Bayer array, and the angle of view, viewpoint, and imaging time of each image are substantially equal. That is, images of substantially the same scene captured at substantially the same viewpoint and time do not differ by more than two pixels in the position of the subject in each captured image. For example, if a scene has a point light source having visible light and infrared components, and only one pixel is captured with high brightness in the visible light image, the pixel captured in the visible light image is also used in the infrared light image. The point light source is imaged near two pixels of the pixel corresponding to the position. Further, “imaging at substantially the same time” indicates that the difference between the imaging times is equal in one frame or less.

  Further, the intensity of the infrared light acquired from the memory is at least three times different from each other when the infrared light emitted from the light source and reflected by the subject is received by the image sensor. In the calculation of the second depth, the second depth may be calculated based on the at least three intensities, the second depth being calculated from at least three intensities measured by exposure of the image sensor.

  Thus, the second depth can be calculated with high accuracy using ToF.

  In the calculation of the first reliability mask, the first reliability mask may be calculated by subtracting the second reliability mask from one.

  This makes it possible to reduce the calculation load as compared with the case where the first reliability mask is calculated from the visible light image.

  In the calculation of the integrated depth, the second depth may be calculated as the integrated depth when the second reliability mask is equal to or larger than a threshold value.

  Accordingly, when the second reliability mask is equal to or larger than the threshold, the reliability of the second depth is sufficiently high, and there is no need to integrate the first depth and the second depth. Therefore, in such a case, by calculating the second depth as the integrated depth, the processing for integration can be omitted, and the processing load can be reduced.

  Further, in the calculation of the integrated depth, when the second depth cannot be calculated, the first depth may be calculated as the integrated depth.

  For example, when infinity such as the sky is treated as a subject, the second depth cannot be appropriately calculated. Therefore, in such a case, by calculating the first depth as the integrated depth, the loss of the second depth can be easily compensated for by the first depth.

  In the calculation of the second reliability mask, the second reliability mask may be calculated based on a ratio between a maximum intensity and a minimum intensity of the at least three intensities.

  The largest of the at least three intensities can be considered as signal strength, and the smallest can be considered as noise strength. Therefore, since the second reliability mask is calculated based on the ratio between the maximum intensity and the minimum intensity, that is, the SN ratio, the second reliability mask can be appropriately calculated.

  Further, a depth acquisition device according to another aspect of the present disclosure includes a memory, and a processor, wherein the processor acquires a visible light image generated by imaging a subject based on visible light and stored in the memory. The infrared light emitted from the light source and reflected by the subject is measured by receiving the infrared light, the intensity of the infrared light stored in the memory is obtained, and based on the visible light image, A first depth that is a distance to the subject is calculated, and a second depth that is a distance to the subject is calculated based on the intensity of the infrared light, the visible light image, the intensity of the infrared light, By inputting the first depth and the second depth to a learning model, an integrated depth obtained by integrating the first depth and the second depth may be obtained from the learning model.

  Accordingly, if the learning model is trained in advance so that a correct integrated depth is output in response to the input of the visible light image, the intensity of the infrared light, the first depth, and the second depth, the first reliability is obtained. The appropriate integrated depth can be easily obtained without calculating the reliability mask and the second reliability mask.

  Note that these comprehensive or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, and the computer program. Alternatively, it may be realized by an arbitrary combination of recording media. Further, the recording medium may be a non-transitory recording medium.

  Hereinafter, embodiments will be specifically described with reference to the drawings.

  Each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and do not limit the present disclosure. In addition, among the components in the following embodiments, components not described in the independent claims indicating the highest concept are described as arbitrary components.

  In addition, each drawing is a schematic diagram, and is not necessarily strictly illustrated. In each drawing, the same components are denoted by the same reference numerals.

(Embodiment)
[Hardware configuration]
FIG. 1 is a block diagram illustrating a hardware configuration of the depth acquisition device 1 according to the embodiment. The depth acquisition device 1 according to the present embodiment captures an image based on infrared light (or near infrared light) and an image based on visible light for substantially the same scene, and substantially It has a hardware configuration that can be obtained by imaging at the same viewpoint and time. In addition, substantially the same means that they are the same to the extent that the effects of the present disclosure can be obtained.

  As shown in FIG. 1, the depth acquisition device 1 includes a light source 10, a solid-state imaging device 20, a processing circuit 30, a diffusion plate 50, a lens 60, and a bandpass filter 70.

  The light source 10 emits irradiation light. More specifically, the light source 10 emits irradiation light to irradiate the subject at a timing indicated by the light emission signal generated by the processing circuit 30.

  The light source 10 includes, for example, a capacitor, a driving circuit, and a light emitting element, and emits light by driving the light emitting element with electric energy stored in the capacitor. The light emitting element is realized by, for example, a laser diode, a light emitting diode, or the like. The light source 10 may be configured to include one type of light emitting element, or may be configured to include a plurality of types of light emitting elements according to the purpose.

  Hereinafter, the light emitting element is, for example, a laser diode that emits near infrared light, a light emitting diode that emits near infrared light, or the like. However, the irradiation light emitted by the light source 10 need not be limited to near-infrared light. The irradiation light emitted by the light source 10 may be, for example, infrared light in a frequency band other than near-infrared light (also referred to as infrared light). Hereinafter, in the present embodiment, the irradiation light emitted from the light source 10 will be described as infrared light, but the infrared light may be near infrared light, or infrared light in a frequency band other than near infrared light. It may be.

  The solid-state imaging device 20 captures an image of a subject and outputs an imaging signal indicating an exposure amount. More specifically, the solid-state imaging device 20 performs exposure at a timing indicated by the exposure signal generated by the processing circuit 30, and outputs an imaging signal indicating an exposure amount.

  The solid-state imaging device 20 has a pixel array in which a first pixel that captures an image using reflected light of irradiation light reflected by a subject and a second pixel that captures an image of the subject are arranged in an array. For example, the solid-state imaging device 20 may have a logic function such as a cover glass and an AD converter as needed.

  Hereinafter, similar to the irradiation light, the description will be made on the assumption that the reflected light is infrared light. However, the reflected light does not need to be limited to the infrared light as long as the irradiation light is light reflected by the subject.

  FIG. 2 is a schematic diagram illustrating the pixel array 2 included in the solid-state imaging device 20.

  As shown in FIG. 2, the pixel array 2 includes a first pixel 21 (IR pixel) that captures an image using reflected light of irradiation light reflected by the subject, and a second pixel 22 (BW pixel) that captures an image of the subject. Are arranged in an array so that are alternately arranged in column units.

  Further, in FIG. 2, in the pixel array 2, the second pixels 22 and the first pixels 21 are arranged so as to be adjacent to each other in the row direction, and are arranged so as to be arranged in a stripe shape in the row direction. Instead, they may be arranged every other row (for example, every other row). That is, the first row in which the second pixels 22 are arranged adjacent to each other in the row direction and the second row in which the first pixels 21 are arranged adjacent to each other in the row direction are arranged every M rows (M is a natural number). They may be arranged alternately. Further, the first row in which the second pixels 22 are arranged adjacent to each other in the row direction and the second row in which the first pixels 21 are arranged adjacent to each other in the row direction are different row intervals (the first row is different). The N rows and the second rows may be arranged such that L rows are alternately repeated (N and L are different natural numbers).

  The first pixel 21 is realized by, for example, an infrared light pixel having sensitivity to infrared light that is reflected light. The second pixel 22 is realized by, for example, a visible light pixel having sensitivity to visible light.

  The infrared light pixel includes, for example, an optical filter (also referred to as an IR filter) that transmits only infrared light, a microlens, a light receiving element serving as a photoelectric conversion unit, and a storage unit that stores charges generated by the light receiving element. It is comprised including. Therefore, an image indicating the luminance of the infrared light is expressed by the imaging signals output from the plurality of infrared light pixels (that is, the first pixels 21) included in the pixel array 2. Hereinafter, this infrared light image is also referred to as an IR image or an infrared light image.

  The visible light pixel includes, for example, an optical filter (also referred to as a BW filter) that transmits only visible light, a microlens, a light receiving element as a photoelectric conversion unit, and a storage unit that accumulates charges converted by the light receiving element. It is comprised including. Therefore, the visible light pixel, that is, the second pixel 22 outputs an imaging signal indicating luminance and color difference. That is, a color image indicating the luminance and color difference of visible light is represented by the imaging signals output from the plurality of second pixels 22 included in the pixel array 2. The optical filter of the visible light pixel may transmit both the visible light and the infrared light, and may specify the visible light such as red (R), green (G), or blue (B). May be transmitted.

  Further, the visible light pixel may detect only the luminance of the visible light. In this case, the visible light pixel, that is, the second pixel 22 outputs an imaging signal indicating luminance. Therefore, a monochrome image indicating the luminance of visible light, in other words, a monochrome image, is represented by the imaging signals output from the plurality of second pixels 22 included in the pixel array 2. Hereinafter, this monochrome image is also referred to as a BW image. The above-described color image and BW image are collectively referred to as a visible light image.

  Returning to FIG. 1 again, the description of the depth acquisition device 1 will be continued.

  The processing circuit 30 calculates subject information on the subject using the imaging signal output by the solid-state imaging device 20.

  The processing circuit 30 is configured by an arithmetic processing device such as a microcomputer, for example. The microcomputer includes a processor (microprocessor), a memory, and the like, and generates a light emission signal and an exposure signal by executing a driving program stored in the memory by the processor. Note that the processing circuit 30 may use an FPGA, an ISP, or the like, and may be configured by one piece of hardware or a plurality of pieces of hardware.

  The processing circuit 30 calculates the distance to the subject by, for example, a ToF distance measurement method performed using an imaging signal from the first pixel 21 of the solid-state imaging device 20.

  Hereinafter, the calculation of the distance to the subject by the ToF ranging method performed by the processing circuit 30 will be described with reference to the drawings.

  FIG. 3 shows the light emission timing of the light emitting element of the light source 10 and the exposure timing of the first pixel 21 of the solid-state imaging device 20 when the processing circuit 30 calculates the distance to the subject using the ToF ranging method. FIG. 4 is a timing chart showing the relationship of FIG.

  In FIG. 3, Tp is a light emission period in which the light emitting element of the light source 10 emits irradiation light, and Td is a reflection light in which the irradiation light is reflected by the subject after the light emitting element of the light source 10 emits irradiation light. , The delay time before returning to the solid-state imaging device 20. The first exposure period has the same timing as the light emission period in which the light source 10 emits the irradiation light, and the second exposure period has the timing from the end of the first exposure period to the elapse of the light emission period Tp. It has become.

  In FIG. 3, q1 indicates the total amount of exposure of the first pixel 21 of the solid-state imaging device 20 due to the reflected light within the first exposure period, and q2 indicates the solid-state imaging device due to the reflected light within the second exposure period. 20 shows the total amount of exposure in the first pixel 21 of FIG.

  The emission of the irradiation light by the light emitting element of the light source 10 and the exposure by the first pixel 21 of the solid-state imaging device 20 are performed at the timing shown in FIG. (Equation 1).

  d = c × Tp / 2 × q2 / (q1 + q2) (formula 1)

  Therefore, the processing circuit 30 can calculate the distance to the subject using the imaging signal from the first pixel 21 of the solid-state imaging device 20 by using (Equation 1).

  Further, the plurality of first pixels 21 of the solid-state imaging device 20 may be exposed for a third exposure period Tp after the first exposure period and the second exposure period have ended. The plurality of first pixels 21 can detect noise other than reflected light based on the exposure amount obtained in the third exposure period Tp. That is, the processing circuit 30 removes noise from each of the exposure amount q1 in the first exposure period and the exposure amount q2 in the second exposure period in the above (Equation 1), thereby more accurately determining the distance d to the subject. Can be calculated.

  Returning to FIG. 1 again, the description of the depth acquisition device 1 will be continued.

  The processing circuit 30 may detect a subject and calculate a distance to the subject, for example, using an imaging signal from the second pixel 22 of the solid-state imaging device 20.

  That is, the processing circuit 30 may detect the subject and calculate the distance to the subject based on the visible light image captured by the plurality of second pixels 22 of the solid-state imaging device 20. Here, the detection of the subject may be realized by, for example, determining the shape by pattern recognition by detecting the edge of a singular point of the subject, or processing such as Deep Learning using a learning model learned in advance. It may be realized by. The calculation of the distance to the subject may be performed using world coordinate transformation. Of course, the detection of the subject may be realized by a multi-modal learning process using not only the visible light image but also the luminance and distance information of the infrared light captured by the first pixel 21.

  The processing circuit 30 generates a light emission signal indicating a light emission timing and an exposure signal indicating an exposure timing. Then, the processing circuit 30 outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging device 20.

  The processing circuit 30 generates and outputs an emission signal so as to cause the light source 10 to emit light at a predetermined cycle, and generates and outputs an exposure signal so as to expose the solid-state imaging device 20 at a predetermined cycle. Alternatively, the depth acquisition device 1 may realize continuous imaging at a predetermined frame rate. Further, the processing circuit 30 includes, for example, a processor (microprocessor), a memory, and the like, and generates a light emission signal and an exposure signal by executing a driving program stored in the memory by the processor.

  The diffusion plate 50 adjusts the intensity distribution and angle of the irradiation light. In adjusting the intensity distribution, the diffusion plate 50 makes the intensity distribution of the irradiation light from the light source 10 uniform. In the example illustrated in FIG. 1, the depth acquisition device 1 includes the diffusion plate 50, but does not have to include the diffusion plate 50.

  The lens 60 is an optical lens that condenses light entering from outside the depth acquisition device 1 on the surface of the pixel array 2 of the solid-state imaging device 20.

  The band-pass filter 70 is an optical filter that transmits infrared light that is reflected light and visible light.

  The depth acquisition device 1 having the above configuration is used by being mounted on a transportation device. For example, the depth acquisition device 1 is used by being mounted on a vehicle traveling on a road surface. The transport equipment on which the depth acquisition device 1 is mounted is not necessarily limited to a vehicle. The depth acquisition device 1 may be used by being mounted on a transportation device other than a vehicle, such as a motorcycle, a boat, and an airplane.

[Function Configuration of Depth Acquisition Device]
The depth acquisition device 1 according to the present embodiment uses the hardware configuration shown in FIG. 1 to capture an IR image and a BW image for substantially the same scene, and has substantially the same viewpoint and time. Is acquired by imaging. Then, the depth acquisition device 1 integrates the depth at each position in the IR image obtained from the IR image and the depth at each position in the BW image obtained from the BW image to obtain the depth. The integrated depth at each position is calculated. In the present embodiment, the IR image indicates the intensity of infrared light at each pixel position. Therefore, acquisition of the IR image means acquisition of the intensity of infrared light measured by each first pixel 21 of the solid-state imaging device 20 receiving infrared light.

  FIG. 4 is a block diagram illustrating an example of a functional configuration of the depth acquisition device 1.

  The depth acquisition device 1 includes a light source 101, an IR camera 102, a BW camera 103, a processor 110a, and a memory 200. The depth acquisition device 1 according to the present embodiment includes the light source 101, the IR camera 102, and the BW camera 103, but may not include these components, and may include only the processor 110a and the memory 200.

  The light source 101 may be composed of the light source 10 and the diffusion plate 50 shown in FIG. 1, and emits infrared light to a subject by emitting light.

  The IR camera 102 is also called an infrared light camera, and may be configured by the plurality of first pixels 21 of the solid-state imaging device 20 shown in FIG. Such an IR camera 102 acquires an IR image by performing imaging based on infrared light of a scene including the subject in accordance with the timing at which the light source 101 irradiates the subject with infrared light.

  The BW camera 103 is also called a visible light camera, and may be configured by the plurality of second pixels 22, the lens 60, and the bandpass filter 70 of the solid-state imaging device 20 illustrated in FIG. Such a BW camera 103 is an imaging based on visible light of a scene that is substantially the same as an IR image, and performs imaging at substantially the same viewpoint and at the same time as the IR image. An optical image (specifically, a BW image) is acquired.

  The memory 200 is a recording medium for storing an IR image obtained by imaging by the IR camera 102 and a BW image obtained by imaging by the BW camera 103. Note that, specifically, such a memory 200 may be a ROM (Read Only Memory), a RAM (Random Access Memory), a solid state drive (SSD), or the like, and may be nonvolatile or volatile. Good. Further, the memory 200 may be a hard disk.

  The processor 110a acquires the IR image and the BW image from the memory 200, and integrates the depth calculated from the IR image and the depth calculated from the BW image. The processor 110a includes a light emission control unit 113, an IR intensity acquisition unit 114, a BW image acquisition unit 115, a BW depth calculation unit 111a, an IR depth calculation unit 111b, an integrated depth calculation unit 111c, And a mask calculation unit 112.

  The light emission control unit 113 controls the light source 101. That is, the light emission control unit 113 causes the light source 101 to emit light by outputting the above light emission signal to the light source 101. As a result, the subject is irradiated with infrared light from the light source 101, and reflected light, which is light reflected by the subject, enters the IR camera 102.

  The BW image acquisition unit 115 acquires a BW image from the BW camera 103 via the memory 200. That is, the BW image acquisition unit 115 acquires a BW image generated by imaging a subject based on visible light and stored in the memory 200.

  The IR intensity acquisition unit 114 acquires an IR image from the IR camera 102 via the memory 200. Here, each pixel of the IR image indicates the intensity of infrared light received at the position of the pixel as a pixel value (specifically, luminance). Therefore, the IR intensity acquisition unit 114 acquires the intensity of infrared light by acquiring an IR image. That is, the IR intensity acquisition unit 114 measures the intensity of the infrared light emitted from the light source 101 and reflected by the subject, and acquires the intensity of the infrared light stored in the memory 200.

  The BW depth calculation unit 111a calculates a BW depth, which is a distance to a subject, based on the BW image acquired by the BW image acquisition unit 115. The BW depth calculation unit 111a corresponds to a first depth calculation unit.

  The IR depth calculation unit 111b calculates the IR depth, which is the distance to the subject, based on the intensity of the infrared light acquired by the IR intensity acquisition unit 114. The IR depth calculator 111b corresponds to a second depth calculator.

  The reliability mask calculation unit 112 calculates a BW reliability mask representing the BW depth reliability in multiple values, and further calculates an IR reliability mask representing the IR depth reliability in multiple values.

  The integrated depth calculation unit 111c calculates an integrated depth by integrating the BW depth and the IR depth based on the BW reliability mask and the IR reliability mask.

  In the depth acquiring device 1 according to the present embodiment, not only the IR depth is calculated as the distance to the subject based on the intensity of the infrared light reflected by the subject, but also the BW depth based on the BW image. Is calculated as the distance to the subject. Then, the BW depth and the IR depth are integrated based on the BW reliability mask and the IR reliability mask. Therefore, even if the reliability of the IR depth calculated based on the intensity of the infrared light is low due to the low reflectance of the infrared light on the subject, the IR depth is corrected using the BW depth, It can be calculated as an integrated depth. In other words, even if the IR depth is missing, the BW depth can compensate for the missing. As a result, the reliability or accuracy of the depth to the subject finally calculated can be improved.

  Further, since the BW reliability mask and the IR reliability mask are each multi-valued, it is possible to precisely integrate the BW depth and the IR depth, and to calculate an appropriate integrated depth.

  In the present embodiment, the infrared light is received by imaging based on infrared light. In the imaging based on visible light and the imaging based on infrared light, the scene to be imaged and the viewpoint of the imaging are different. And the time are substantially the same. This makes it possible to appropriately associate each pixel of the BW image obtained by imaging based on visible light with each pixel of the IR image obtained by imaging based on infrared light. Therefore, the integrated depth can be calculated with high accuracy using the mutually corresponding BW depth and IR depth.

[Infrared light intensity]
FIG. 5 shows an example of the intensity of the infrared light acquired by the IR intensity acquisition unit 114.

  The solid-state imaging device 20 used in the IR camera 102 according to the present embodiment receives infrared light emitted from the light source 101 and reflected by the subject. Here, the infrared light emitted from the light source 101 is pulse light, and the intensity of the infrared light reflected by the subject is reflected or internally scattered as shown in FIG. Spread in the time direction.

  When causing the solid-state imaging device 20 to receive reflected light, the IR intensity acquisition unit 114 in the present embodiment outputs the above-described exposure signal to the solid-state imaging device 20 according to the timing at which the light source 101 irradiates infrared light. . As a result, the solid-state imaging device 20 is exposed at least three different times. The timing is also referred to as exposure timing. In this embodiment, the exposure timing is three times.

When the solid-state imaging device 20 receives the infrared light that is the reflected light, the solid-state imaging device 20 receives a section T1 and a section T2 subsequent to the section T1 as shown in FIG. Infrared light is received in each of the section T3 following the section T2. The sections T1, T2 and T3 are sections exposed at different exposure timings, and the time intervals of these sections are the same. This time interval is determined based on the range of the distance to be measured. For example, if the distance L to the main subject is 15 m and the light speed c is 3 × 10 ⁸ m / s, the time tr in which the reflected light returns ideally is tr = 2 L / c = 100 ns. Therefore, the reflected light can be received by using the time 100 ns at which the reflected light of the measurement target ideally returns in the above-mentioned time interval. Note that this time interval is an example, and the distance can be calculated if the reflected light is received. Therefore, it is not necessary to use the time interval exactly.

  When the subject is close to the IR camera 102, the intensity of the reflected light received by the solid-state imaging device 20 is large in the earliest section T1 of the sections T1 to T3 and decreases with time. The relationship between the distance to the subject and the reflected light intensity will be described later. Note that the solid-state imaging device 20 receives background light as background noise in any section. Most of the infrared light received in the section T3 is not reflected light from the light source 101 but background noise.

  Here, the distance between the subject and the IR camera 102 will be described. Assuming that the time when the infrared light is emitted is 0 and the time at the center of the section T1 is T1M, the distance LM that the light propagates in the time from time 0 to time T1M is LM = c × T1M. Focusing on a certain pixel, if the distance to the subject corresponding to the pixel is smaller than LM / 2, it means that the distance between the subject and the IR camera 102 is short. On the other hand, if the distance to the subject corresponding to the pixel is larger than LM / 2, it means that the distance between the subject and the IR camera 102 is long. In other words, the case where the peak of the reflected light has already passed during the exposure time of the section T1 is near, and the case where it has not passed yet is far. Therefore, when the peak of the reflected light has already passed in the section T1, the intensity of the reflected light does not increase in the sections T2 and T3. On the other hand, when the distance to the subject is long, since the peak has not yet reached in the section T1, the peak of the intensity of the reflected light approaches or reaches the section T2 and the section T3. An intensity higher than the intensity of the reflected light is observed in the section T2, and an intensity higher than the section T2 is observed in the section T3. However, when the spread of the reflected light in the time direction is equal to or less than the above-described time interval, and the peak of the reflected light is observed in the section T2, the intensity of the reflected light observed in the section T3 is smaller than that in the section T2.

  As described above, in the present embodiment, the intensity of the infrared light acquired from the memory 200 by the IR intensity acquisition unit 114 is such that the infrared light emitted from the light source 101 and reflected by the subject is received by the solid-state imaging device 20. At least three intensities measured by exposure of the solid-state imaging device 20 at at least three different timings.

  Further, as shown in FIG. 5B, each first pixel 21 of the solid-state imaging device 20 calculates the accumulated value of the intensity of the reflected light (that is, the infrared light) in each section by the time corresponding to the section. Is output as the intensity of the infrared light at. That is, the solid-state imaging device 20 outputs the cumulative value of the intensity of the infrared light in the section T1 as the intensity of the infrared light at the time t1. Similarly, the solid-state imaging device 20 outputs the cumulative value of the infrared light intensity in the sections T2 and T3 as the infrared light intensity at time t2 and time t3, respectively. The IR intensity acquisition unit 114 acquires the accumulated value of the intensity of the infrared light in each of the three sections as three intensities measured by exposure of the solid-state imaging device 20 at three different timings. I do.

  Note that the times t1, t2, and t3 are intermediate times in each of the sections T1, T2, and T3, and may be treated as different exposure timings. These times indicate the elapsed time from the time when the light source 101 emits infrared light. Further, in the following description, the intensity of the infrared light received or acquired at the exposure timing means a cumulative value of the intensity of the infrared light in a section corresponding to the exposure timing.

  FIG. 6 shows another example of the intensity of the infrared light acquired by the IR intensity acquisition unit 114.

  When the subject is close to the IR camera 102, as shown in FIG. 6A, the intensity of the infrared light output from the solid-state imaging device 20 is largest at the exposure timing t1, and is small at the subsequent exposure timing t2. It is the smallest at the later exposure timing t3. Note that the intensity of the infrared light at the exposure timing t3 is close to the intensity of the background noise.

  Conversely, when the subject is far from the IR camera 102, as shown in FIG. 6B, the intensity of the infrared light output from the solid-state imaging device 20 is the smallest at the exposure timing t1, and the subsequent exposure timing t2 Is larger at the later exposure timing t3. Note that the intensity of the infrared light at the exposure timing t1 is close to the intensity of the background noise.

[Method of calculating IR depth]
FIG. 7 illustrates an example of the intensity of the infrared light acquired by the IR intensity acquiring unit 114 and the background noise.

  As described above, the distance measurement method using pulsed light is called ToF (Time of Flight). In this ToF, the depth, which is the distance to the subject, is calculated based on the time from the irradiation of the pulse light to the reflection by the subject until the reflected light of the pulse light is observed. Assuming that the depth is L, the speed of light is c, and the elapsed time is Δt, the depth is ideally calculated by L = cΔt / 2. Since the pulse light is reflected, the optical path is twice the depth.

  In order to increase the distance resolution, a large number of exposures may be performed in a short time, and it may be determined at which exposure timing the reflected light is observed. However, if the exposure time is short, the intensity of the received light decreases. That is, the SN ratio decreases, and the reliability of the calculated depth decreases.

  Therefore, in the present embodiment, reflected light is observed with a small number of exposures, that is, three exposures. In the present embodiment, although infrared light is used as pulse light, ambient light (that is, background light) also includes an infrared component. Therefore, as shown in FIG. 7, the intensity Iback of the background light is always observed as background noise. This background light is considered to be included by the same intensity in the infrared light acquired at each of all the exposure timings. Therefore, the influence of the background light can be removed by using the difference in the intensity of the infrared light obtained at different exposure timings.

  There are a plurality of methods for calculating the distance from the intensity of infrared light acquired at three exposure timings. In a simple example, the IR depth calculation unit 111b regards the infrared light having the minimum intensity as the background light among the infrared lights acquired at each exposure timing, and determines the infrared light other than the minimum infrared light. Exclude background light intensity from intensity. This makes it possible to acquire the intensity of the infrared light that is not affected by the background light. Specifically, as shown in FIG. 7, the IR depth calculation unit 111b regards the intensity I3 of the infrared light acquired at the exposure timing t3 as the intensity Iback of the background light. Then, the IR depth calculation unit 111b removes the intensity I3 as the background light intensity Iback from the intensity I1 of the infrared light acquired at the exposure timing t1, and calculates the background light from the intensity I2 of the infrared light acquired at the exposure timing t3. The intensity I3 is excluded as the intensity Iback. Thus, at each of the times t1 and t2, the intensity of the infrared light not affected by the background light is calculated.

  Here, in the example shown in FIG. 7, the IR depth calculation unit 111b may use the elapsed time until the exposure timing t1 when the infrared light having the maximum intensity I1 is received as the elapsed time Δt of the ToF. Further, the IR depth calculation unit 111b uses the exposure timing t1 at which the infrared light having the maximum intensity I1 is received and the exposure timing t2 at which the infrared light having the intensity I2 next to the intensity I1 is received. The elapsed time Δt of ToF may be calculated.

  For example, when the intensity I1 of the infrared light is equal to the intensity I2, the IR depth calculation unit 111b may calculate the elapsed time up to an intermediate timing between the exposure timings t1 and t2 as the elapsed time Δt of the ToF. . That is, the IR depth calculation unit 111b may calculate the elapsed time Δt from a weighted average of the exposure timings t1 and t2 using the intensity I1 and the intensity I2 as weighting factors.

[Reliability of infrared light intensity]
The output of the infrared light pulse when the light source 101 emits light can be regarded as constant. Therefore, the intensity of the reflected light observed in the solid-state imaging device 20 is expected to monotonously decrease or monotonically increase as shown in FIG. However, in practice, the intensity may not monotonically increase or decrease. Note that the monotonous increase or the monotone decrease in the present embodiment does not include a state where the intensity does not change.

  More specifically, in addition to the above-described monotone increase and monotone decrease, there are, for example, the following three patterns 1, pattern 2, and pattern 3 in the time-varying patterns of the infrared light intensities I1, I2, and I3.

  In pattern 1, the observed intensities I1, I2, and I3 are equal (ie, I1 = I2 = I3). In such a pattern 1, all of these intensities can be regarded as the intensity Iback of the background light, and it is considered that no reflected light was observed. For example, when the subject is an object at infinity such as the sky or hardly reflected, the state of the pattern 1 can occur. Alternatively, when the subject is an object having an extremely high reflectance such as a mirror surface or some road signs, the intensity of the reflected light exceeds the upper limit of the input of the solid-state imaging device 20 (ie, saturates) at all exposure timings. . As a result, all of the intensities I1, I2 and I3 have the same maximum intensity Imax.

  In pattern 2, the intensity I2 at the exposure timing t2 is minimum. Such a state of the pattern 2 occurs when the background light is not constant due to a change in the brightness of external light other than the light source 101 (for example, a light emitting light including an infrared component being turned on). .

  In pattern 3, the intensity I1 is the maximum and the intensity I2 and the intensity I3 are the same, or the intensity I3 is the maximum and the intensity I1 and the intensity I2 are the same. That is, the solid-state imaging device 20 receives the reflected light at the exposure timing t1, and receives only the background light at the exposure timings t2 and t3. That is, I1> I2 = I3 = Iback is satisfied. Alternatively, the solid-state imaging device 20 receives the reflected light at the exposure timing t3, and receives only the background light at the exposure timings t1 and t2. That is, I3> I1 = I2 = Iback is satisfied. Further, as described above, when the spread of the reflected light in the time direction is equal to or less than the exposure time interval and the peak of the reflected light is observed in the section T2 (I2 is maximum), the intensity of the reflected light observed in the section T3 Becomes smaller than the section T2. These intensities are expected normal intensities and the calculated depth is reliable.

  As described above, when the pattern of the intensity of the infrared light at the three exposure timings is the pattern 1 or the pattern 2, the intensity of the three exposure timings is not based on the assumed light receiving intensity but based on those intensities. The reliability of the calculated IR depth is also low.

  FIG. 8 shows an infrared light intensity pattern with low reliability.

  For example, as shown in FIG. 8A, the intensity I1 of the infrared light at the exposure timing t1, the intensity I2 of the infrared light at the exposure timing t2, and the intensity I3 of the infrared light at the exposure timing t3. Are the same as pattern 1 described above. In this case, the infrared light reflected from the subject is too weak or too strong. For example, since the subject is at infinity from the IR camera 102, only the infrared light that is the background light is received as noise by the solid-state imaging device 20 of the IR camera 102. Alternatively, the intensity of the infrared light received by the solid-state imaging device 20 is saturated. Therefore, the reliability of the IR depth calculated based on the intensity of the infrared light is low.

  Further, as shown in FIG. 8B, the intensity I2 of the infrared light at the exposure timing t2 is the intensity I1 of the infrared light at the exposure timing t1, and the intensity I3 of the infrared light at the exposure timing t3. A pattern smaller than any of the above corresponds to the above-described pattern 2. In this case, since the infrared light reflected from the subject cannot be appropriately received due to the influence of the change of the external light, the reliability of the IR depth calculated based on the intensity of the infrared light is not obtained. Sex is low.

  FIG. 9 shows a pattern of infrared light intensity with high reliability.

  For example, as shown in FIG. 9A, a pattern in which the intensity I1, I2, and I3 of the infrared light monotonically decreases with time is a pattern assumed when the subject is close to the IR camera 102. Therefore, the reliability of the depth calculated based on the infrared light intensities I1, I2, and I3 is high. Similarly, as shown in FIG. 9C, a pattern in which the infrared light intensities I1, I2, and I3 monotonically increase with time is a pattern assumed when the subject is far from the IR camera 102. Therefore, the reliability of the depth calculated based on the infrared light intensities I1, I2, and I3 is high.

  Also, as shown in FIG. 9B, the intensity I1 of the infrared light at the exposure timing t1 is the maximum, the intensity I2 of the infrared light at the exposure timing t2, and the intensity of the infrared light at the exposure timing t3. A pattern having the same light intensity I3 corresponds to the above-described pattern 3. Such a pattern is also assumed when the subject is close to the IR camera 102. That is, it is assumed that the infrared light reflected by the subject is received at the exposure timing t1, and that the background light is received instead of the reflected light at the exposure timings t2 and t3. Therefore, the reliability of the depth calculated based on the infrared light intensities I1, I2, and I3 is high.

  As shown in FIG. 9D, the intensity I2 of the infrared light at the exposure timing t2 is the intensity I1 of the infrared light at the exposure timing t1, and the intensity I3 of the infrared light at the exposure timing t3. Patterns larger than any of the above are included in the above-described pattern 3. The pattern shown in FIG. 9D is a pattern assumed when the spread of the reflected light in the time direction is equal to or less than the exposure time interval and the peak of the reflected light is observed in the section T2. Therefore, the reliability of the depth calculated based on these infrared light intensities I1, I2, and I3 is high.

  Thus, from the patterns of the infrared light intensities I1, I2, and I3, the reliability of the infrared light intensities can be determined. Therefore, when the reliability of the intensity of the infrared light is low, the IR depth calculation unit 111b according to the present embodiment clearly shows that the reliability of the IR depth calculated based on the intensity of the infrared light is also low. Therefore, the IR depth need not be calculated.

[Method of calculating IR reliability mask]
FIG. 10 shows an example of a method of calculating an IR reliability mask.

  When calculating the IR reliability mask, the reliability mask calculation unit 112 uses the maximum value and the minimum value of the infrared light intensities I1, I2, and I3 obtained at three exposure timings. That is, the reliability mask calculation unit 112 calculates the SN ratio (signal-noise ratio) by using the maximum value of the intensities as the signal intensity and using the minimum value of the intensities as the noise intensity. I do. Then, the reliability mask calculation unit 112 converts the SN ratio into a multi-valued IR reliability mask indicating a numerical value of 0 or more and 1 or less by normalizing the SN ratio.

  For example, as shown in FIGS. 10A and 10B, the maximum value of the infrared light intensities I1, I2, and I3 obtained at three exposure timings is the intensity I1, and the minimum value is the intensity I3. It is. Therefore, the reliability mask calculation unit 112 calculates the SN ratio by using the intensity I1 as the signal intensity and using the intensity I3 as the noise intensity.

  More specifically, in the case shown in FIG. 10A, the reliability mask calculation unit 112 uses the intensity I1 as the signal intensity S1 and uses the intensity I3 as the noise intensity N1, thereby setting the SN ratio r1 to r1 = S1. / N1. Similarly, in the case shown in FIG. 10B, the reliability mask calculation unit 112 uses the intensity I1 as the signal intensity S2 and the intensity I3 as the noise intensity N2, and thereby sets the SN ratio r2 to r2 = S2 / N2. Is calculated by

  Here, the example shown in FIG. 10A has a larger SN ratio than the example shown in FIG. 10B. As a result, the IR depth calculated based on the infrared light intensities I1, I2, and I3 shown in FIG. 10A is better than the infrared light intensities I1, I2, and I2 shown in FIG. The reliability is higher than the IR depth calculated based on I3. In other words, it can be said that the SN ratio indicates the reliability of the IR depth calculated from the intensity of the infrared light.

  Therefore, the reliability mask calculation unit 112 according to the present embodiment normalizes the SN ratio to convert the SN ratio into a multi-valued IR reliability mask indicating a numerical value of 0 or more and 1 or less as described above. Convert. For example, the reliability mask calculation unit 112 actually calculates the S / N ratio using the intensity of infrared light reflected by the subject having a high reflectance, that is, the S / N ratio assumed to be the maximum, to 1 The SN ratio based on the intensity of the observed infrared light may be normalized.

  As described above, in the present embodiment, the reliability mask calculation unit 112 calculates the IR reliability mask based on the ratio between the maximum intensity and the minimum intensity of the at least three infrared light intensities. . As a result, the IR reliability mask can be appropriately calculated.

  The reliability mask calculation unit 112 uses the common logarithm log (S / N) or natural logarithm In (S / N) instead of the SN ratio, and normalizes the log (S / N) and the like. , IR reliability masks may be calculated. Note that S is the maximum value among the three infrared light intensities I1, I2, and I3 as described above, and N is the minimum value among them. Such an IR reliability mask is calculated for each of the plurality of first pixels 21 included in the solid-state imaging device 20 with respect to the infrared light received by the first pixels 21.

[Method of calculating BW depth]
The BW depth calculation unit 111a calculates a BW depth, which is a distance to a subject, based on the BW image. Specifically, the BW depth calculation unit 111a calculates the BW depth by SLAM (Simultaneous Localization and Mapping). and Map Building (SLAM) Problem, IEEE Transactions on Robotics and Automation, Vol. 17, No. 3, 2001). In other words, when the depth acquisition device 1 is mounted on a vehicle and the vehicle is moving, the BW depth calculation unit 111a obtains two different time frames obtained by imaging the BW camera 103 while moving. BW depth is calculated using the BW image. Such SLAM makes it possible to perform stereo ranging even with the BW camera 103 which is a monocular camera, and the depth of the subject can be calculated as the BW depth. Note that a method of simultaneously estimating the three-dimensional position and the self-position of the subject is generally referred to as SLAM. In particular, a method using only the BW image as in the present embodiment is also referred to as Visual SLAM.

[Method of calculating BW reliability mask]
The reliability mask calculation unit 112 calculates a BW reliability mask based on the BW image acquired by the BW image acquisition unit 115. In this case, the reliability mask calculation unit 112 calculates, for each pixel of the BW image, a texture amount corresponding to the pixel. For example, the reliability mask calculation unit 112 calculates, for each pixel of the BW image, the sum of luminance gradients in a region including the pixel as a texture amount. Then, the reliability mask calculation unit 112 converts the texture amount into a multi-valued BW reliability mask indicating a numerical value of 0 or more and 1 or less by normalizing the texture amount. For example, the reliability mask calculation unit 112 may normalize the texture amount based on the actually acquired BW image so that the maximum assumed texture amount is normalized to 1.

  Alternatively, the reliability mask calculation unit 112 may calculate the BW reliability mask using the IR reliability mask calculated as described above. In this case, the reliability mask calculation unit 112 calculates the BW reliability mask by subtracting the IR reliability mask from 1. Accordingly, the calculation load can be reduced as compared with the case where the BW reliability mask is calculated from the BW image.

[Method of calculating integrated depth]
The integrated depth calculation unit 111c calculates the integrated depth by integrating the BW depth and the IR depth based on the BW reliability mask and the IR reliability mask calculated as described above.

For example, when the BW reliability mask is calculated from the BW image, that is, when the BW reliability mask and the IR reliability mask are calculated independently, the integrated depth calculation unit 111c determines the IR reliability mask and the BW reliability mask. The reliability masks may be adjusted so that the sum with the reliability mask is 1. Specifically, the integrated depth calculation unit 111c adjusts the BW reliability mask W _BW by W _BW = W _BW / (W _BW + W _IR ), and adjusts the IR reliability mask W _IR to W _IR = W _IR / (W _BW + _WIR ). Then, the integrated depth calculation unit 111c calculates the integrated depth D according to D = W _BW × D _BW + W _IR × D _IR . Here, _DBW is the BW depth calculated by the BW depth calculator 111a, and _DIR is the IR depth calculated by the IR depth calculator 111b.

  That is, the BW reliability mask and the IR reliability mask each have a coefficient of 0 or more and 1 or less. Then, the integrated depth calculation unit 111c calculates the integrated depth by performing weighted addition of the BW depth and the IR depth using the BW reliability mask and the IR reliability mask as a weight coefficient for the BW depth and a weight coefficient for the IR depth, respectively. calculate.

When the BW reliability mask is calculated from the IR reliability mask, the integrated depth calculation unit 111c calculates the integrated depth D according to D = (1−W _IE ) × D _BW + W _IR × D _IR . .

[Processing flow]
FIG. 11 is a flowchart illustrating a processing operation of the depth acquisition device 1 according to the present embodiment.

(Step S110)
First, the BW camera 103 of the depth acquisition device 1 acquires a BW image by imaging a subject based on visible light. This BW image is stored in the memory 200. Therefore, the BW image acquisition unit 115 of the processor 110a acquires the BW image from the BW camera 103 via the memory 200.

(Step S120)
Next, the IR camera 102 acquires an IR image by capturing an image of a subject irradiated with infrared light from the light source 101. Here, the luminance of each pixel of the IR image indicates the intensity of the infrared light. Therefore, in step S120, the IR camera 102 receives the infrared light emitted from the light source 101 and reflected by the subject to acquire the intensity of the infrared light. The IR image is stored in the memory 200. Therefore, the IR intensity acquisition unit 114 of the processor 110a acquires the IR image from the IR camera 102 via the memory 200. That is, the IR intensity acquiring unit 114 acquires the intensity of the infrared light measured by receiving the infrared light reflected by the subject and stored in the memory 200. Note that the acquired intensity of the infrared light is an intensity measured by each first pixel 21 included in the solid-state imaging device 20.

(Step S130)
Next, the BW depth calculation unit 111a calculates the BW depth, which is the distance to the subject, based on the BW image. Specifically, the BW depth calculation unit 111a calculates the BW depth using SLAM. In other words, when the depth acquisition device 1 is mounted on a vehicle and the vehicle is moving, the BW depth calculation unit 111a obtains two different time frames obtained by imaging the BW camera 103 while moving. BW depth is calculated using the BW image.

(Step S140)
Next, the IR depth calculation unit 111b calculates the IR depth, which is the distance to the subject, based on the intensity of the infrared light acquired in step S120. That is, the IR depth calculation unit 111b calculates the elapsed time Δt using the infrared light intensities I1, I2, and I3 obtained at the three exposure timings t1, t2, and t3, and applies the elapsed time Δt to the ToF method. Is applied to calculate the IR depth. This IR depth is calculated for each pixel of the IR image, that is, for each first pixel 21 of the solid-state imaging device 20.

  Here, when the reliability of the infrared light intensities I1, I2, and I3 at the three exposure timings t1, t2, and t3 is low, the IR depth calculator 111b calculates the IR depth based on those intensities. It is not necessary. For example, when the pattern of the temporal change in the intensity of the infrared light is the pattern shown in FIGS. 8A and 8B, that is, when the pattern is 1 or 2 described above, the IR depth calculation unit 111b Since the reliability of these infrared light intensities is low, it is not necessary to calculate the IR depth.

(Step S150)
Next, the reliability mask calculation unit 112 calculates a BW reliability mask representing the reliability of the BW depth calculated in step S130 in multiple values, and calculates the reliability of the IR depth calculated in step S140 in multiple values. The IR reliability mask to be represented is calculated. The BW reliability mask and the IR reliability mask each have a numerical value of 0 or more and 1 or less.

(Step S160)
Next, the integrated depth calculation unit 111c integrates the BW depth and the IR depth calculated in steps S130 and S140 based on the BW reliability mask and the IR reliability mask calculated in step S150, thereby calculating the integrated depth. calculate.

When the IR reliability mask is equal to or larger than the threshold, the integrated depth calculation unit 111c may calculate the IR depth as the integrated depth. That is, when IR reliability mask W _IR ≧ threshold Th, integrated depth D is calculated by D = D _IR . In other cases, that is, when the IR reliability mask W _IR <the threshold Th, the integrated depth D is calculated by D = W _BW × D _BW + W _IR × D _IR . The threshold value Th is, for example, Th = 0.5.

  Accordingly, when the IR reliability mask is equal to or larger than the threshold, the reliability of the IR depth is sufficiently high, and there is no need to integrate the BW depth and the IR depth. Therefore, in such a case, by calculating the IR depth as the integrated depth, the processing for integration can be omitted, and the processing load can be reduced.

When the integrated depth calculation unit 111c cannot calculate the IR depth, the integrated depth calculation unit 111c may calculate the BW depth as the integrated depth. For example, as described above, when the pattern of the temporal change in the intensity of the infrared light is the pattern shown in FIGS. 8A and 8B, that is, when the pattern is the above-described pattern 1 or 2, the IR depth Is not calculated. In such a case, integration depth computing unit 111c calculates the BW depth _{D BW} as an integrated depth D.

  For example, when infinity such as the sky is treated as a subject, the IR depth cannot be appropriately calculated. Therefore, in such a case, by calculating the BW depth as the integrated depth, the loss of the IR depth can be easily compensated for by the BW depth.

  FIG. 12 is a flowchart illustrating a detailed processing operation of calculating a reliability mask by the depth acquisition device 1. That is, FIG. 12 shows the detailed processing operation of step S150 shown in FIG.

(Step S151)
First, the reliability mask calculation unit 112 calculates an IR reliability mask. At this time, the reliability mask calculation unit 112 calculates, for each of the plurality of first pixels 21 included in the solid-state imaging device 20, the infrared light at three exposure timings t1, t2, and t3 measured by the first pixel 21. The light intensity I1, I2 and I3 are obtained.

(Step S152)
Next, the reliability mask calculation unit 112 calculates the SN ratio as shown in FIG. 10 based on the infrared light intensities I1, I2, and I3 acquired in step S151. Then, the reliability mask calculation unit 112 normalizes the S / N ratio, and thereby provides a multi-valued IR reliability mask indicating the reliability of the IR depth calculated in step S140 based on the intensity of the infrared light. Is calculated. The IR reliability mask is calculated for each of the plurality of first pixels 21 included in the solid-state imaging device 20.

(Step S153)
Next, the reliability mask calculation unit 112 calculates a BW reliability mask. At this time, the reliability mask calculation unit 112 first calculates, for each of a plurality of pixels included in the BW image, a texture amount corresponding to the pixel. For example, the reliability mask calculation unit 112 calculates, for each pixel of the BW image, the sum of luminance gradients in a region including the pixel as a texture amount.

(Step S154)
For each of the plurality of pixels included in the BW image, the reliability mask calculation unit 112 normalizes, for example, the texture amount corresponding to the pixel calculated in step S153. Thus, the reliability mask calculation unit 112 calculates a BW reliability mask that indicates the reliability of the BW depth corresponding to the pixel calculated in step S130 in multiple values.

  In the above-described example, the reliability mask calculation unit 112 calculates the BW reliability mask based on the BW image. However, as described above, the reliability mask calculation unit 112 subtracts the IR reliability mask from 1 without using the BW image. A BW reliability mask may be calculated.

(Modification)
In the above embodiment, a reliability mask such as a BW reliability mask and an IR reliability mask is used for calculating the integrated depth, but the integrated depth is calculated using a learning model without using the reliability mask. You may.

  FIG. 13 is a block diagram illustrating an example of a functional configuration of the depth acquisition device 1 according to the present modification. Note that among the components illustrated in FIG. 13, the same components as those illustrated in FIG. 4 are denoted by the same reference numerals as the components illustrated in FIG. 4, and detailed description will be omitted.

  The depth acquisition device 1 according to the present modification includes a processor 100b instead of the processor 110a illustrated in FIG. 4, and further includes a learning model 104.

  The learning model 104 is, for example, a neural network, and is configured by deep learning. For example, input data input to the learning model 104 include a BW image, a BW depth calculated from the BW image, the intensity of infrared light at each of the three exposure timings, and the intensity of the infrared light. Any data of at least two of an IR image based on the intensity and an IR depth calculated from the intensity of the infrared light may be used. In the learning model 104, learning has been already performed so that a correct integrated depth is output for a combination of at least two pieces of input data that are arbitrary data. Note that the processor 110b does not include the reliability mask calculation unit 112 as described later. However, when the processor 110b includes the reliability mask calculation unit 112, at least one of the BW reliability mask and the IR reliability mask output from the reliability mask calculation unit 112 is used as input data. You may. Note that only the BW reliability mask and the IR reliability mask are not used as input data.

  Image interpolation technology using deep learning is described in Non-Patent Documents (Satoshi Iizuka, Edgar Simo-Serra and Hiroshi Ishikawa “Globally and Locally Consistent Image Completion”, ACM Transactions on Graphics, Vol. 36, No. 4, Article 107). Publication date: July 2017.) However, in the learning model used in this non-patent document, an RGB image and a binary mask indicating a missing interpolation position are input as input data, and the RGB image subjected to the missing interpolation is output data based on the input data. Is output from the learning model.

  The processor 100b does not include the reliability mask calculation unit 112 illustrated in FIG. 4, but includes an integrated depth calculation unit 111d instead of the integrated depth calculation unit 111c illustrated in FIG.

  The integrated depth calculation unit 111d inputs the above-described input data to the above-described learning model 104. As a result, the integrated depth calculation unit 111d acquires the integrated depth from the learning model 104 as output data for the input data.

  That is, the depth acquisition device 1 illustrated in FIG. 13 includes the memory 200 and the processor 110b. The BW image acquisition unit 115 of the processor 110b acquires a BW image generated by imaging the subject based on visible light by the BW camera 103 and stored in the memory 200. The IR intensity acquisition unit 114 acquires the intensity of the infrared light measured by receiving the infrared light emitted from the light source 101 and reflected by the subject, and stored in the memory 200. The BW depth calculation unit 111a calculates a BW depth, which is a distance to a subject, based on the BW image. The IR depth calculator 111b calculates an IR depth, which is a distance to a subject, based on the intensity of the infrared light. By inputting the BW image, the intensity of infrared light, the BW depth, and the IR depth to the learning model 104, the integrated depth calculation unit 111d calculates the integrated depth obtained by integrating the BW depth and the IR depth from the learning model 104. get.

  Accordingly, if the learning model 104 is trained in advance so that a correct integrated depth is output with respect to the input of the BW image, the intensity of the infrared light, the BW depth, and the IR depth, the BW reliability mask and the An appropriate integrated depth can be easily obtained without calculating an IR reliability mask.

  As described above, in the depth acquisition device 1 according to the present embodiment and its modification, since the reflectance of the subject with the infrared light is low, the reliability of the second depth calculated based on the intensity of the infrared light is low. However, it is possible to correct the second depth using the first depth and calculate the integrated depth. As a result, the reliability or accuracy of the depth to the subject finally calculated can be improved.

  In each of the above embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software for realizing the depth acquisition device and the like of the above-described embodiment and the modified example causes a computer to execute each step included in any one of the flowcharts of FIGS. 11 and 12.

  As described above, the depth acquisition device according to one or more aspects has been described based on the embodiment and the modification thereof, but the present disclosure is not limited to the embodiment and the modification thereof. Unless departing from the gist of the present disclosure, various modifications conceivable by those skilled in the art may be applied to the present embodiment and the modification, and a form constructed by combining the components in the embodiment and the modification may be implemented by the present disclosure. It may be included in the range.

  For example, in the above-described embodiment and its modified example, the IR intensity acquisition unit 114 acquires an IR image, but without acquiring an IR image, the first pixels 21 included in the solid-state imaging device 20 emit infrared light. May be acquired to obtain the intensity of the infrared light measured. That is, the depth acquisition device 1 does not need to generate an IR image.

  Further, the IR intensity acquisition unit 114 acquires the three infrared light intensities measured at three different exposure timings, and determines the IR depth and the IR reliability mask based on the three infrared light intensities. calculate. However, the number of the exposure timing and the intensity of the infrared light is not limited to three, and may be four or more.

  Further, in the present disclosure, all or a part of a unit or a device, or all or a part of a functional block in the block diagrams illustrated in FIGS. 1, 4, and 13 includes a semiconductor device, a semiconductor integrated circuit (IC), Alternatively, it may be executed by one or more electronic circuits including an LSI (large scale integration). The LSI or IC may be integrated on a single chip, or may be configured by combining a plurality of chips. For example, functional blocks other than the storage element may be integrated on one chip. Here, although they are called LSIs or ICs, their names change depending on the degree of integration, and may be called system LSIs, VLSIs (very large scale integration), or ULSIs (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the manufacture of the LSI, or a reconfigurable logic device capable of reconfiguring junction relations inside the LSI or setting up circuit sections inside the LSI can also be used for the same purpose.

  Furthermore, all or some of the functions or operations of the unit, the device, or a part of the device can be performed by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROM, optical disk, hard disk drive, etc., and when the software is executed by a processor, the software is Causes a processor and peripheral devices to perform certain functions. The system or apparatus may include one or more non-transitory storage media on which the software is recorded, a processor, and any required hardware devices, such as an interface.

  The present disclosure is applicable to a depth acquisition device that acquires a depth that is a distance to a subject, and can be used as, for example, an in-vehicle device.

DESCRIPTION OF SYMBOLS 1 Depth acquisition device 10, 101 Light source 20 Solid-state image sensor 21 First pixel (IR)
22 Second pixel (BW)
Reference Signs List 30 processing circuit 50 diffuser 60 lens 70 bandpass filter 102 IR camera 103 BW camera 104 learning model 110a, 110b processor 111a BW depth calculation unit 111b IR depth calculation unit 111c integrated depth calculation unit 112 reliability mask calculation unit 113 emission control unit 114 IR intensity acquisition unit 115 BW image acquisition unit 200 Memory

Claims (15)

Memory and
With a processor,
The processor comprises:
Obtain a visible light image generated by imaging a subject based on visible light and stored in the memory,
It is measured by receiving the infrared light emitted from the light source and reflected by the subject, and acquires the intensity of the infrared light stored in the memory,
Calculating a first depth, which is a distance to the subject, based on the visible light image;
Based on the intensity of the infrared light, calculate a second depth that is a distance to the subject,
Calculating a first reliability mask representing the reliability of the first depth in multiple values;
Calculating a second reliability mask representing the reliability of the second depth in multiple values;
Calculating an integrated depth by integrating the first depth and the second depth based on the first reliability mask and the second reliability mask;
Depth acquisition device. The reception of the infrared light is performed by imaging based on the infrared light,
In the imaging based on the visible light and the imaging based on the infrared light, the scene to be imaged and the viewpoint and time of the imaging are substantially the same,
The depth acquisition device according to claim 1. The intensity of the infrared light obtained from the memory,
When infrared light emitted from the light source and reflected by the subject is received by the image sensor, the image sensor includes at least three intensities measured by exposure of the image sensor at at least three different times,
In the calculation of the second depth,
Calculating the second depth based on the at least three intensities;
The depth acquisition device according to claim 1. The first reliability mask and the second reliability mask each have a coefficient of 0 or more and 1 or less,
In the calculation of the integrated depth,
The weighted addition of the first depth and the second depth using the first reliability mask and the second reliability mask as a weighting factor for the first depth and a weighting factor for the second depth, respectively, Calculate the integrated depth,
The depth acquisition device according to claim 1. In the calculation of the first reliability mask,
Calculating the first reliability mask by subtracting the second reliability mask from 1;
The depth acquisition device according to claim 4. In the calculation of the integrated depth,
If the second reliability mask is greater than or equal to a threshold,
Calculating the second depth as the integrated depth,
The depth acquisition device according to claim 4. In the calculation of the integrated depth,
If the second depth cannot be calculated,
Calculating the first depth as the integrated depth,
The depth acquisition device according to claim 1. In the calculation of the second reliability mask,
Calculating the second reliability mask based on a ratio between a maximum intensity and a minimum intensity of the at least three intensities;
The depth acquisition device according to claim 3. Memory and
With a processor,
The processor comprises:
Obtain a visible light image generated by imaging a subject based on visible light and stored in the memory,
It is measured by receiving the infrared light emitted from the light source and reflected by the subject, and acquires the intensity of the infrared light stored in the memory,
Calculating a first depth, which is a distance to the subject, based on the visible light image;
Based on the intensity of the infrared light, calculate a second depth that is a distance to the subject,
By inputting the visible light image, the intensity of the infrared light, the first depth and the second depth into a learning model, the integrated depth obtained by integrating the first depth and the second depth, Obtained from the learning model,
Depth acquisition device. A visible light camera that acquires a visible light image by performing imaging of a subject based on visible light,
An infrared light camera that obtains the intensity of the infrared light by receiving infrared light emitted from a light source and reflected by the subject,
A first depth calculation unit that calculates a first depth that is a distance to the subject based on the visible light image;
A second depth calculation unit that calculates a second depth that is a distance to the subject based on the intensity of the infrared light;
A reliability mask calculation unit that calculates a first reliability mask that represents the reliability of the first depth in multiple values, and calculates a second reliability mask that represents the reliability of the second depth in multiple values;
An integrated depth calculation unit that calculates an integrated depth by integrating the first depth and the second depth based on the first reliability mask and the second reliability mask;
A depth acquisition device comprising: A visible light camera that acquires a visible light image by performing imaging of a subject based on visible light,
An infrared light camera that obtains the intensity of the infrared light by receiving infrared light emitted from a light source and reflected by the subject,
A first depth calculation unit that calculates a first depth that is a distance to the subject based on the visible light image;
A second depth calculation unit that calculates a second depth that is a distance to the subject based on the intensity of the infrared light;
By inputting the visible light image, the intensity of the infrared light, the first depth and the second depth into a learning model, the integrated depth obtained by integrating the first depth and the second depth, An integrated depth calculation unit obtained from the learning model;
A depth acquisition device comprising: A depth acquisition method in which a computer acquires a distance to a subject as a depth,
Obtain a visible light image generated by imaging the subject based on the visible light and stored in the memory,
It is measured by receiving the infrared light emitted from the light source and reflected by the subject, and acquires the intensity of the infrared light stored in the memory,
Calculating a first depth, which is a distance to the subject, based on the visible light image;
Based on the intensity of the infrared light, calculate a second depth that is a distance to the subject,
Calculating a first reliability mask representing the reliability of the first depth in multiple values;
Calculating a second reliability mask representing the reliability of the second depth in multiple values;
Calculating an integrated depth by integrating the first depth and the second depth based on the first reliability mask and the second reliability mask;
How to get depth. A depth acquisition method in which a computer acquires a distance to a subject as a depth,
Obtain a visible light image generated by imaging the subject based on the visible light and stored in the memory,
It is measured by receiving the infrared light emitted from the light source and reflected by the subject, and acquires the intensity of the infrared light stored in the memory,
Calculating a first depth, which is a distance to the subject, based on the visible light image;
Based on the intensity of the infrared light, calculate a second depth that is a distance to the subject,
By inputting the visible light image, the intensity of the infrared light, the first depth and the second depth into a learning model, the integrated depth obtained by integrating the first depth and the second depth, Obtained from the learning model,
How to get depth. A program for acquiring a distance to a subject as a depth,
Obtain a visible light image generated by imaging the subject based on the visible light and stored in the memory,
It is measured by receiving the infrared light emitted from the light source and reflected by the subject, and acquires the intensity of the infrared light stored in the memory,
Calculating a first depth, which is a distance to the subject, based on the visible light image;
Based on the intensity of the infrared light, calculate a second depth that is a distance to the subject,
Calculating a first reliability mask representing the reliability of the first depth in multiple values;
Calculating a second reliability mask representing the reliability of the second depth in multiple values;
Calculating an integrated depth by integrating the first depth and the second depth based on the first reliability mask and the second reliability mask;
A program that causes a computer to do something. A program for acquiring a distance to a subject as a depth,
Obtain a visible light image generated by imaging the subject based on the visible light and stored in the memory,
It is measured by receiving the infrared light emitted from the light source and reflected by the subject, and acquires the intensity of the infrared light stored in the memory,
Calculating a first depth, which is a distance to the subject, based on the visible light image;
Based on the intensity of the infrared light, calculate a second depth that is a distance to the subject,
By inputting the visible light image, the intensity of the infrared light, the first depth and the second depth into a learning model, the integrated depth obtained by integrating the first depth and the second depth, Obtained from the learning model,
A program that causes a computer to do something.

Images