JP2015121430A - Distance image creation device and distance image creation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately calculate, in a light flying time type distance image sensor, a distance value of a structure with inclination or a moving object with inclination with a compact and inexpensive device without causing a reduction in processing speed and frame rate.SOLUTION: A distance image creation device estimates, for a pixel of which distance value cannot be calculated due to the presence of inclination of a reflection surface of an object, a distance value of the pixel by using a distance value and a brightness value of a neighboring pixel of which distance value has been successfully calculated. The distance image creation device specifies the inclination direction of the reflection surface of the object, calculates the inclination of the reflection surface, and performs the estimation by using the inclination.

Description

  The present invention relates to a distance image generation technique using an optical time-of-flight distance image sensor, and more particularly to a technique for correcting the influence of a flare phenomenon.

  There is a distance image generation device that generates a distance image having a pixel value of a distance from an object in an imaging target space using the optical flight distance image sensor. Among them, a time-of-flight type (TOF method: Time Of Flight method) distance image sensor is composed of a light source that emits modulated light and a light receiving element that receives the modulated light. The modulated light is emitted and received for each pixel. A distance value is calculated based on the reflected light (see, for example, Patent Document 1). There is also a sorting-type TOF method in which charges generated by reflected light of received modulated light are synchronized with the modulation frequency of the modulated light and the charges are distributed (see, for example, Patent Document 2).

  In order to calculate the distance value of an object, it is necessary to receive reflected light from the object. However, when the object has an extreme inclination, the reflected light of the modulated light does not return and the distance value may not be calculated for most of the inclined part. This problem can occur in all active range image sensors. In particular, this occurs more markedly in the case of an object that has no irregular reflection or scattering due to surface roughness such as a mirror surface.

  In order to solve this, there is a method of detecting an object position from a plurality of laser sensors at different positions from background data acquired in advance in a laser sensor (see, for example, Patent Document 3). In addition, there is a method in which distance information / gradient information of a detected object is stored in advance, gradient information is calculated from the measured distance information, alignment is performed, and an object is detected (see, for example, Patent Document 4). Furthermore, in order to improve the detection accuracy in the low reflection state when the object is tilted, there is a method for improving the distance accuracy by improving the light receiving unit so that the influence of noise when receiving low intensity reflected light is reduced. Yes (see, for example, Patent Document 5).

JP 2005-235893 A JP 2011-179925 A JP 2009-85927 A JP 2006-145352 A JP 2006-084429 A

  The technique described in Patent Document 3 requires background data in advance. Therefore, it cannot be used in situations where the background changes frequently, such as when installed in a vehicle. Furthermore, since a plurality of sensors are used, the apparatus becomes large and the price is high. Further, since a plurality of sensors are used, the processing becomes enormous and the frame rate is lowered.

  In the method described in Patent Document 4, it is essential to have original image information in advance as a template. Therefore, only a specific object can be detected. Moreover, since it is necessary to hold data in advance, processing information increases.

  In the method described in Patent Document 5, the light receiving unit needs to be improved, and the price of the sensor is increased. In addition, since the structure is complicated, it is difficult to manufacture. Further, when the reflected light itself cannot be received, the distance value cannot be calculated.

  The present invention has been made in view of the above circumstances, and in a time-of-flight distance image sensor, even if it is a structure having an inclination or an animal body having an inclination, the processing speed and the frame rate are reduced. The objective is to calculate the distance value with a small and inexpensive device with high accuracy.

  The present invention estimates a distance value of a pixel using a distance value and a luminance value of a pixel for which a distance value of a neighborhood can be calculated for a pixel for which a distance value cannot be calculated due to the reflection surface of the object having an inclination. . The estimation is performed by specifying the inclination direction of the reflection surface of the object, calculating the inclination of the reflection surface, and using the inclination.

  Specifically, a light source unit that irradiates modulated light to the target space, and incident light including reflected light that is irradiated from the light source unit and reflected by the surface of the target object in the target space is received and charged. The photoelectric conversion element provided for each pixel to be converted, a plurality of charge storage units provided for each photoelectric conversion element, and the charge converted by the photoelectric conversion element in synchronization with the modulation of the modulated light A distance image generation method in a distance image generation device including a charge distribution unit that distributes to a plurality of charge storage units, wherein a luminance image whose pixel value is a luminance value using charges stored in the charge storage unit An image generation step of generating a distance image having a pixel value as a distance value using the charges accumulated in the plurality of charge accumulation units, and the distance value is determined by the inclination of the surface of the object. Unacceptable distance A distance value calculating step of calculating a distance value of a pixel whose value has not been calculated by using the luminance value and the distance value of a pixel in the vicinity of the pixel whose distance value has not been calculated. provide.

In addition, a light source unit that irradiates modulated light that is modulated into the target space, and incident light including reflected light that is irradiated from the light source unit and reflected by the surface of the target object in the target space is received and converted into charges. A photoelectric conversion element provided for each pixel; a plurality of charge storage units provided for each photoelectric conversion element;
In synchronism with the modulation of the modulated light, the charge distribution unit that distributes the charges converted by the photoelectric conversion element to the plurality of charge storage units, and the pixel value using the charges stored in the charge storage units A luminance image generation unit that generates a luminance image that is a value; and a distance image generation unit that generates a distance image whose pixel value is a distance value using charges accumulated in the plurality of charge accumulation units, The distance image generation unit obtains the distance value of a distance value uncalculated pixel for which the distance value cannot be obtained due to the inclination of the surface of the object, the luminance value of the pixel in the vicinity of the distance value uncalculated pixel, and the Provided is a distance image generation device including a distance value calculation unit that calculates using a distance value.

  According to the present invention, in a time-of-flight distance image sensor, even a structure having an inclination or an animal body having an inclination is a small and inexpensive device without causing a reduction in processing speed and frame rate. The distance value can be calculated with high accuracy.

It is a block diagram of the distance image generation device of the embodiment of the present invention. It is explanatory drawing for demonstrating the relationship between the modulation signal and gate signal of embodiment of this invention. It is explanatory drawing for demonstrating the principle of the distance image generation by the optical time-of-flight distance image generation apparatus. (A) And (b) is explanatory drawing for demonstrating that a distance value is not obtained in the optical time-of-flight distance image generation device, when the reflective surface of a target object has inclination. It is explanatory drawing for demonstrating that a distance value cannot be obtained in the optical time-of-flight distance image generation device, when the reflective surface of a target object has inclination. It is explanatory drawing for demonstrating that a distance value cannot be obtained in the optical time-of-flight distance image generation device, when the reflective surface of a target object has inclination. (A) is explanatory drawing for demonstrating arrangement | positioning with a time-of-flight type distance image generation apparatus and a target object, (b) and (c) are each when imaged by the arrangement | positioning shown to (a). It is explanatory drawing for demonstrating the distance image and brightness | luminance image which are obtained. It is explanatory drawing for demonstrating the pixel position of the distance image and luminance image which are obtained with the distance image generation device of this embodiment. It is a flowchart of the estimation distance value calculation process with respect to one pixel of this embodiment. It is a flowchart of the inclination direction discrimination | determination process of this embodiment. (A) is an explanatory view for explaining a storage area of the memory X of the present embodiment, and (b) is an explanatory diagram for explaining a storage area of the memory Y of the present embodiment. It is a flowchart of -X direction estimation distance value calculation processing of this embodiment. It is a flowchart of -Y direction estimated distance value calculation processing of this embodiment. It is a flowchart of + X direction estimated distance value calculation processing of the present embodiment. It is a flowchart of + Y direction estimated distance value calculation processing of the present embodiment. It is explanatory drawing for demonstrating the specific example 1 of the estimated distance value calculation process of this embodiment. It is explanatory drawing for demonstrating the specific example 2 of the estimated distance value calculation process of this embodiment. It is explanatory drawing for demonstrating the specific example 3 of the estimated distance value calculation process of this embodiment. It is explanatory drawing for demonstrating the specific example 4 of the estimated distance value calculation process of this embodiment. It is a flowchart of the distance image generation process of this embodiment. It is explanatory drawing for demonstrating the relationship between the modulation signal and gate signal of the modification of this invention of this embodiment. It is explanatory drawing for demonstrating the relationship between the modulation signal and gate signal of the modification of this invention of this embodiment.

  Hereinafter, embodiments to which the present invention is applied will be described. Hereinafter, in all the drawings for explaining the embodiments of the present invention, those having the same function are denoted by the same reference numerals, and repeated explanation thereof is omitted.

<Distance image generator configuration>
First, the configuration of the distance image generation device of this embodiment will be described. FIG. 1 is a functional block diagram of the distance image generation apparatus of the present embodiment.

  As shown in the figure, the distance image generating apparatus 100 of the present embodiment generates a distance image using a phase difference between irradiated modulated light 111 and incident light 112, and includes a light source unit 110, photoelectric conversion, and the like. Unit 120, charge distribution unit 130, control unit 140, charge storage unit 150, luminance image generation unit 160, and distance image generation unit 170 including an estimated distance value calculation unit 171.

  Note that the distance image generation device 100 of this embodiment includes a CPU, a memory, and a storage device. The charge distribution unit 130, the control unit 140, the luminance image generation unit 160, and the distance image generation unit 170 are realized by the CPU loading a program stored in the storage device in advance into the memory and executing it. .

  The light source unit 110 is a light source that irradiates the target space with modulated light (for example, infrared light or visible light modulated at high speed with a sine wave or a rectangular wave) under the control of the control unit 140 described later. As the light source 110, a device capable of high-speed modulation such as an LED is used.

  The photoelectric conversion unit 120 receives incident light 112 including reflected light, which is reflected from the object 113 in the target space (the field of view of the distance image generation device 100), of the modulated light 111 emitted from the light source unit 110, and converts it into electric charges. Convert. The photoelectric conversion unit 120 includes a plurality of photoelectric conversion elements that convert charges according to the amount of received light. This photoelectric conversion element forms a pixel. Therefore, each photoelectric conversion element may be provided with a lens in front of the photoelectric conversion unit 120 that is regularly arranged in association with each pixel of the luminance image and the distance image.

  The charge distribution unit 130 distributes the charges converted by the photoelectric conversion unit 120 to the charge storage unit 150 (to be described later) under the control of the control unit 140 (to be described later). The sorting is performed for each photoelectric conversion element.

  The charge storage unit 150 stores the charge distributed by the charge distribution unit 130. The phase difference between the modulated light 111 and the incident light 112, which is distance information, is calculated using the charges accumulated in the charge accumulation unit 150. In order to calculate this phase difference, at least three or more pieces of phase information (charges accumulated in the charge accumulation unit) are necessary.

  The acquisition of three or more pieces of phase information may be realized by one charge storage unit 150 or may be realized by a plurality of charge storage units 150. Hereinafter, in this embodiment, a specific example will be described assuming that there are four charge storage units 150. When it is necessary to distinguish these four charge storage units 150, they are denoted as 151, 152, 153, and 154, respectively. This set of four charge storage units 150 is provided for each pixel.

  One or more charge storage units 150 that acquire phase information are provided for each pixel (photoelectric conversion element). The charge storage unit 150 may store the charge itself, or may store data after AD conversion of this charge amount. In the present embodiment, what is stored in the charge storage unit 150 without distinction between the charge itself and the data after AD conversion is referred to as a charge.

  The control unit 140 controls the light source unit 110 and the charge distribution unit 130 in synchronization. Control is performed by the modulation signal and the gate signal.

  The modulation signal is a signal that instructs modulation and output timing of the modulated light, and is output toward the light source unit 110. The light source unit 110 generates modulated light 111 modulated by the modulation signal and irradiates the target space. The irradiated modulated light 111 is reflected by the object 113 and becomes reflected light. The reflected light enters the photoelectric conversion unit 120 as incident light 112, is converted into charges, and is sent to the charge distribution unit 130.

  The gate signal is a signal for instructing the charge distribution to each charge storage unit 150 and is output toward the charge distribution unit 130. The charge distribution unit 130 distributes the charges to the charge storage units 150 (151, 152, 153, 154) according to the gate signal.

  The control unit 140 generates a gate signal that is synchronized with the modulation period of the light source unit 110 and is shifted by a time corresponding to the modulation period and the number of the charge distribution units 130. For example, if the modulation period is 10 MHz and there are four charge storage units, the gate signal is generated at a timing that is shifted by a time (25 ns) obtained by dividing the modulation period into four. As a result, the charge is distributed to each charge storage unit 150 in accordance with the gate signal.

  An example of the output timing of the modulation signal and the gate signal of this embodiment is shown in FIG. Here, the modulation signal 114 and the gate signal 115 are shown in the case where charges are sequentially distributed to the four charge storage units 150 in a period obtained by dividing one period Tq of the modulation signal by four.

  In addition, the control unit 140 transmits a read signal to the luminance image generation unit 160 and the distance image generation unit 170 to read out the charges accumulated in the charge accumulation unit 150, respectively. Note that the modulation frequency of the modulated light 111 is several tens of MHz. Therefore, one modulation period is about several tens of ns. In order to obtain a distance image, a charge accumulation time of several hundred to several hundred thousand cycles is required. The control unit 140 according to the present embodiment outputs a read signal so that the luminance image generation unit 160 and the distance image generation unit 170 read charges from the charge storage unit 150 in units of the charge storage time.

  The modulation signal, the gate signal, and the readout signal are collectively referred to as a control signal.

  The luminance image generation unit 160 calculates the luminance value of each pixel from the charges stored in each charge storage unit 150, and generates a luminance image using the calculated luminance value as a pixel value. Details of the calculation will be described later.

  The distance image generation unit 170 calculates a distance value of each pixel from the charges stored in each charge storage unit 150, and generates a distance image using the calculated distance value as a pixel value. The distance value is a value of the distance to the object in the visual field. Details of the calculation will be described later.

<Principle of TOF method of distance image generation>
Here, the principle of distance image generation in the time-of-flight (TOF method) distance image generation apparatus 100 will be described. FIG. 3 is a diagram for explaining the principle of distance image generation.

  When the intensity of the modulated light 111 emitted from the light source unit 110 changes so as to draw a sine curve as shown in the figure, the intensity of the incident light 112 to the photoelectric conversion unit 120 also changes so as to draw a sine curve. At this time, the modulated light 111 and the incident light 112 have a phase delay (phase difference φ) due to the time of flight of the light traveling back and forth to the object.

Since the speed c of light is known, the distance value D to the object can be obtained by the following formula (1) using the phase difference φ and the modulation frequency f.

  Therefore, if the phase difference φ is known, the distance value D can be obtained. In the present embodiment, one period Tp of the modulated light is divided into four equal parts, and the charge amount is accumulated in the four charge accumulation units 150 in each period.

The phase difference φ between the modulated light 111 and the incident light 112 is defined as Tq (T1, T2, T3, T4) where each period obtained by dividing one period of the modulated light 111 into four equal parts, and the amount of charge accumulated in each period. Assuming C1, C2, C3, and C4, they are represented by the following formula (2).
Each period Tq (T1, T2, T3, T4) obtained by dividing one period Tp into four is, for example, between 0 degrees and 90 degrees, between 90 degrees and 180 degrees, between 180 degrees and 270 degrees, Between 270 degrees and 0 degrees.

  The distance image generation unit 170 uses the charge amounts C1, C2, C3, and C4 accumulated at the above-described charge accumulation time interval to obtain the phase difference φ for each pixel according to the equation (2). Then, using the obtained phase difference φ, a distance value D to the object 113 is obtained according to the equation (1), and a distance image having the distance value as a pixel value is generated.

In addition, the luminance image generation unit 160 according to the present embodiment calculates a luminance value for each pixel using the charges (C1, C2, C3, and C4) accumulated in each charge accumulation unit 150. The method for calculating the luminance value may be a method for converting the charge amount into a relative value, and is not particularly defined. For example, the luminance value B is obtained according to the following equation (3), and a luminance image having the luminance value as a pixel value is generated.
B = C1 + C2 + C3 + C4 (3)

  A single distance image and luminance image obtained at every charge accumulation time Tf interval are referred to as a frame. The charge accumulation time Tf is referred to as 1 frame time Tf.

<Reason why distance value cannot be calculated due to inclined surface>
In the TOF range image generation apparatus 100, the modulated light 111 emitted from the light source unit 110 is reflected by the object 113 and received by the photoelectric conversion unit 120 as described above. Then, the distance image generation unit 170 calculates a distance value from the phase difference of the reflected light (incident light 112). This is shown in FIG. For this reason, if the reflected light cannot be received, the distance value cannot be calculated in principle.

  For example, when the reflection surface of the object 113 in FIG. 4B has an extreme inclination, only reflected light from a part of the inclination may be received. In this case, the distance value cannot actually be calculated despite the presence of the object 113.

  Specifically, it is as follows.

  Assume that the number of pixels of the distance image generating apparatus 100 of the present embodiment is N × N (N is an integer of 1 or more), and the angle of view is θ. The imaging range in the horizontal and vertical directions per pixel is a range of θ / N. Therefore, the imaging range of the nth pixel (n is an integer of 1 to N) from the central pixel is from (n−1) × θ / N to n × θ / N. The vertical direction on the light receiving surface of the distance image generating device 100 is set to 0 degree.

  As shown in FIG. 5, the light emitted from the light source unit 110 of the distance image generation device 100 is reflected by the front object 113 and is incident on the nth pixel (a photoelectric conversion element arranged correspondingly). In order to achieve this, the inclination of the surface serving as the reflecting surface of the object 113 needs to be less than (90−n × θ / 2) / 2 degrees. The object 113 is a mirror flat plate. For this reason, when the reflecting surface of the object 113 has an inclination of (90−n × θ / 2) / 2 degrees or more, the nth pixel cannot receive the reflected light and the distance value cannot be calculated.

  In addition, as illustrated in FIG. 6, when the inclination of the reflection surface of the object 113 is θa with respect to the light receiving surface of the distance image generation device 100, and the distance between the distance image generation device 100 and the object 113 is D. The light source unit 110 needs to be installed at a position away from the photoelectric conversion unit 120 sensor, which is a light receiving unit, by D · cos (n × θ / N) · {1 + tan (n × θ / N + 2θa)}.

  Therefore, even if the inclination is less than (90−n × θ / 2) / 2 degrees, the light source unit 110 is placed at a sufficient distance from the photoelectric conversion unit 120 as the light receiving unit in order to obtain reflected light. There is a need to. For this reason, it is necessary to enlarge the size of the device itself. If these distances cannot be secured between the light source unit 110 and the photoelectric conversion unit 120, light cannot be received and the distance value cannot be calculated.

  On the other hand, even if the reflected light of the modulated light 111 emitted from the light source unit 110 cannot be received, the object 113 actually exists, and thus the reflected light from the surrounding environmental light is received by the photoelectric conversion unit 120. And it can be output as luminance.

  7 (a) to 7 (c), the range image generation apparatus 100 includes a range in which an object 113a having an inclined reflection surface and an object (wall) 113b having no inclination are included in the visual field. A distance image 210 and a luminance image 220 obtained by photographing will be described. FIG. 7A is a diagram for explaining the arrangement of the distance image generation device 100 and the objects 113a and 113b. FIG. 7B is a distance image of the imaging space, and FIG. , The same luminance image.

  The distance image 210 is an image in which a distance value between a part of the inclination of the reflection surface of the object 113a and the rear wall 113b is a pixel value. The luminance image 220 is an image having a pixel value as a luminance value by the entire inclination of the reflection surface of the object 113a and the rear wall 113b.

  As described above, in most of the pixels corresponding to the inclination of the reflection surface of the object 113a, the distance value is 0 although the luminance value exists. By comparing the corresponding pixel values of the distance image 210 and the luminance image 220, it can be seen that the distance value is not calculated by the inclination, and the pixel corresponding to the inclined portion can be determined. Furthermore, a distance value can be calculated for a part of the slope. Therefore, the inclination of the inclination can be calculated using the distance value between adjacent pixels or the distance value between pixels at a predetermined interval. Then, an uncalculated distance value of the inclined portion can be calculated using the calculated inclination of the inclination.

  In the present embodiment, this is used to estimate the distance value that could not be calculated due to the inclination by using the luminance value of each pixel and the distance value of the pixel for which the distance value could be calculated. That is, as shown in FIG. 1, the distance image generation unit 170 according to the present embodiment calculates the distance value of the distance value uncalculated pixels whose distance value cannot be obtained due to the inclination of the reflection surface of the object 113 as the estimated distance. An estimated distance value calculation unit (distance value calculation unit) 171 that is calculated as a value is provided. For the calculation of the estimated distance value, the luminance value and the distance value of the pixels in the vicinity of the pixel whose distance value has not been calculated are used.

  The estimated distance value calculation unit 171 uses the luminance image generated by the luminance image generation unit 160 and the distance image generated by the distance image generation unit 170 to calculate an estimated distance value of a pixel whose distance value has not been calculated, and the pixel of the distance image. Replace with the pixel value of. Hereinafter, the estimated distance value calculation process by the estimated distance value calculation unit 171 will be described.

  Prior to the description of the estimated distance value calculation process, FIG. 8 shows a pixel position 300 of the distance image and the luminance image obtained by the distance image generation device 100 of the present embodiment. As shown in the figure, the distance image and the luminance image of the present embodiment are N × M pixels of N pixels in the X direction and M pixels in the Y direction (N and M are integers of 1 or more). The X direction and the Y direction are orthogonal to each other. In the drawing, the right side is the + direction and the lower side is the + direction. The pixel position ranges from 1 to N and 1 to M, and the upper left corner of the image is the pixel position (1, 1).

  Prior to the estimated distance value calculation process, the estimated distance value calculation unit 171 reads out the distance value and the luminance value generated by the distance image generation unit 170 and the luminance image generation unit 160, and uses the pixel position as an address, respectively, in an internal memory area. The data is stored in D (memory D) and memory area B (memory B). The memory D is secured in advance as a distance value storage area, and the memory B is secured in advance as a brightness value storage area.

  The estimated distance value calculation unit 171 of the present embodiment determines the presence / absence of inclination for each pixel in a predetermined order. If it is determined that there is an inclination, the estimated distance value calculation unit 171 determines the direction of the inclination. An estimated distance value calculation process for calculating is performed. In this embodiment, the process is repeated from n = 1 to N in the X direction and from m = 1 to M in the Y direction. Hereinafter, the flow of the estimated distance value calculation process for one processing target pixel (n, m) will be described.

<Estimated distance value calculation process>
FIG. 9 shows a processing flow of an estimated distance value calculation process for the processing target pixel (n, m) by the estimated distance value calculation unit 171 of the present embodiment.

  First, in step S1001 and step S1002, the estimated distance value calculation unit 171 determines whether or not the processing target pixel (n, m) is a distance value non-calculated pixel. That is, it is determined whether or not the reflection surface of the object 113 within the field of view of the processing target pixel (n, m) having a distance value of 0 is inclined. Here, if the luminance value is equal to or greater than a predetermined luminance threshold Bth, it is determined that there is an inclination. This is because even when the distance value is 0 and the luminance value is small, the distance 113 cannot be calculated because there is no object 113 in the area corresponding to the processing target pixel (n, m) in the field of view. This is because it is judged.

  Specifically, first, it is determined whether or not the distance value of the processing target pixel (n, m) is 0 (step S1001). If the distance value is 0, the luminance value of the processing target pixel (n, m) is compared with a predetermined luminance threshold Bth (step S1002). If the luminance value is equal to or higher than the luminance threshold Bth, It is discriminated that there is an inclination, that is, the processing target pixel (n, m) is a pixel whose distance value has not been calculated.

  Note that the luminance threshold value Bth used here varies greatly depending on the surrounding environment, and therefore cannot be determined unconditionally, but is set in an actual use environment.

  Hereinafter, in order to simplify the description, the reflection surface of the object 113 in the field of view of the pixel (n, m) is simply referred to as the surface of the pixel (n, m).

  If it is determined in step S1002 that the distance value has not been calculated, that is, the pixel has an inclination, it is determined whether the inclination is in the X direction or the Y direction (steps S1101 and S1201). Details of the inclination direction determination processing will be described later.

  If it is determined that the direction is the X direction, -X direction estimated distance value calculation processing is performed (step S1102), the estimated distance value Dc of the processing target pixel (n, m) is calculated, and the processing ends. In the -X direction estimated distance value calculation processing, the processing target pixel ((X, Y, M)) is processed using the distance value of the pixel whose distance value or estimated distance value Dc has already been calculated in the -X direction. n, m) of the estimated distance value Dc is calculated. Details of the processing will be described later.

  On the other hand, if it is determined that the direction is the Y direction, a -Y direction estimated distance value calculation process is performed (step S1202), an estimated distance value Dc of the processing target pixel (n, m) is calculated, and the process ends. -Y direction estimated distance value calculation processing is also performed using a distance value of a pixel whose distance value or estimated distance value Dc has already been calculated in the -Y direction with respect to the processing target pixel (n, m). Details of the process of calculating the estimated distance value Dc of (n, m) will be described later.

  On the other hand, when the inclination cannot be discriminated from any direction, a process when the direction cannot be discriminated is performed (step S1301), and the process ends.

  If the distance value of the processing target pixel (n, m) is not 0 in step S1001, it is not necessary to calculate the estimated distance value Dc for the processing target pixel (n, m). However, when there is a distance value uncalculated pixel in a pixel adjacent to the −X direction or the −Y direction, the distance value of the uncalculated pixel is calculated using the processing target pixel (n, m) or the like.

  Therefore, here, first, it is determined whether or not a pixel adjacent in the −X direction or the −Y direction is a distance value uncalculated pixel (step S1401). In the determination, the estimated distance value calculation unit 171 uses the memory area X (hereinafter simply referred to as “memory X”) and the memory area Y (hereinafter simply referred to as “memory Y”) used as work memory when calculating the estimated distance value. )). Whether or not the pixel (n−1, m) adjacent in the −X direction has not been calculated is whether or not the pixel (n, m−1) adjacent in the −Y direction has not been calculated using the memory X. The determination is made using the memory Y. Details of the configuration of each memory area and details of determination will be described later.

  When the pixel (n−1, m) adjacent in the −X direction is a distance value uncalculated pixel (step S1401), the surface of the pixel (n−1, m) and the processing target pixel (n, m) It is determined whether or not the surface is continuous (step S1402). Details of the determination will be described later.

  When the distance value non-calculated pixel (n−1, m) adjacent to the −X direction is continuous with the surface, + X direction estimated distance value calculation processing is performed (step S1403). In the + X direction estimated distance value calculation process, the distance value non-calculated pixel (n−1, m) adjacent to the −X direction, which is the estimated distance value Dc calculation target, for which the distance value is calculated in the + X direction Is used to calculate an estimated distance value Dc of a pixel whose distance value has not been calculated. The pixel for which the estimated distance value Dc is to be calculated is a pixel for which the distance value has not been calculated corresponding to the pixel (n−1, m) and a surface continuous with the pixel (n−1, m). Details of the processing will be described later.

  If it is determined in step S1402 that the surface is not a continuous surface, and if the + X direction estimated distance value calculation process is completed, the memory X used for the determination is reset (step S1404). Then, it is determined whether or not the pixel (n, m−1) adjacent in the −Y direction is a distance value non-calculated pixel (step S1405). If it is determined in step S1401 that the pixel (n−1, m) adjacent in the −X direction is not a distance value uncalculated pixel, the process proceeds to step S1404.

  When the pixel (n, m−1) adjacent in the Y direction is a distance value non-calculated pixel (step S1405), the surface of the pixel (n, m−1) and the surface of the processing target pixel (n, m) Whether or not is continuous (step S1406). Details of the determination will be described later.

  When it is continuous with the surface of the distance value uncalculated pixel (n, m−1) adjacent in the −Y direction, + Y direction estimated distance value calculation processing is performed (step S1407). In the + Y direction estimated distance value calculation process, the distance value non-calculated pixel (n, m−1) adjacent to the −Y direction, which is the estimated distance value Dc calculation target, for which the distance value is calculated in the + Y direction Is used to calculate the estimated distance value Dc of the pixel whose distance value has not been calculated. The pixel for which the estimated distance value Dc is to be calculated is a pixel for which the distance value has not yet been calculated, corresponding to the pixel (n, m−1) and a surface continuous with the pixel (n, m−1). Details of the processing will be described later.

  When it is determined in step S1406 that the surface is not a continuous surface, and when the + Y direction estimated distance value calculation process is completed, the area of the address n in the memory Y is reset (step S1408), and the process ends. If it is determined in step S1405 that the pixel (n, m−1) adjacent in the −Y direction is not yet calculated, the process proceeds to step S1408.

  Note that the estimated distance value calculation unit 171 of the present embodiment repeats the above process for all pixels. Every time processing of each pixel is finished, it is determined whether the pixel is a terminal pixel in the X direction, and if it is a terminal pixel, the memory X is reset. Further, it is determined whether the pixel is a terminal pixel in the Y direction, and if it is a terminal pixel, the area of the address n in the memory Y is reset. Thereby, when the processing of all the pixels is finished, all the areas of the memory X and the memory Y are reset.

<Inclination direction discrimination processing>
Next, details of the inclination direction determination processing by the estimated distance value calculation unit 171 in step S1101 and step S1201 will be described. In the present embodiment, a luminance value ratio is calculated between adjacent pixels, and if the luminance value ratio is within a predetermined range, as described above, the surface is a continuous surface, and the inclination is also determined from the direction. To do. FIG. 10 is a processing flow of the inclination direction determination processing of the present embodiment.

  First, in order to check whether the inclination is in the X direction, the luminance value B (n−1, m) of the pixel (n−1, m) adjacent to the processing target pixel (n, m) in the −X direction is used. Obtain (step S2101). Then, a luminance value ratio RB = B (n−1, m) / B (n, m) is calculated between the luminance value B (n, m) of the pixel (n, m) to be processed (step) S2102).

  Then, it is determined whether or not the calculated luminance value ratio RB is within the range of the luminance value ratio threshold RBth (step S2103). If it is within the range, it is a continuous surface in the X direction, and the inclination is also in the X direction. It is determined that it exists (step S2104).

  On the other hand, if it is out of range in step S2103, it is next determined whether or not the surface is a continuous surface in the Y direction. The determination is performed using the luminance value B (n, m−1) of the pixel (n, m−1) adjacent in the −Y direction.

  First, the luminance value B (n, m−1) of the pixel (n, m−1) adjacent in the −Y direction is acquired (step 2105). Then, the luminance value ratio RB = B (n, m−1) / B (m, m) is calculated (step S2106), and it is determined whether or not the luminance value ratio is within the range of the luminance value ratio threshold RBth (step S2107). .

  If it is within the range, it is determined that the surface is a continuous surface in the Y direction and the inclination is also in the Y direction (step S2108).

  Finally, in any direction, if the luminance value ratio is outside the range of the luminance value ratio threshold value, it is determined that the determination is impossible (step S2109).

  Note that the luminance value ratio threshold value RBth used here is also largely different depending on the surrounding environment, so the value cannot be determined unconditionally, and is set in actual use environment.

  Although not used in this processing, when acquiring the luminance values of the adjacent pixels (n−1, m) and (n, m−1), the distance value is also acquired.

<Memory X, Memory Y>
As illustrated in FIG. 9, when the inclination direction is determined, the estimated distance value calculation unit 171 of the present embodiment uses the distance value of the pixel adjacent to the minus direction in the direction to determine the estimated distance value Dc of the processing target pixel. Is calculated. Hereinafter, the memory X and the memory Y used here will be described with reference to FIGS. 11 (a) and 11 (b).

  The memory X410 according to the present embodiment is a working memory used for -X direction estimated distance value calculation processing, + X direction estimated distance value calculation processing, and processing to be described later when direction determination is impossible. As shown in FIG. 11A, the memory X410 includes a pixel position 1 (411) for storing information (pixel position) specifying the pixel position of the processing target pixel (n, m), and a processing target pixel (n, m, n). m) a pixel position 2 (412) for storing the pixel position at the end in the −X direction on the surface continuous with the surface, and a gradient 413 for storing the inclination in the X direction on the surface including the processing target pixel (n, m). With.

  Similarly, the memory Y420 of the present embodiment is a working memory used for the -Y direction estimated distance value calculation process, the + Y direction estimated distance value calculation process, and a later-described direction discrimination impossible process. As shown in FIG. 11B, the memory Y420 stores the pixel position of the processing target pixel (n, m) at the pixel position in the X direction of the distance image and the luminance image as the address 424, and the pixel position 1 (421). Y of the surface including the pixel to be processed (n, m), the pixel position 2 (422) for storing the pixel position at the end in the −Y direction, and the surface including the pixel to be processed (n, m). An inclination 423 for storing the inclination of the direction.

  In each of the following processes, the estimated distance value of a pixel whose distance value has not been calculated is calculated using these memories X and Y.

<-X direction estimated distance value calculation processing>
Next, the -X direction estimated distance value calculation process executed by the estimated distance value calculation unit 171 when the inclination is determined to be the X direction in step S1102 will be described. As described above, the −X direction estimated distance value calculation process uses the distance value of the pixel whose distance value or estimated distance value Dc is already calculated in the −X direction from the processing target pixel (n, m). This is a process of calculating the estimated distance value Dc of the processing target pixel (n, m). FIG. 12 shows a process flow of the -X direction estimated distance value calculation process of the present embodiment.

  First, the contents of the address n in the memory Y are reset (step S3101). This is because in this process, since the inclination is determined to be in the X direction, the contents of the memory Y used for calculating the inclination in the Y direction are not necessary.

  Next, it is determined whether or not any information is stored in the memory X (step S3102). When the distance value of the pixel (n−1, m) adjacent to the processing target pixel (n, m) in the −X direction is calculated, no information is stored in the memory X. That is, the current pixel to be processed (n, m) is a pixel that has not been calculated for a distance value that appears for the first time after a pixel for which a distance value has been calculated while processing pixels in the X direction. It becomes.

  In addition, here, the distance value and the luminance value of the pixel (n−1, m) adjacent in the −X direction are already acquired at the time of determining the tilt direction, and the continuity with the surface of this pixel is already determined. It is done. Therefore, here, the pixel at the end of the continuous surface is determined.

  For this reason, a distance value and a luminance value are sequentially acquired for each pixel in the −X direction, and continuity is determined using the distance value and the luminance value of adjacent pixels (steps S3103, 3104, and 3105). The discrimination is performed using the distance value and the luminance value between the search pixel (n−i, m) and the pixel (n−i−1, m) adjacent in the X direction.

  The distance value of the search pixel (n−i, m) is D (n−1, m), the luminance value is B (n−i, m), and the distance value of the adjacent pixel (n−i−1, m) is If D (n−i−1, m) and the luminance value is B (n−i−1, m), the distance difference between them Ddif = D (n−i, m) −D (n−i−1, The absolute value of m) is less than or equal to a predetermined distance threshold Dth, and the luminance value ratio RB = B (n−i, m) / B (n−i−1, m) is the luminance value ratio. When it is within the range of the threshold Bth, it is determined as a continuous surface.

  When it is determined as a continuous surface, it is determined whether or not the search pixel (n−i, m) is a pixel at the end in the X direction of the distance image and the luminance image (step S3106). If the pixel is not an end pixel, the next pixel in the −X direction is set as a search pixel (i = i + 1; step S3107), and the processing from step S3104 is repeated.

On the other hand, if the pixel is an end pixel, the distance value D (n−i, m) of the search pixel (n−i, m) and the pixel (n that is adjacent to the processing target pixel (n, m) in the −X direction) The inclination of the continuous surface is calculated using the distance value D (n−i, m) of (−1, m) (step S3108). The inclination K is calculated by the following equation (4).
K = (D (n−1, m) −D (n−i, m)) / (i−1) (4)
The calculated slope K is stored in the memory X.

Then, using the calculated slope K, an estimated distance value Dc of the processing target pixel (n, m) is calculated according to the following equation (5) (step S3109).
Dc (n, m) = D (n-1, m) + K (5)

  Then, the pixel position (n, m) of the processing target pixel (n, m) is stored in the pixel position 1 of the memory X (step S3110), and the process ends.

  On the other hand, if it is determined in step S3105 that the surface of the search pixel (n−i, m) is not continuous with the surface of the pixel adjacent to the pixel in the X direction, first, i at this time is 2. It is determined whether or not there is (step S3111).

  When i is not 2, that is, when i is larger than 2, there are two or more pixels that are continuous surfaces and have a distance value in the −X direction of the processing target pixel (n, m). Therefore, the inclination of the surface including the processing target pixel (n, m) can be calculated using these pixel values. Therefore, in this case, the process proceeds to step S3108, and the subsequent processing is performed.

  On the other hand, when i is 2, there is only one pixel which is a continuous surface and has a distance value in the −X direction of the processing target pixel (n, m). That is, only the pixel (n−1, m) adjacent to the processing target pixel (n, m). Therefore, the slope cannot be calculated. Therefore, in this case, the pixel position (n, m) of the processing target pixel (n, m) is stored in the pixel position 1 of the memory X, and the pixel position (n−1) of the adjacent pixel (n−1, m) is stored. , M) are stored in the pixel position 2 of the memory X as pixels at the end of the continuous surface (step S3112), and the process ends. Accordingly, although the inclination K is not calculated, the processing target pixel (n, m) and the pixels at the end in the −X direction of the continuous surface including the surface of the pixel are held in the memory X.

  On the other hand, if any information is stored in the memory X in step S3102, the current processing target pixel (n, m) is the distance value that does not appear for the first time after the pixel whose distance value is calculated. It is not a calculated pixel.

  First, it is determined whether or not the inclination K is stored (step S3113). Here, the surface of the processing target pixel (n, m) is continuous with the surface of the pixel (n-1, m) adjacent in the −X direction. Accordingly, the estimated distance value Dc can be calculated using the gradient used in the pixel (n−1, m) adjacent in the −X direction as it is. Therefore, if the tilt is stored, the process proceeds to step S3109, and the subsequent processing is executed.

  On the other hand, when the inclination K is not stored, the pixel position 2 is stored. In this case, even if the processing target pixel (n, m) having no distance value is added, the inclination cannot be calculated. Therefore, the process proceeds to step S3110, and the pixel position 1 in the memory X is set to the processing target pixel (n, m). (Step S3110). As a result, the range of the continuous surface and the range of the pixel whose distance value has not been calculated are stored.

<-Y direction estimated distance value calculation processing>
Next, the -Y direction estimated distance value calculation process executed by the estimated distance value calculation unit 171 when the inclination is determined to be the Y direction in step S1202 will be described. FIG. 13 shows a process flow of the −Y direction correction value calculation process of the present embodiment.

  The -Y direction estimated distance value calculation process is basically the same as the -X direction estimated distance value calculation process described above. However, the search direction and the storage mode in the memory Y are different. Hereinafter, a description will be given focusing on differences from the -X direction estimated distance value calculation processing.

  First, the contents of the memory X are reset (step S3202). This is because in this process, it is determined that the inclination is in the Y direction.

  Then, it is determined whether or not any information is stored in a column of the memory Y where the pixel position n in the X direction of the processing target pixel (n, m) is an address (step 3202). If not stored, the distance value and the luminance value are sequentially acquired pixel by pixel in the −Y direction, and continuity is determined (steps S3203, 3204, and 3205). In the same manner as in the −X direction estimated distance value calculation process, the distance value and the luminance value between the search pixel (n, mi) and the pixel (n, mi−1) adjacent in the Y direction are determined. To do.

  The above process is repeated until it is determined to be an end, and if it is determined to be an end (step S3206), the inclination of this continuous surface is calculated (step S3208), as in the -X direction estimated distance value calculation process. Then, the calculated inclination K is stored in the address n column of the memory Y.

  Then, the estimated distance value Dc of the processing target pixel (n, m) is calculated using the slope K (step S3209), and the processing target pixel (n, m) pixel is located at the pixel position 1 of the address n in the memory Y. Is stored (step S3210), and the process is terminated.

  On the other hand, if it is determined in step S3205 that the surface of the search pixel (n, mi) is not continuous with the surface of the pixel adjacent to the pixel in the Y direction, first, i at this time is 2. It is determined whether or not there is (step S3211).

  If i is not 2, the process proceeds to step S3208, and the subsequent processing is performed. On the other hand, when i is 2, there is only one pixel that is a continuous surface and has a distance value in the −Y direction of the processing target pixel (n, m). Therefore, in this case, the pixel position (n, m) of the processing target pixel (n, m) is stored in the pixel position 1 of the address n in the memory Y, and the pixel position ( n, m-1) is stored in the pixel position 2 of the address n in the memory Y as a pixel at the end of the continuous surface (step S3212), and the process is terminated.

  On the other hand, if any information is stored in the address n of the memory Y in step S3202, it is first determined whether or not the slope K is stored (step S3213). If the tilt is stored, the process proceeds to step S3209, and the subsequent processing is executed.

  On the other hand, if the slope K is not stored, the process proceeds to step S3210, and the pixel position 1 at the address n in the memory Y is updated to the processing target pixel (n, m) (step S3210). As a result, the range of the continuous surface and the range of the pixel whose distance value has not been calculated are stored.

<Processing when direction cannot be determined>
A process when the direction cannot be determined in step S1301 of the estimated distance value calculation process will be described. This processing is performed when there is an inclination but the direction cannot be determined. Here, the processing target pixel (n, m) is stored in the pixel position 1 and the pixel position 2 of the memory X. In addition, the processing target pixel (n, m) is stored in the pixel position 1 and the pixel position 2 of the address n in the memory Y. This is used for + X direction estimated distance value calculation processing or + Y direction estimated distance value calculation processing described later.

<Distance value uncalculated pixel presence / absence determination process>
Next, a method for determining whether or not there is a distance value uncalculated pixel in adjacent pixels, which is performed in step S1401 and step S1405, will be described.

  As described above, when the estimated distance value Dc of the processing target pixel (n, m) cannot be calculated in the −X direction estimated distance value calculation processing, the processing target pixel (n, m) and the processing are stored in the memory X. The pixel position at the end in the −X direction of the surface continuous with the surface of the target pixel (n, m) is stored in the pixel position 1 and the pixel position 2, respectively. Nothing is stored in the slope K.

  Therefore, in this step, using this, the estimated distance value calculation unit 171 accesses the memory X and determines whether or not the pixel position 2 is stored or whether the slope K is stored or not. It is determined whether or not the pixel (n−1, m) adjacent in the X direction is a distance value uncalculated pixel. That is, when the pixel position information is stored in the pixel position 2 or the inclination K is not stored, the pixel (n−1, m) is determined to be a pixel whose distance value has not been calculated.

  The determination in step S1405 is also the same, and the estimated distance value calculation unit 171 accesses the address n column of the memory Y, and the pixel position information is stored in the pixel position 2 as described above, or the slope K Is not stored, the pixel (n, m−1) is determined as a pixel whose distance value has not been calculated.

<Continuity discrimination processing>
Next, the determination method of whether it is a continuous surface of step S1402 and step S1406 is demonstrated.

  Here, as described above, the determination is made using the luminance value ratio RB. That is, the luminance value ratio RB is calculated between the luminance value of the processing target pixel (n, m) and the luminance value of the pixel (n−1, m) that is adjacent to the −X direction and the distance value has not been calculated. When it is within the range of the luminance value ratio threshold RBth, it is determined as a continuous surface.

  The determination method in step S1406 is the same, and the luminance value between the luminance value of the processing target pixel (n, m) and the luminance value of the pixel (n, m−1) that is adjacent to the −Y direction and has not yet been calculated. When the ratio RB is calculated and is within the range of the luminance value ratio threshold RBth, it is determined as a continuous surface.

<+ X direction estimated distance value calculation processing>
Next, the + X direction estimated distance value calculation process in step S1403 will be described. As described above, by using the distance value of the pixel whose distance value has already been calculated in the + X direction from the processing target pixel (n, m), the -X side of the processing target pixel (n, m) This is a process of calculating a distance value of a pixel whose distance value has not been calculated. FIG. 14 shows a process flow of the + X direction estimated distance value calculation process of the present embodiment.

  In this case, the memory X stores a pixel (n−1, m) adjacent to the processing target pixel (n, m) in the −X direction at the pixel position 1 and a processing target pixel (n , M), the pixel positions at the ends in the −X direction of the plane are stored.

  First, it is determined whether or not the pixel at the end in the −X direction stored in the pixel position 2 has a distance value (step S3301). As described above, in the -X direction estimated distance value calculation process, the slope of the continuous surface cannot be calculated, and the distance value is not calculated when all the pixels in the -X direction of the continuous surface have no distance value. This is the case where only one pixel at the end has a distance value.

  In the + X direction estimated distance value calculation process, since the processing target pixel (n, m) has a distance value, even when only one pixel at the end in the −X direction of the continuous surface has a distance value, The inclination can be calculated using the distance value of the processing target pixel (n, m).

  Therefore, when the pixel stored in the pixel position 2 has a distance value, the gradient K is calculated using the distance value of the processing target pixel (n, m) and the distance value of the pixel stored in the pixel position 2. (Step S3307), and using the inclination K, an estimated distance value of a pixel whose distance value from the pixel position 1 to the pixel position 2 has not been calculated is calculated (step S3308). Here, an estimated distance value from a pixel adjacent to the pixel position 2 in the + X direction to a pixel at the pixel position 1 (a pixel adjacent to the processing target pixel (n, m) in the −X direction) is calculated.

  Further, since the surface from the pixel at the pixel position 2 to the pixel at the pixel position 1 in the memory X is inclined in the X direction, the area of the memory Y using these as addresses is reset (step S3309). The process is terminated.

  On the other hand, in step S3301, when the pixel at the pixel position 2 has no distance value, the distance value of the uncalculated pixel cannot be calculated using the distance value of the pixel in the −X direction. Therefore, the distance value and the luminance value are acquired pixel by pixel in the + X direction to the end of the continuous surface, and the inclination of the continuous surface is calculated using these values (steps S3302 to S3307). After calculating the inclination, in step S3308, the estimated distance value of the pixel whose distance value has not been calculated between the pixel position 2 and the pixel position 1 is calculated (step S3308).

  The determination method of the continuous surface in step S3304 is the same as that of each estimated distance value calculation process described above, the absolute value of the distance value difference Ddif is equal to or less than the distance threshold value, and the luminance value ratio RB is the luminance value ratio threshold value RBth. If it is within the range, it is determined as a continuous surface.

  When the processing target pixel (n, m) is the end of the continuous surface in the + X direction, that is, when i is 1 in the processing flow (step S3310), the distance value is obtained on the continuous surface. Only the processing target pixel (n, m) is obtained. Therefore, in this case, even if searching in the + X direction, the inclination cannot be obtained, and the estimated distance value of the pixel whose distance value has not been calculated cannot be obtained. In this case, the process proceeds to step S3309.

<+ Y direction estimated distance value calculation processing>
Next, the + Y direction estimated distance value calculation process in step S1407 will be described. As described above, by using the distance value of the pixel whose distance value has already been calculated in the + Y direction from the processing target pixel (n, m), on the −Y side from the processing target pixel (n, m), This is a process of calculating a distance value of a pixel whose distance value has not been calculated. FIG. 15 shows a process flow of the + Y direction estimated distance value calculation process of the present embodiment. This process is basically the same as the + X direction estimated distance value calculation process. The following description will focus on differences from the + X direction estimated distance value calculation process.

  In this case, the pixel (n, m−1) adjacent to the processing target pixel (n, m) in the −X direction is stored at the address n of the memory Y at the pixel position 1, and the processing target is stored at the pixel position 2. The pixel position at the end in the −Y direction of the surface of the pixel (n, m) is stored.

  First, it is determined whether or not the pixel at the end in the −Y direction stored in the pixel position 2 has a distance value (step S3401). When the pixel stored at the pixel position 2 has a distance value, the inclination K is calculated using the distance value of the processing target pixel (n, m) and the distance value of the pixel stored at the pixel position 2. (Step S3407), the estimated distance value of the pixel whose distance value is not calculated between the pixel position 1 and the pixel position 2 is calculated using the inclination K (Step S3408). Here, the estimated distance value from the pixel adjacent to the pixel position 2 in the + Y direction to the pixel at the pixel position 1 (the pixel adjacent to the processing target pixel (n, m) in the −Y direction) is calculated, and the processing is ended. To do.

  On the other hand, if there is no distance value in the pixel at pixel position 2 in step S3401, the distance value and the luminance value are acquired one pixel at a time in the + Y direction to the edge of the continuous surface, and these are used to The inclination is calculated (steps S3402 to S3407).

  The determination method of the continuous surface in step S3404 is the same as the above-described estimated distance value calculation processing, the absolute value of the distance value difference Ddif is less than or equal to the distance threshold value, and the luminance value ratio RB is the luminance value ratio threshold value RBth. If it is within the range, it is determined as a continuous surface.

  Note that if the pixel to be processed (n, m) is an end of a continuous surface in the + X direction, that is, if i is 1 in the processing flow (step S3409), the inclination is obtained even if searching in the + Y direction. It is not possible to obtain the estimated distance value of the pixel whose distance value has not been calculated. Therefore, the process is terminated as it is.

<Description of Estimated Distance Value Calculation Processing Using Specific Example>
The estimated distance value calculation process will be described with a specific example. In the following examples, it is assumed that an object exists only in the illustrated pixel, and no object exists in the pixel not illustrated. That is, the luminance values of the illustrated pixels are all equal to or higher than the luminance threshold Bth. Also, it is assumed that no information is stored in the memories X and Y. Further, it is assumed that none of the pixels is a terminal pixel.

<Specific example 1>
FIG. 16 shows the shape of the reflecting surface of the object 113 and the arrangement of corresponding pixels. An example of calculating the distance value along the pixel readout direction will be described. This is an example in which the estimated distance value of a pixel whose distance value has not been calculated is calculated using the distance value of the pixel in the −X direction of the pixel. This corresponds to the flow in which the estimated distance value can be calculated by obtaining the slope in step S1102 of FIG. 9 and FIG.

  Since the distance value is not 0 for the pixel (n, m), the processing after step S1401 is performed. First, uncalculated pixel presence / absence determination processing is performed. Read information from memory X. Nothing is stored in the memory X. Next, information is read from the memory Y. Since nothing is stored in the memory Y, it is determined that there is no distance value uncalculated pixel in the adjacent pixel, and the processing of this pixel is finished. Then, the process proceeds to the next pixel (n + 1, m).

  Since the distance value is not 0 for the pixel (n + 1, m), the same processing as that of the pixel (n, m) is repeated, and the processing of the pixel (n + 2, m) is performed. Thereafter, the same processing is performed on the pixel (n + 2, m) and the pixel (n + 3, m), and the processing on the pixel (n + 4, m) is performed.

  Since the distance value is 0 for the pixel (n + 4, m), the process proceeds to step S1101 to determine the tilt direction. Here, it is assumed that the X direction is determined. Accordingly, the process proceeds to the -X direction correction distance value calculation process in step S1102.

  First, the area of the memory Y with n + 4 as an address is reset. Then, the memory X is read out and it is determined whether or not the inclination is stored.

  Since the inclination is not stored, the distance value and the luminance value of the X direction -2 pixel, that is, the pixel (n + 2, m) are read, the distance difference / luminance value ratio is calculated, and the edge search process of the continuous plane (steps S3104 to S3104) Step S3107) is performed. Between the pixel (n + 3, m) and the pixel (n + 2, m), the distance difference is within the distance threshold range, and the luminance value ratio is also within the luminance ratio threshold range. Therefore, it is determined as a continuous plane. Further, the pixel (n + 2, m) is not the starting end pixel.

  Accordingly, the distance value / luminance value of the X direction-3 pixel, that is, the pixel (n + 1, m) is read out, and the search process is repeated in the same manner. The distance difference between the pixel (n + 2, m) and the pixel (n + 1, m) is outside the distance threshold range (less than the range). Therefore, it is determined as a discontinuous surface. At this time, since i is 3, the process proceeds to the calculation of the slope in step S3108.

  Using the distance values Q and R between the pixel (n + 2, m) and the pixel (n + 3, m), the slope K is calculated as (R−Q) / (n + 3−n−2) = R−Q. Then, using the gradient K = R−Q, the distance value of the pixel (n + 4, m) is calculated as 2R−Q.

  The pixel (n + 4, m) is stored in the pixel position 1 (411) of the memory X, and the inclination K is stored in the inclination (413) of the memory X. Then, the process proceeds to the next pixel (n + 5, m).

  For the pixel (n + 5, m), the luminance value is greater than or equal to the luminance threshold value, and therefore it is determined that an object exists. Since the distance value is 0, the process proceeds to step S1101 to determine the tilt direction. Here, it is assumed that the X direction is determined. That is, the surface is continuous with the surface of the pixel (n + 4, m). Therefore, the process proceeds to the -X direction correction distance value calculation process in step S1102.

  First, the area of the memory Y with n + 5 as an address is reset. Then, the memory X is read out, and it is determined whether or not the inclination is stored in association with the adjacent pixels (n + 4, m) in the −X direction.

  Since the inclination is stored, the distance value 3R-2Q of the pixel (n + 5, m) is calculated using the inclination RQ and the distance value 2R-Q of the pixel (n + 4, m). Then, the pixel position 1 in the memory X is updated to the pixel (n + 5, m), and the next pixel (n + 6, m) is transferred.

  Since the distance value is not 0 for the pixel (n + 6, m), first, a distance value non-calculated pixel presence / absence determination process is performed. Read memory X. Here, the inclination is stored in the memory X. Accordingly, it is determined that there is no distance value uncalculated pixel in pixels adjacent in the −X direction. Therefore, the memory X is reset and reading of the memory Y is started.

  Since nothing is stored in the memory Y, the process proceeds to the process for the next pixel and the process is repeated.

<Specific example 2>
Next, processing when the shape and position of the target object 113 are in the form shown in FIG. 17 will be described. This is an example in which the estimated distance value of a pixel whose distance value has not been calculated is calculated using a pixel in the + X direction of the pixel. In FIG. 9, the distance value cannot be obtained in the −X direction estimated distance value calculation process of S1102, and step S1403 and the step of + X direction estimated distance calculation process of FIG. 14 are performed using the distance values of pixels adjacent in the + X direction. This is an example in which the processing of S3302 to S3306 is performed, the inclination is calculated, and the distance value is obtained.

  Since the distance value of the processing target pixel (n, m) is not 0, the processing after step S1401 is performed. It is confirmed whether the pixel position is stored in the memory X. Since the pixel position is not stored, the memory X is reset and the memory Y is checked. Since the pixel position is not stored in the memory Y, the process proceeds to the next pixel (n + 1, m). The pixel (n + 1, m) is processed in the same manner as the pixel (n, m), and the process proceeds to the pixel (n + 2, m).

  Pixel (n + 2, m) has a distance value of zero. Here, it is assumed that the inclination direction cannot be determined. In this case, the pixel (n + 2, m) is stored in the pixel position 1 and the pixel position 2 of the memory X. Similarly, the pixel position (n + 2, m) is stored in the pixel position 1 and the pixel position 2 of the line n + 2 of the memory Y. Then, the process proceeds to the next pixel (n + 3, m).

  Since the distance value of the next processing target pixel (n + 3, m) is 0, the inclination direction is determined. Here, it is assumed that the X direction can be determined. Thereafter, -X direction estimated distance value calculation processing is performed.

  First, the area at the address n + 3 in the memory Y is reset, and it is determined whether the pixel position 1, the pixel position 2, or the inclination is stored in the memory X. Here, the pixel position 1 and the pixel position 2 are stored, and the inclination is not stored. Therefore, the process proceeds to step S3110, the pixel position 1 is updated to the pixel (n + 3, m), and the next pixel (n + 4, m) is transferred. Note that the pixel position 2 remains the pixel (n + 2, m).

  The processing target pixel (n + 4, m) is subjected to the same processing as that of the pixel (n + 3, m), is determined in the same manner, and proceeds to the processing of the pixel (n + 5, m).

  Since the processing target pixel (n + 5, m) has a distance value that is not 0, the processing after step S1401 is performed. First, a distance value non-calculated pixel presence / absence determination process is performed.

  First, the memory X is read, and it is determined whether or not position information is stored at the pixel position 2. Since the position information is stored at the pixel position 2, it is determined whether or not it is continuous with the surface of the pixel adjacent in the −X direction. Here, it shall be continuous. Therefore, + X direction estimated distance value calculation processing is performed.

  Next, the distance value and the luminance value of the X direction + 1 pixel, that is, the pixel (n + 6, m) are acquired, and it is determined whether or not the processing target pixel (n + 5, m) is a continuous surface. Here, it is assumed that it is determined as a continuous surface. Further, the distance value and the luminance value in the X direction +2 pixels, that is, (n + 7, m) are acquired, and the continuity is determined. Here, it is assumed that it is determined as discontinuous.

  In this case, since i is 2 in step S3310, two distance values can be obtained on the same continuous surface, and the slope can be calculated. That is, the slope K is calculated using the distance value Q of the pixel (n + 5, m) and the distance value R of the pixel (n + 6, m). The calculated slope is RQ.

  A distance value is calculated using this inclination. Here, the distance value is calculated in order from the pixel position 1 stored in the memory X toward the pixel position 2. The distance value of the pixel (n + 4, m) stored at the pixel position 1 is calculated as 2Q−R since the slope is R−Q and the distance value of the pixel (n + 5, m) is Q. Similarly, the distance value of the pixel (n + 3, m) is calculated as 3Q-2R, and the distance value of the pixel (n + 2, m) is calculated as 4Q-3R.

  Since the pixel (n + 2, m) is at the pixel position 2, after calculating the distance value so far, the area of the memory Y having the range (n + 4 to n + 2) corresponding to the pixel position 1 to the pixel position 2 as an address is reset. . Then, the memory X is reset.

  Next, the information at the address (n + 5) of the memory Y is read. Here, position information is not stored at pixel position 2. Therefore, the process proceeds to the next pixel (n + 6, m).

  The processing target pixel (n + 6, m) does not have a distance value of 0, and therefore the processing from step S1401 is performed. First, the memory X is read, and it is determined whether or not position information is stored at the pixel position 2. Position information is not stored at pixel position 2. Therefore, the memory X is reset and the memory Y is read. Here, no position information is recorded at pixel position 2 at address n + 6 in memory Y. Accordingly, it is determined that there is no distance value uncalculated pixel in the adjacent pixels, the process proceeds to the next pixel (n + 7, m), and the process is repeated.

<Specific example 3>
Next, an example of calculating a distance value using pixel values of pixels corresponding to both ends of the same plane will be described in the case where the shape and position of the object 113 are in the form shown in FIG. As in the second specific example, in FIG. 9, the distance value cannot be obtained in the −X direction estimated distance value calculation process of S1102, and the distance value of pixels adjacent in the + X direction is used to perform step S1403 and the + X direction estimated distance of FIG. It is an example which obtains a distance value by calculation processing. However, this example is an example in which the process proceeds from step S3301 to step S3307.

  Since the distance value of the processing target pixel (n, m) is not 0, the processing after step S1401 is performed. Here, it is confirmed whether or not the pixel position is stored in the memory X. Since the pixel position is not stored, the memory X is reset and the memory Y is checked. Since the pixel position is not stored in the memory Y, the process proceeds to the next pixel (n + 1, m).

  Since the processing target pixel (n + 1, m) has a distance value that is not 0, the processing after step S1401 is performed. Here, the same processing as that for the pixel (n, m) is performed, and the processing proceeds to the processing for the next pixel (n + 2, m).

  The processing target pixel (n + 2, m) has a distance value of 0. Therefore, first, the inclination direction is determined. Here, it is assumed that the X direction can be determined. Therefore, -X direction estimated distance value calculation processing is performed.

  The memory Y at the address (n + 2) is reset. Then, the memory X is read to determine whether data is stored. Since nothing is stored here, the process goes to step S3103 and subsequent steps for searching for the end of the continuous surface. Here, the adjacent pixel (n + 1, m) is specified as the end of the continuous surface. That is, i is 2 at this time.

  Therefore, there are only pixels (n + 1, m) on the −X direction side of the continuous surface of the processing target pixel (n + 2, m), and the inclination of the surface cannot be calculated. Accordingly, in step S3112 the processing target pixel (n + 2, m) is stored in the pixel position 1 of the memory X, and the pixel (n + 1, m) corresponding to the edge of the continuous surface is stored in the pixel position 2 of the memory X. . Then, the process proceeds to the next pixel (n + 3, m).

  Since the processing target pixel (n + 3, m) has a distance value of 0, first, the direction of inclination is determined. It is assumed that the direction is the X direction, and the -X direction estimated distance value calculation process is performed.

  The memory Y at the address (n + 3) is reset. Then, the memory X is read. Here, although the pixel position 1 and the pixel position 2 are stored, but the inclination is not stored, the process proceeds to step S3110, and the pixel position 1 is updated to the processing target pixel (n + 3, m). The process proceeds to the processing of the pixel (n + 4, m).

  The pixel (n + 4, m) is processed in the same manner as the pixel (n + 3, m) and moves to the next pixel (n + 5, m). At that time, the pixel position 1 of the memory X is updated to the pixel position (n + 4, m).

  Since the processing target pixel (n + 5, m) has a distance value that is not 0, the processing after step S1401 is performed. First, a distance value non-calculated pixel presence / absence determination process is performed. Here, since the memory X is read and the pixel position is stored at the pixel position 2, it is determined that there is an uncalculated pixel in the pixel adjacent in the −X direction.

  Then, the continuity between the processing target pixel (n + 5, m) and the pixel (n + 4, m) adjacent in the −X direction is determined. Here, it is assumed that it is determined to be continuous. Therefore, + X direction estimated distance value calculation processing is performed.

  It is determined whether or not the distance value of the pixel stored in the pixel position 2 of the memory X, that is, the pixel at the end of the continuous surface is calculated. Here, (n + 1, m) is stored in the pixel position 2. Further, the distance value of this pixel is calculated as Q.

  Accordingly, the process proceeds to step S3307, and the inclination of the continuous surface is calculated using the distance value R of the processing target pixel (n + 5, m) and the distance value Q of the pixel (n + 1, m). The slope K calculated here is (R−Q) / 4. And using this inclination, the estimated distance value of the pixel whose distance value has not been calculated between the pixel position 2 and the pixel position 1 is calculated.

  Finally, the area of the memory Y whose address is a range corresponding to the pixel position 1 to the pixel position 2 in the memory X is reset. Then, the memory X is reset.

  Next, the information at the address (n + 5) of the memory Y is read. Here, position information is not stored at pixel position 2. Therefore, the process proceeds to the next pixel (n + 6, m).

  The processing target pixel (n + 6, m) does not have a distance value of 0, and therefore the processing from step S1401 is performed. First, a distance value non-calculated pixel presence / absence determination process is performed. Here, position information is not stored in the pixel position 2 of the memory X. The same applies to the memory Y. Therefore, it is determined that there is no distance value uncalculated pixel in the adjacent pixels, and the process proceeds to the next pixel (n + 7, m), and the process is repeated.

<Specific Example 4>
Next, processing when the shape and position of the target object 113 are in the form shown in FIG. 19 will be described. This is an example in which the estimated distance value cannot be calculated because the distance values of all the pixels corresponding to the inclined surface are not obtained.

  Since the processing target pixel (n, m) has a distance value which is not 0, the processing after step S1401 is performed. Here, it is confirmed whether or not the pixel position is stored in the memory X. Since the pixel position is not stored, the memory X is reset and the memory Y is checked. Since the pixel position is not stored in the memory Y, the process proceeds to the next pixel (n + 1, m).

  The processing target pixel (n + 1, m) has a distance value of 0. Therefore, first, the inclination direction is determined. Here, it is assumed that the inclination direction cannot be determined. In this case, the pixel (n + 1, m) is stored in the pixel position 1 and the pixel position 2 of the memory X. Similarly, the pixel position (n + 1, m) is stored in the pixel position 1 and the pixel position 2 of the line n + 1 of the memory Y. Then, the process proceeds to the next pixel (n + 2, m).

  Since the distance value of the next processing target pixel (n + 2, m) is 0, the inclination direction is determined. Here, it is assumed that the X direction can be determined. Thereafter, -X direction estimated distance value calculation processing is performed.

  First, the area at the address n + 2 in the memory Y is reset, and it is determined whether the pixel position 1, the pixel position 2, or the inclination is stored in the memory X. Here, the pixel position 1 and the pixel position 2 are stored, and the inclination is not stored. Therefore, the process proceeds to step S3110, where the pixel position 1 is updated to the pixel (n + 2, m), and the next pixel (n + 3, m) is transferred. Note that the pixel position 2 remains the pixel (n + 1, m).

  The processing target pixel (n + 3, m), the processing target pixel (n + 4, m), and the processing target pixel (n + 5, m) are also processed in the same manner as the pixel (n + 2, m), respectively, and are similarly determined. The process proceeds to processing of the pixel (n + 6, m). At this time, in the memory X, the pixel (n + 5, m) is stored in the pixel position 1, the pixel (n + 1, m) is stored in the pixel position 2, and the inclination is not stored.

  Since the distance value of the pixel (n + 6, m) is not 0, the processing after step S1401 is performed. First, a distance value non-calculated pixel presence / absence determination process is performed. Here, a pixel (n + 1, m) is stored in the pixel position 2 of the memory X. Therefore, it is determined that there is a distance value uncalculated pixel in the adjacent pixels in the X direction.

  First, it is determined whether there is continuity between adjacent pixel surfaces and the X direction. Here, it is determined that there is no continuity, the memory X is reset, and it is determined whether or not there is a distance value uncalculated pixel in the adjacent pixels in the Y direction. Here, it is assumed that pixel position information is not stored at the same address in the memory Y. Therefore, in the Y direction, it is determined that there is no distance value uncalculated pixel in the adjacent pixels.

  Accordingly, in this case, the memory X is also an area of the memory Y having the pixel position n in the X direction of the processing target pixel (n + 6, m) as an address in a state where the distance value of the X direction adjacent pixels whose distance value has not been calculated cannot be calculated. Is also reset, and the process proceeds to the next pixel.

  Thus, in this example, the inclination of the pixel from the pixel (n + 1, m) to the pixel (n + 5, m) is not calculated, and the distance value remains 0.

<Flow of distance image generation processing>
Finally, the flow of the distance image generation process of the present embodiment by the above units will be described. FIG. 20 is a processing flow of the distance image generation processing of the present embodiment.

  First, the light source unit 110 emits modulated light, the incident light including the reflected light is received by the photoelectric conversion unit 120, and the charge distribution unit 130 distributes the light to each charge storage unit 150. This continues until enough charge is accumulated to generate the distance image. At this time, if charges sufficient to generate a distance image are accumulated (step S1901), the luminance image generation unit 160 generates a luminance image, and the distance image generation unit 170 generates a distance image (step S1902). .

  Thereafter, the estimated distance value calculation unit 171 calculates the estimated distance value of the pixel for which the distance value has not been calculated by the above method (step S1903), and the distance image generation unit 170 uses the calculated estimated distance value as the pixel for which the distance value has not been calculated. (Step S1904), and the process ends.

As described above, the distance image generation device according to the present embodiment has the light source unit 110 that irradiates the modulated light 111 that is modulated into the target space, and the surface of the target object 113 that is irradiated from the light source unit and that is within the target space. A photoelectric conversion element (photoelectric conversion unit 120) provided for each pixel that receives incident light 112 including reflected reflected light and converts it into charges, and a plurality of charge storage units 150 provided for each photoelectric conversion element; The charge distribution unit 130 that distributes the charges converted by the photoelectric conversion elements to the plurality of charge storage units 150 in synchronization with the modulation of the light source unit 110 and the charges stored in the charge storage unit 150 are used. A luminance image generation unit 160 that generates a luminance image 220 whose pixel value is a luminance value, and a distance image 210 whose pixel value is a distance value using the charges accumulated in the plurality of charge accumulation units 150. A distance image generation unit 170 that performs the distance value calculation of the distance value uncalculated pixels for which the distance value cannot be obtained due to the inclination of the surface of the object 113. A distance value calculation unit (estimated distance value calculation unit 171) that calculates using the luminance value and the distance value of the pixels in the vicinity of the value non-calculated pixel is provided.
The neighboring pixels used here are pixels that are adjacent to the non-distance value calculation pixels adjacent to each other in a predetermined direction as searched by changing i in the above-mentioned -X direction estimated distance value calculation processing or the like. Thus, the luminance value ratio between adjacent pixels is a pixel in a predetermined range.

  Further, the estimated distance value calculation unit 171 is described in the loop of steps S3103 to S3106 of the -X direction estimated distance value calculation process, the loop of steps S3302 to 3305 of the + X direction estimated distance value calculation process, and the same S3301. As described above, when there are a plurality of pixels for which the distance value is calculated in the neighboring pixels, the distance value is calculated by using the plurality of pixels to calculate the inclination of the surface of the object. The distance value of the uncalculated pixel is calculated.

  The estimated distance value calculation unit 171 of the present embodiment determines, for each pixel, whether or not the pixel is a pixel for which a distance value has not been calculated (distance value uncalculated pixel) in a predetermined order. When it is determined that the pixel is inclined, as in steps S1101 and S1201 described above, the inclination direction of the surface of the object 113 is specified, and the direction of the specified inclination direction is opposite to the direction in which the pixels are determined in the order. A distance value of a pixel whose distance value has not been calculated is estimated using a pixel that is adjacent to the pixel and has a luminance value ratio (luminance value ratio) with a neighboring pixel as a neighboring pixel.

  In the present embodiment, as described above, the specified inclination direction is either the X direction or the Y direction. Therefore, when the X direction is specified, the estimated X-direction distance value calculation process in step S1102 is performed. In this -X direction estimated distance value calculation process, as in steps S3103 to S3107, a pixel range that is continuous in the -X direction and has a continuous surface in the field of view is specified, and the distance value of these pixels is determined. The distance value of the pixel whose distance value has not been calculated is calculated. The same applies when the Y direction is specified.

  On the other hand, if it is determined NO, the estimated distance value calculation unit 171 of the present embodiment determines whether or not a pixel adjacent to the determination target pixel in the reverse direction to the pixel determination order is a distance value non-calculated pixel. If the pixel is determined to be a pixel whose distance value has not yet been calculated, pixels that are adjacent to each other in the inclination direction and in the direction in which the pixels are sequentially determined, and that have a luminance value ratio with the adjacent pixels in a predetermined range Is used as the neighboring pixel, and the distance value of the pixel whose distance value has not been calculated is calculated.

  As described above, the estimated distance value calculation unit 171 of the present embodiment proceeds to step S1401 when the processing target pixel is not a distance value non-calculated pixel. First, it is determined whether a pixel that is adjacent to the −X direction and the surface in the field of view is a continuous surface has no distance value calculated pixel. If there is, the + X direction estimated distance value is calculated in step S1403. Proceed to processing. Then, as in steps S3302 to S3306, a pixel range in which surfaces in the field of view are continuous in the + X direction is specified, and the distance value of the distance value uncalculated pixels is calculated using the distance values of these pixels.

  Whether the surfaces in the field of view are continuous or not is determined using the luminance value ratio as described above. If the luminance value ratio is within a predetermined range, it is determined that the reflecting surface of the object 113 in the field of view of both pixels is continuous.

  As described above, according to the present embodiment, even if there is a portion where there is no reflected light due to the modulated light, the estimated distance value is calculated using the luminance value of the portion, the nearby luminance value, and the distance value. Since the reflecting surface of the object 113 is inclined, the estimated distance value can be calculated even when the distance value cannot be obtained. Therefore, a highly accurate distance image can be obtained regardless of the shape of the object.

  At this time, according to the present embodiment, the same processing is prevented from being repeated by performing memory storage / reading processing of the inclination value and the pixel position. For example, the previous distance value non-calculated pixel is included in the search range and the inclination of the reflecting surface is calculated as in the process of shifting from step S3113 to step S3109 in the -X direction estimated distance value calculation process. If it is, the estimated distance value is calculated using this inclination.

  As described above, in the present embodiment, the slope and information calculated once are stored and used in the subsequent process, thereby preventing a decrease in processing speed due to an increase in the same arithmetic processing and suppressing a significant decrease in frame rate. it can.

  Moreover, according to this embodiment, since the estimated distance value is calculated using information obtained by one distance image generation device 100, it is not necessary to use a plurality of distance image generation devices. Therefore, it is possible to reduce the cost and size.

  Further, the estimated distance value is calculated using the distance value and the luminance value calculated from the accumulated charges. Therefore, since background data serving as a reference is unnecessary, an estimated distance value can be obtained in real time even for a moving object.

<Modification>
In the above embodiment, the light source unit 110 always emits modulated light, and the incident light including the reflected light calculates the distance value and the luminance value from the accumulated charges, thereby obtaining the distance image and the luminance image, respectively. ing. However, the method of the distance image and the luminance image is not limited to this. For example, you may comprise so that it may acquire at an independent timing.

  That is, a luminance value calculation charge acquisition period and a distance value calculation charge acquisition period are provided. These periods are determined in advance as timing charts and stored in a storage device or the like.

  In this case, the control unit 140 sends a control signal (read signal) to the luminance image generation unit 160 during the luminance value calculation charge acquisition period to read out the charge, and in the case of the distance value calculation charge, the distance image generation unit 170. Then, a control signal (read signal) is sent to read the charge.

  When the luminance value is calculated, phase information is not necessary because it is finally added as described above. Only intensity information needs to be obtained. Therefore, it is not necessary to distribute the plurality of charge storage units every predetermined period. Therefore, for example, as illustrated in FIG. 21, the control unit 140 may output the gate signal so that charges are accumulated only in one charge accumulation unit 150 during the luminance value calculation charge acquisition period. With such control, the luminance value is calculated from the modulated light and the ambient light.

  Further, no modulated light is required for calculating the luminance value. Therefore, the modulation signal to the light source unit 110 may be controlled. That is, the control unit 140 transmits a modulation signal to the light source unit 110 during the distance value calculation charge acquisition period, and does not transmit a modulation signal to the light source unit 110 during the luminance value calculation charge acquisition period. That is, the control unit 140 controls the light source unit 110 not to emit the modulated light during the luminance value calculation charge acquisition period. This is performed according to a predetermined timing chart. In this case, the control unit 140 outputs the gate signal in synchronization with the modulation signal only during the distance value calculating charge acquisition period.

  An example of the modulation signal and the gate signal in the luminance value calculation charge acquisition period in this case is shown in FIG. Thus, no modulated signal is transmitted during this period. On the other hand, the gate signal is output so that charges are stored only in one charge storage unit 150, as in FIG.

  The light source unit 110 emits light according to the modulation signal only when the modulation signal is received. The photoelectric conversion unit 120 receives incident light including reflected light of modulated light. In addition, while the light source unit 110 is not emitting light, the photoelectric conversion unit 120 receives only ambient light.

  The luminance value calculation charge acquisition period may be provided at a ratio of 1: 1 or 1:10 for each period of the modulation signal, and can be freely set.

  With this configuration, a period for receiving the luminance of each pixel and a period for receiving the reflected light of each pixel are provided, and the luminance value and the distance value output as a result are mutually determined by the above-described method. Utilizing this, the distance value of the inclined portion is calculated.

  As a result, even when there is no reflected light due to the modulated light, it is possible to reliably obtain the luminance value and to calculate a more accurate estimated distance value.

  In the above embodiment, the estimated distance value calculation unit 171 is described as being included in the distance image generation unit 170, but is not limited to this configuration. For example, an information processing device that can transmit and receive data to and from the distance image generation device 100 and that is provided independently of the distance image generation device 100 may be configured to realize the estimated distance value calculation unit 171.

  Similarly, the luminance image generation unit 160 and the distance image generation unit 170 may be realized by an external device. In this case, every time a predetermined amount of charge is accumulated, the control unit 140 transmits the charge to an external device that implements the luminance image generation unit 160 and the distance image generation unit 170.

  DESCRIPTION OF SYMBOLS 100: Distance image generation apparatus, 110: Light source part, 111: Modulated light, 112: Incident light, 113: Object, 113a: Object, 113b: Wall, 114: Modulation signal, 115: Gate signal, 120: Photoelectric conversion Unit: 130: charge distribution unit, 140: control unit, 150: charge storage unit, 160: luminance image generation unit, 170: distance image generation unit, 171: estimated distance value calculation unit, 210: distance image, 220: luminance Image: 300: Pixel position, 410: Memory X, 411: Pixel position 1, 412: Pixel position 2, 413: Tilt, 420: Memory Y, 421: Pixel position 1, 422: Pixel position 2, 423: Tilt, 424 :address

Claims (11)

A light source unit that emits modulated light modulated into the target space;
A photoelectric conversion element provided for each pixel that receives incident light including reflected light irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the incident light into charges;
A plurality of charge storage portions provided for each photoelectric conversion element;
A distance image generation method in a distance image generation apparatus comprising: a charge distribution unit that distributes charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with modulation of the modulated light,
A luminance image whose pixel value is a luminance value is generated using the charge accumulated in the charge accumulation unit, and a distance image whose pixel value is a distance value using the charge accumulated in the plurality of charge accumulation units. An image generation step to generate;
A distance value of a distance value non-calculated pixel for which the distance value cannot be obtained due to an inclination on the surface of the object is calculated using the luminance value and the distance value of pixels in the vicinity of the distance value non-calculated pixel. A distance value calculating step. A distance image generating method comprising: A light source unit that emits modulated light modulated into the target space;
A photoelectric conversion element provided for each pixel that receives incident light including reflected light irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the incident light into charges;
A plurality of charge storage portions provided for each photoelectric conversion element;
A charge distribution unit that distributes the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the modulated light, and is stored in the charge storage unit. Generating a luminance image whose pixel value is a luminance value using the measured charge, and generating a distance image whose pixel value is a distance value using the charges accumulated in the plurality of charge accumulation units, and the target A procedure for calculating a distance value of a distance value non-calculated pixel for which the distance value cannot be obtained due to an inclination of the surface of the object, using the luminance value and the distance value of a pixel in the vicinity of the distance value non-calculated pixel. A program for running A light source unit that emits modulated light modulated into the target space;
A photoelectric conversion element provided for each pixel that receives incident light including reflected light irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the incident light into charges;
A plurality of charge storage portions provided for each photoelectric conversion element;
A charge distribution unit that distributes the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the modulated light;
A luminance image generation unit that generates a luminance image whose pixel value is a luminance value using the charge accumulated in the charge accumulation unit;
A distance image generation unit that generates a distance image in which a pixel value is a distance value using charges accumulated in the plurality of charge accumulation units;
The distance image generation unit calculates a distance value of a distance value non-calculated pixel for which the distance value cannot be obtained due to an inclination on a surface of the object, the luminance value of a pixel in the vicinity of the distance value uncalculated pixel, and A distance image generation apparatus comprising a distance value calculation unit that calculates using the distance value. The distance image generating device according to claim 3,
The neighboring pixel is a pixel adjacent to and adjacent to the pixel whose distance value has not been calculated in a predetermined direction, and a luminance value ratio with the adjacent pixel is a pixel within a predetermined range. The distance image generating device according to claim 4,
When there are a plurality of pixels for which the distance value is calculated in the neighboring pixels, the distance value calculation unit calculates the inclination of the surface of the object using the plurality of pixels, and uses the inclination A distance image generation apparatus characterized by calculating a distance value of the pixel whose distance value has not been calculated. The distance image generating device according to claim 4 or 5,
The distance value calculation unit determines, for each pixel, whether or not the pixel is a distance value non-calculated pixel in a predetermined order. If the distance value calculation unit determines that the pixel is a distance value non-calculated pixel, the inclination of the surface of the object is determined. The direction is specified, and the pixels in the specified inclination direction are adjacent to each other in the direction opposite to the direction in which the pixels are determined in the order, and the pixels whose luminance value ratio with the adjacent pixels is within a predetermined range A distance image generating apparatus, wherein a distance value of the pixel whose distance value has not been calculated is calculated as a pixel. The distance image generation device according to claim 6,
When it is determined that the distance value calculation unit is not, the distance value calculation unit determines whether or not a pixel adjacent to the determination target pixel in the reverse direction to the pixel determination order is a distance value non-calculated pixel. When it is determined as an uncalculated pixel, a pixel that is adjacent to and in a direction in which the pixels are sequentially determined in the tilt direction and that has a luminance value ratio with the adjacent pixel in a predetermined range is set as the neighboring pixel. A distance image generating apparatus that calculates a distance value of the distance value non-calculated pixels. The distance image generation device according to claim 6,
The distance value calculation unit, for a pixel determined to be a distance value non-calculated pixel, a pixel previously determined to be a distance value non-calculated pixel is included in the neighboring pixels and the inclination is calculated And calculating the distance value using the inclination. The distance image generating device according to any one of claims 6 to 8,
The distance value calculation unit identifies an end pixel of the neighboring pixels, and uses the distance value of the end pixel for calculating the inclination. The distance image generation device according to any one of claims 3 to 9,
A control unit for controlling the charge distribution unit;
The control unit controls the charge distribution unit to store all charges in one predetermined charge storage unit when acquiring the charge for generating the luminance image. Generator. The distance image generation device according to any one of claims 3 to 9,
A control unit for controlling the light source unit;
The distance image generation device, wherein the control unit controls the light source unit not to irradiate the modulated light when acquiring the charge for generating the luminance image.