JP2009192499A - Apparatus for generating distance image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a distance image generation apparatus capable of generating a distance image with higher accuracy. <P>SOLUTION: The apparatus includes a light source which irradiates a modulated light onto an object space; an image sensor comprising a plurality of photoelectric conversion elements which receive a reflection light irradiated from the light source and reflected by an object in the object space and converts the light into electric charges, a plurality of charge accumulating sections provided for each of the photoelectric conversion elements, and a means which synchronizes with the modulation of the light source and distributes the electric charge converted by the photoelectric conversion elements to the plurality of charge accumulating sections; a distance image generating section which performs predetermined operations based on the electric charge accumulated in the plurality of charge accumulating sections to generate a distance image whose pixel value is a distance value; and a reflectivity image generating section which performs predetermined operations based on an amplitude of the reflection light received by the photoelectric conversion elements and the pixel value of the distance value generated by the distance image generating section to generate a reflectivity image whose pixel value is a reflective coefficient correlated with a reflectivity. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a distance image generation device, and more particularly to a distance image generation device capable of generating a distance image with higher accuracy.

  2. Description of the Related Art Conventionally, an apparatus that generates a so-called distance image using a distance image sensor (there is a CCD type or a CMOS type) is known (for example, Patent Document 1).

The distance image generation device described in Patent Document 1 emits light whose intensity is modulated at a predetermined light emission frequency into a space, receives a reflected light of light emitted from the light source, and receives a signal level corresponding to the received light intensity. And an image sensor as a distance image sensor including a plurality of photoelectric conversion units that output the received light signal. In the distance image generating device described in Patent Document 1, reflected light is received and converted into electric charges within a modulation period, and a predetermined calculation is performed based on the converted electric charges. A phase difference is calculated and a distance image is generated.
JP 2004-32682 A

  However, in the distance image generation device described in Patent Document 1, since the light amount of the light emission source is constant, there is a possibility of greatly affecting the accuracy of the distance image.

  For example, when the modulated light source is too dark, the ratio of the modulated light from the modulated light source becomes extremely smaller than the background light other than the modulated light source, so that the SN ratio is lowered and the measurement distance error is increased. On the other hand, if the modulated light source is too bright, the charge accumulated in the image sensor is saturated and distance measurement cannot be performed.

  In addition, since light attenuates in inverse proportion to the square of the distance from the light source, the light quantity of the modulated light source is constant when the distance to the object is relatively close, such as indoors, and the measurement distance range is limited. Even so, the measurement results are unlikely to fluctuate greatly. However, when measuring the distance in the outdoors, for example, in front of the vehicle, the objects are roads, landscapes, pedestrians, preceding vehicles, oncoming vehicles, and the like, and their distances change from several meters to infinity. Therefore, it is considered that the optimum light amount of the modulated light source is different between the object at a short distance and the object at a long distance, so it is desirable that the light amount of the modulated light source is dynamically changed.

  Furthermore, even when measuring the distance of an object existing at the same distance, the intensity of the reflected light varies depending on the reflectance of the object, so it is desirable to change the light amount of the modulated light source.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a distance image generation apparatus capable of generating a distance image with higher accuracy.

  In order to solve the above problems, the present invention provides a light source that emits modulated light to a target space, and reflected light that is irradiated from the light source and reflected by an object in the target space. A plurality of photoelectric conversion elements that receive light and convert it into charges, a plurality of charge storage units provided for each of the photoelectric conversion elements, and charges converted by the photoelectric conversion elements in synchronization with modulation of the light emission source An image sensor comprising: means for allocating to the plurality of charge storage units; and a predetermined calculation based on the charges stored in the plurality of charge storage units to generate a distance image in which the pixel value is a distance value A predetermined calculation is performed based on the distance image generation unit, the amplitude of the reflected light received by the photoelectric conversion element, and the pixel value of the distance image generated by the distance image generation unit, and the pixel value correlates with the reflectance. Reflectivity, the reflection coefficient Characterized in that it comprises a and a reflectance image generating unit that generates an image.

  According to the first aspect of the present invention, it is possible to generate a reflectance image that can be used to generate a distance image with higher accuracy.

  The invention according to claim 2 further includes a light amount control unit that controls the light amount of the light emitting source based on the reflectance image generated by the distance image generation unit in the invention according to claim 1. It is characterized by.

  According to the second aspect of the invention, since the light amount of the light source is controlled (increased / decreased) based on the reflectance image by the light amount control unit, it is possible to control the light amount to the optimum amount according to the reflectance of the object. It becomes possible. For this reason, regardless of the reflectance of the object, it is possible to keep the amount of reflected light reflected back from the object optimally. Accordingly, it is possible to generate a distance image with higher accuracy (improvement of distance detection accuracy). Further, since the light amount of the light source is not always constant, it is possible to suppress unnecessary light emission power (especially when the light amount is controlled in a direction in which the light amount decreases).

  In addition, the reflectance of the object in the image being captured can be obtained in real time. Furthermore, it becomes possible to simultaneously generate a reflectance image and a distance image.

  According to a third aspect of the present invention, in the invention according to the second aspect, the light emitting source includes a plurality of light emitters that irradiate modulated light to a target space, and the plurality of light emitters are divided into a plurality of regions. The reflectance image is divided into a plurality of regions respectively corresponding to the plurality of regions, and the light amount control unit is configured to correspond to the region for each of the divided reflectance images. The light quantity of the illuminant existing in the light source is controlled.

  According to the third aspect of the present invention, the light amount control unit optimally controls the light amount of the light source for each region of the divided reflectance image, so that it is possible to generate a distance image with higher accuracy. (Further improvement in distance detection accuracy) Further, since the light amount of the light source is not always constant, it is possible to suppress unnecessary light emission power (especially when the light amount is adjusted in a decreasing direction).

  According to a fourth aspect of the present invention, in the invention according to the third aspect, the light amount control unit includes a region having a reflection coefficient lower than a predetermined threshold in the divided reflectance image region. Increasing the amount of light of the illuminant existing in the corresponding region, and for the region including the reflection coefficient higher than a predetermined threshold among the divided reflectance image regions, the light emission existing in the region corresponding to the region It is characterized by lowering the amount of light of the body.

  This is an example of control by the light quantity control unit.

  According to the invention described in claim 4, regarding the region including the reflection coefficient lower than a predetermined threshold among the divided reflectance image regions, the illuminant existing in the region corresponding to the region is included. Increase the amount of light. As a result, it is possible to increase the reflected light from the object having a low reflectance, and it is possible to prevent or reduce the occurrence of errors in distance measurement.

  According to the fourth aspect of the present invention, the light emission existing in the region corresponding to the region including the reflection coefficient higher than a predetermined threshold among the divided reflectance image regions. Reduce the amount of light in the body. Thereby, it is possible to reduce the reflected light from the object having a high reflectance, and it is possible to prevent or reduce the saturation of the image sensor (the charge storage unit thereof).

  That is, according to the invention described in claim 4, since it is possible to keep the amount of modulated light reflected by the object optimal regardless of the reflectance of the object, a more accurate range image can be generated. (Distance detection accuracy is improved).

  According to the present invention, it is possible to provide a distance image generation device that can generate a distance image with higher accuracy.

[First embodiment]
Hereinafter, a distance image generating apparatus according to a first embodiment of the present invention will be described with reference to the drawings.

[Configuration of Distance Image Generation Device]
First, the configuration of the distance image generation apparatus according to the first embodiment will be described.

  FIG. 1 is a block diagram of a distance image generation apparatus according to the first embodiment.

  As shown in FIG. 1, the distance image generating apparatus 1 of the present embodiment includes an optical time-of-flight distance image sensor 10 (hereinafter referred to as a distance image sensor 10). The distance image sensor 10 includes a light source 11, an image sensor 15, a control unit 16, a distance image generation unit 17, a reflectance image generation unit 18, a light amount control unit 19, and the like.

  The light source 11 is a light emitting source that irradiates light modulated in a target space (for example, infrared light or visible light modulated at high speed with a sine wave or a rectangular wave). Used.

  The imaging element 15 receives a plurality of photoelectric conversion elements (also referred to as pixels) that receive incident light 14 including reflected light that is irradiated from the light source 11 and reflected by an object in the target space, and converts the incident light 14 into charges. Provided with a plurality of charge storage units, and means for distributing the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the light source 11 (all not shown). .

  The control unit 16 is for controlling the light source 11 and the image pickup device 15 in synchronization. The image pickup device 15 rapidly transfers charges converted by the photoelectric conversion elements to a plurality of charge storage units according to the synchronization signal of the control unit 16. Sort out.

  The distance image generation unit 17 performs a predetermined calculation based on the distributed charges, calculates a phase difference with the light emission source, and generates a distance image whose pixel value is a distance value.

The reflectance image generation unit 18 is generated by the amplitude of the reflected light (reflected light emitted from the light source 11 and reflected by the object in the target space) received by the photoelectric conversion element of the imaging device 15 and the distance image generation unit 17. The predetermined value is calculated based on the pixel value (distance value) of the distance image, and the pixel value
More precisely, a reflectance image that is a reflection coefficient correlated with the reflectance is generated.

  The light quantity control unit 19 controls the light quantity of the light source 11 based on the reflectance image generated by the distance image generation unit 17.

[Principle of generating distance image and reflectance image]
Next, the principle of generating a distance image will be described.

  FIG. 2 is a diagram for explaining the principle of generating a distance image. In FIG. 2, the sine wave 21 represents the modulated light of the light source 11, and the sine wave 22 represents the incident light of the image sensor 15. The phase difference φ between the sine wave 21 and the sine wave 22 represents a delay caused by the flight time of light to the object.

  In FIG. 2, one cycle of modulation of the light source 11 is divided into four periods, and the charges are distributed to the four charge storage units. If each period is T1, T2, T3, and T4, and the amount of charge accumulated in each period is C1, C2, C3, and C4, the phase difference φ is expressed by the following equation.

  Since the speed of light is known, the distance to the object can be obtained by obtaining this phase difference φ, and a distance image whose pixel value is a distance value can be generated.

  The charge amount average A used as general image data is expressed by the following equation.

  The amplitude amount B of the modulated light component reflected by the object is expressed by the following equation.

  In general, the modulation frequency of the light source is several tens of MHz, and therefore one period of modulation is about several tens of ns. Therefore, in order to obtain a distance image, a charge accumulation time of several hundred to several hundred thousand cycles is required.

  Furthermore, the distance L is expressed by the following expression from Expression 1.

  Here, c is the speed of light and Fs is the frequency of the modulated light.

  Moreover, generally the reflectance R of a target object is represented by the following formula.

  FIG. 3 shows two objects 31 and 32 arranged at the same distance with the same size and different reflectance. Since the two objects 31 and 32 have the same size and are arranged at the same distance, the luminous fluxes irradiated per unit area of the surfaces of the objects 31 and 32 are equal. However, since the two objects 31 and 32 have different reflectivities, the amounts of reflected light reflected and returned to the image sensor 15 are different. For this reason, the object 32 having a low reflectance appears darker than the object 31.

  FIG. 4 shows the relationship between the light emitted from the light source 11 to the target space and the reflected light reflected from the two objects 31 and 32 shown in FIG. FIG.

  Since the objects 31 and 32 are arranged at the same distance, the light 40 irradiated from the light source 11 to the target space and the two objects 31 and 32 shown in FIG. The phase differences of the reflected lights 41 and 42 returned in the same manner are the same, but the light amounts (amplitudes) are different. Further, the luminous flux irradiated per unit area is inversely proportional to the square of the distance.

  For this reason, if the distance L to the objects 31 and 32 and the amplitude B of the reflected light that is irradiated from the light source 11 and reflected by the two objects 31 and 32 in the target space are used, the reflection of the objects 31 and 32 is performed. It is possible to determine the rate. This reflectance (more precisely, the reflection coefficient correlated with the reflectance) is not an absolute value but a relative value. This reflectance is expressed by the following equation.

[Operation of Distance Image Generating Device 1]
Next, the operation of the distance image generating apparatus 1 having the above configuration will be described with reference to the drawings.

[Reflectance Image Generation Processing in Distance Image Generation Device 1]
FIG. 5 is a flowchart for explaining the reflectance image generation processing in the distance image generation apparatus 1. The following processing is mainly performed by the control unit 16 or a control unit (not shown) provided separately from the control unit 16.

  First, the light source 11 irradiates the target space with modulated light according to the synchronization signal of the control unit 16. The image sensor 15 receives incident light including reflected light irradiated from the light source 11 and reflected by an object in the target space, converts the incident light into charges by the photoelectric conversion element, and converts the incident light according to the synchronization signal of the control unit 16. The charged charges are distributed to a plurality of charge storage portions. The above processing is continued until the light accumulation time ends (elapses) (step S1).

  The distance image generation unit 17 reads the distributed charges (for example, C1, C2, C3, and C4) from the plurality of charge storage units (step S2), and converts the read charges (for example, C1, C2, C3, and C4). Based on this, a predetermined calculation (calculations of the above formulas 1 and 4) is performed to calculate a phase difference φ with respect to the light source 11, and a distance image P1 (one pixel) whose pixel value is a distance value is generated (step S3). . The distance image generation unit 17 executes the processes in steps S2 and S3 for all the pixels. Thereby, a distance image P1 (one frame) whose pixel value is a distance value is generated. The distance image generation unit 17 repeats the above processing.

  On the other hand, the reflectance image generation unit 18 performs a predetermined calculation (the calculation of Equation 3 above) in parallel with the processing of steps S2 and S3 of the distance image generation unit 17, and the image sensor 15 (photoelectric conversion element thereof) receives light. The amplitude B of the reflected light (reflected light emitted from the light source 11 and reflected by the object in the target space. For example, the sine wave 41 or 42 in FIG. 4) is calculated (step S4). The reflectance image generation unit 18 performs a predetermined calculation (the above formula 6) based on the pixel value (distance value) of the distance image generated by the distance image generation unit 17 and the amplitude B calculated in step S4, and the pixel value Is a reflectance image P2 (for one pixel) having a reflectance (more precisely, a reflection coefficient correlated with the reflectance) (step S5). The reflectance image generation unit 18 executes the processes in steps S4 and S5 for all pixels. Thereby, the reflectance image P2 (for one frame) whose pixel value is the reflectance is generated. The reflectance image generation unit 18 repeats the above processing. It should be noted that the ratio of the generation of the distance image and the reflectance image is an appropriate ratio such as the ratio of one reflectance image to one distance image and the ratio of one reflectance image to two distance images. It is possible to generate in proportion.

  As described above, according to the reflectance image generation processing in the distance image generation device 1, it is possible to generate the reflectance image P2 that can be used for generating a distance image with higher accuracy. Become.

[Operation example 1 of the light quantity control unit 19]
Next, processing for controlling the light amount of the light source 11 using the reflectance image P2 generated as described above will be described.

  The light amount control unit 19 controls the light amount of the light source 11 based on the reflectance image P2 generated by the distance image generation unit 17 as described above.

  For example, the light quantity control unit 19 increases the light quantity of the light source 11 when the reflectance image P2 includes an object having a reflectance lower than a predetermined threshold. As a result, it is possible to increase the reflected light from the object having a low reflectance, and it is possible to prevent or reduce the occurrence of errors in distance measurement.

  Moreover, the light quantity control part 19 reduces the light quantity of the light source 11, when the reflectance image P2 contains the target object with a reflectance higher than a predetermined threshold value. Thereby, it is possible to reduce the reflected light from the object having a high reflectance, and it is possible to prevent or reduce the saturation of the image sensor 15 (the charge storage unit thereof).

  That is, according to the operation example 1 of the light quantity control unit 19, since the light quantity control unit 19 controls (increases and decreases) the light quantity of the light source 11 based on the reflectance image P2, the optimal light quantity according to the reflectance of the target object. It becomes possible to control to. For this reason, regardless of the reflectance of the object, it is possible to keep the amount of reflected light reflected back from the object optimally. Accordingly, it is possible to generate a distance image with higher accuracy (improvement of distance detection accuracy). In addition, since the light amount of the light source 11 is not always constant, it is possible to suppress excessive light emission power (particularly when the light amount is adjusted in a direction in which the light amount decreases). In addition, the reflectance of the object in the image being captured can be obtained in real time. Furthermore, it becomes possible to simultaneously generate a reflectance image and a distance image.

[Operation example 2 of the light quantity control unit 19]
In the operation example 1 described above, an example in which the light amount of the entire light source 11 is controlled regardless of whether the number of the light sources 11 is one or plural is described.

  In the second operation example, as illustrated in FIG. 6, the light source 11 includes a plurality of light emitters 11 a, 11 a... That irradiate the target space with modulated light, and the plurality of light emitters 11 a, 11 a. Are divided into a plurality of regions 61 to 69. Further, as shown in FIG. 7, the reflectance image P <b> 2 generated by the distance image generation unit 17 is divided into a plurality of regions 71 to 79 corresponding to the plurality of regions 61 to 69, respectively. The light quantity control unit 19, for each of the regions 71 to 79 of the divided reflectance image P <b> 2, corresponds to the region (any one of the regions 71 to 79) (any one of the regions 61 to 69). The amount of light of the illuminant 11a existing in the region) is controlled.

  FIG. 8 is a flowchart for explaining an operation example 2 of the light quantity control unit 19.

  First, the light quantity control unit 19 acquires a reflectance image of each region 71 to 79 of the reflectance image P2 generated by the distance image generation unit 17 (step S10), and the acquired reflectance image is It is determined whether or not a pixel having a reflectance (reflection coefficient) lower than a predetermined threshold (lower threshold) is included (steps S11 and S12). And when the pixel of the reflectance (reflection coefficient) lower than this threshold value (lower threshold) is included (step S11: Yes, step S12: Yes), the light quantity of the light-emitting body 11a which exists in the area | region corresponding to this area is raised. (Step S13). As a result, it is possible to increase the reflected light from the object having a low reflectance, and it is possible to prevent or reduce the occurrence of errors in distance measurement.

  Moreover, the light quantity control part 19 is based on the threshold value (upper limit threshold value) from which the reflectance image of this area | region (any area | region of the areas 71-79) is predetermined for every area | region 71-79 of the reflectance image P2. It is determined whether or not a pixel having a higher reflectance (reflection coefficient) is included (steps S11 and S14). And when the pixel of reflectance (reflection coefficient) higher than this threshold value (upper limit threshold value) is included (step S11: Yes, step S12: No, step S14: Yes), the light-emitting body which exists in the area | region corresponding to this area | region The amount of light 11a is lowered (step S15). Thereby, it is possible to reduce the reflected light from the object having a high reflectance, and it is possible to prevent or reduce the saturation of the image sensor (the charge storage unit thereof).

  That is, according to the operation example 2 of the light quantity control unit 19, the light quantity control unit 19 controls (increases / decreases) the light quantity of the light source 11 based on the reflectance image P <b> 2. It becomes possible to control to the optimal light quantity according to the reflectance of a target object for every area | regions 61-69 corresponding to. For this reason, regardless of the reflectance of the object, the amount of reflected light reflected and returned by the object can be kept optimal for each of the regions 61 to 69. Accordingly, it is possible to generate a distance image with higher accuracy (further improvement in distance detection accuracy). In addition, since the light amount of the light source 11 is not always constant, it is possible to suppress unnecessary light emission power (particularly when the light amount is adjusted in a decreasing direction). In addition, the reflectance of the object in the image being captured can be obtained in real time. Furthermore, it becomes possible to simultaneously generate a reflectance image and a distance image.

  As described above, according to the distance image generation device 1 of the present embodiment, it is possible to generate a reflectance image that can be used for generating a distance image with higher accuracy.

  Further, according to the distance image generating apparatus 1 of the present embodiment, the light amount control unit 19 controls (increases / decreases) the light amount of the light source 11 based on the reflectance image P2, so that the object (objects 31, 32, etc.) is controlled. It becomes possible to control the light quantity to an optimum amount according to the reflectance. For this reason, regardless of the reflectance of the object, it is possible to keep the amount of reflected light reflected back from the object optimally. Accordingly, it is possible to generate a distance image with higher accuracy (improvement of distance detection accuracy). Further, since the light amount of the light source 11 is not always constant, it is possible to suppress unnecessary light emission power (particularly when the light amount is controlled in a direction in which the light amount decreases).

  Next, a modified example will be described.

  In the above embodiment, an example in which there are nine divided areas has been described, but the present invention is not limited to this. There may be more than nine divided areas or fewer than nine. Further, the number of divisions of the light source 11 and the number of divisions of the reflectance image may be different as long as the correspondence between the division region of the light source 11 and the division region of the image sensor is taken.

  In the above embodiment, the reflectance image P2 (the same applies to the reflectance image P2 of the divided area) is an object having a reflectance lower than a predetermined threshold and an object having a reflectance higher than a predetermined threshold. (That is, the reflectance image P2 (the same applies to the reflectance image P2 of the divided area) includes both objects), for example, the area of the object (number of pixels) is large. It is conceivable that the process is compared with a predetermined threshold (lower threshold or upper threshold) (for example, FIG. 8, steps S11, S12, S14). Alternatively, it is conceivable to perform processing by comparing an average of pixels of both objects with a predetermined threshold value (lower limit value or upper limit value) (for example, steps S11, S12, and S14 in FIG. 8).

  The above embodiment is merely an example in all respects. The present invention is not construed as being limited to these descriptions. The present invention can be implemented in various other forms without departing from the spirit or main features thereof.

1 is a block diagram of a distance image generation apparatus that is a first embodiment of the present invention. FIG. It is a figure for demonstrating the principle which produces | generates a distance image. It is a figure for demonstrating the principle which produces | generates a reflectance image. It is a figure for demonstrating that although a phase difference is the same when a reflectance differs, an amplitude changes. 5 is a flowchart for explaining reflectance image generation processing in the distance image generation device 1; It is an example of the light source 11 used for the operation example 2 of the light quantity control unit 19. It is a figure for demonstrating the image of the reflectance image P2. 5 is a flowchart for explaining an operation example 2 of the light amount control unit 19;

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Distance image production | generation apparatus, 10 ... Optical time-of-flight type distance image sensor (distance image sensor), 11 ... Light source, 11a ... Luminescent body, 14 ... Incident light, 15 ... Image sensor, 16 ... Control part, 17 ... Distance image Generation unit, 18 ... reflectance image generation unit, 19 ... light quantity control unit, 61-69 ... divided area, 71-79 ... photoelectric conversion element area, φ ... phase difference

Claims (4)

A light source that emits modulated light to the target space;
A plurality of photoelectric conversion elements that receive reflected light irradiated from the light emitting source and reflected by an object in a target space and convert it into charges, a plurality of charge storage units provided for each of the photoelectric conversion elements, and the light emission Means for distributing the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the source, and
A distance image generation unit that performs a predetermined calculation based on the charges accumulated in the plurality of charge accumulation units and generates a distance image in which a pixel value is a distance value;
A predetermined calculation is performed based on the amplitude of the reflected light received by the photoelectric conversion element and the pixel value of the distance image generated by the distance image generation unit, and the pixel value is a reflection coefficient that correlates with the reflectance. A reflectance image generation unit for generating a rate image;
A distance image generating apparatus comprising:   The distance image generation apparatus according to claim 1, further comprising a light amount control unit that controls a light amount of the light emission source based on a reflectance image generated by the distance image generation unit. The light emission source includes a plurality of light emitters that irradiate modulated light to a target space,
The plurality of light emitters are divided into a plurality of regions,
The reflectance image is divided into a plurality of regions respectively corresponding to the plurality of regions,
The distance image generation device according to claim 2, wherein the light amount control unit controls the light amount of a light emitting body existing in a region corresponding to the region of the divided reflectance image. The light amount control unit
For the region including the reflection coefficient lower than a predetermined threshold among the divided reflectance image regions, increase the light amount of the illuminant existing in the region corresponding to the region,
4. The light quantity of a light emitter existing in a region corresponding to the region of the divided reflectance image region that includes a reflection coefficient higher than a predetermined threshold is reduced. The distance image generation device described in 1.

Images