JP5431810B2 - Image sensor and light receiving device used for the image sensor - Google Patents

Description

  The present invention relates to an image sensor that acquires image data. The present invention also relates to a light receiving device used for the image sensor.

  Technology has been developed in which light is emitted from a light source toward an object, and image data is acquired based on a difference between the irradiated light and reflected light reflected by the object. Patent Documents 1 to 3 disclose a technique for acquiring distance image data by converting a distance to an object from a time difference between irradiation light irradiated from a light source and reflected light reflected by the object. Patent Document 4 discloses a technique for acquiring distance image data by converting a distance to an object from a phase difference between irradiation light irradiated from a light source and reflected light reflected by the object. Patent Document 5 discloses a technique for acquiring part discrimination image data by irradiating irradiation light of different wavelengths from a light source and estimating the part of the object from the reflection characteristics of reflected light from the object of each wavelength. Is disclosed.

JP 2004-294420 A JP 2005-235893 A JP 2005-291985 A Special Table 2000-517427 JP 2006-242909 A

  When both the distance image data and the part determination image data are required, if a device for acquiring the distance image data and a device for acquiring the part determination image data are respectively provided, the distance image data and the part determination image data are provided. It becomes possible to acquire both. However, providing each device separately is not preferable in terms of an increase in installation space and an increase in cost due to an increase in the number of components. An object of this invention is to provide the image sensor which can acquire both distance image data and site | part discrimination | determination image data, and the light-receiving device used for the image sensor.

  The technology disclosed in this specification can acquire both distance image data and part determination image data by sharing an irradiation apparatus and an imaging apparatus used to acquire distance image data and part determination image data. An image sensor and a light receiving device for the image sensor are constructed. The technology disclosed in this specification has succeeded in simplifying the image sensor and the light receiving device for the image sensor by sharing the irradiation device and the imaging device. According to the technology disclosed in the present specification, it is possible to acquire both distance image data and site determination image data by using a simplified image sensor and a light receiving device for the image sensor.

  The technology disclosed in the present specification is embodied in an image sensor that acquires distance image data and site determination image data. The image sensor includes an irradiation device, an imaging device, and an arithmetic device. The irradiation apparatus irradiates the target with first wavelength light having a first wavelength and second wavelength light having a second wavelength different from the first wavelength. The imaging apparatus accumulates a charge amount according to the first reflected light of the first wavelength light reflected by the object and accumulates a charge amount according to the second reflected light of the second wavelength light reflected by the object. . The computing device computes the distance image data and the part determination image data based on the charge amount accumulated in the imaging device. Here, the arithmetic device uses at least the first wavelength light to calculate the distance image data based on the distance until the first wavelength light is reflected from the irradiation device by the object and reaches the imaging device. If necessary, the computing device computes distance image data based on the distance from the irradiation device to the imaging device after the second wavelength light is reflected by the object using the second wavelength light. Also good. Furthermore, the calculation device calculates the part determination image data based on the reflection characteristic of the first reflected light on the object and the reflection characteristic of the second reflected light on the object. In the image sensor, in order to calculate the information necessary for calculating the distance image data and the part determination image data by using the shared irradiation device and controlling the charge accumulation timing in the shared imaging device. Both necessary information can be acquired. By sharing the irradiation apparatus and the imaging apparatus, a simplified image sensor can be realized.

In the image sensor disclosed in the present specification, the imaging device has a first charge accumulation unit that accumulates a charge amount corresponding to the first reflected light in the first time, and a charge corresponding to the first reflected light in the second time. a second charge accumulation portion that accumulates the amount, that have at least a third charge storage section for storing a quantity of electric charge corresponding to the second reflected light in the third hour. Further, the arithmetic device calculates the first distance image data based on the charge amount accumulated in the first charge accumulation unit and the charge amount accumulated in the second charge accumulation unit, and is accumulated in the first charge accumulation unit. The region determination image data is calculated based on the obtained charge amount and the charge amount stored in the third charge storage unit. The first time and the second time are different. The first time and the third time may be the same time or different. The second time and the third time may be the same time or different.
Here, “time” refers to a period from a certain start time to another end time. “Time is different” means that at least one of the start time and the end time is different. “Same time” means that both the start time and the end time coincide. In addition, the “time” here includes not only continuous items but also intermittent items.
In the image sensor, the charge amount accumulated in the first charge accumulation unit of the imaging device is used as information for calculating the first distance image data, and also as information for calculating the part determination image data. Used. For this reason, the first charge storage unit of the imaging device is used for acquiring the first distance image data and also for acquiring the part determination image data. By sharing the first charge storage unit of the imaging device, the image sensor and the light receiving device for the image sensor are simplified. In addition, since the charge amount stored in the first charge storage unit is used as information for calculating the first distance image data and the part determination image data, charges are stored for the calculation of the distance image data and the part determination image data, respectively. Compared to the case, the time required for obtaining the first distance image data and the part determination image data is shortened.

In the image sensor disclosed in this specification, it is preferable that the imaging device further includes a fourth charge accumulation unit that accumulates a charge amount according to the second reflected light in the fourth time. In this case, the calculation device calculates the second distance image data based on the charge amount accumulated in the third charge accumulation unit and the charge amount accumulated in the fourth charge accumulation unit. The third time and the fourth time are different. The first time and the fourth time may be the same time or different. The second time and the fourth time may be the same time or different.
In the image sensor, the first distance image data is acquired using the first wavelength light, and the second distance image data is also acquired using the second wavelength light. For example, by selecting one of the first distance image data obtained using the first wavelength light and the second distance image data obtained using the second wavelength light according to the situation, more accurate Distance image data can be obtained. Alternatively, more accurate distance image data can be obtained by synthesizing the first distance image data obtained using the first wavelength light and the second distance image data obtained using the second wavelength light. Is possible.

When obtaining both the first distance image data of the first wavelength light and the second distance image data of the second wavelength light, one of the first distance image data and the second distance image data is present at a short distance. It is preferable that the other is used for a distance image of an object and the other is used for a distance image of an object existing at a long distance.
The time difference and phase difference between the irradiated light and the reflected light vary depending on the distance to the object. When the distance to the object is short (in the case of a short distance), the time difference and the phase difference between the irradiation light and the reflected light are small. On the other hand, when the distance to the object is long (in the case of a long distance), the time difference and the phase difference between the irradiation light and the reflected light are large. For this reason, in order to acquire the distance image data of the object existing at a short distance, the charge accumulation timing corresponding to the time difference and the phase difference is desirable, and the distance image data of the object existing at a long distance is acquired. For this purpose, charge accumulation timing according to the time difference and phase difference is desirable.
In the above-described image sensor, it is possible to acquire highly accurate and accurate distance image data by properly using data used for calculation according to the distance to the object.

In the image sensor disclosed in this specification, the imaging device receives a reflected light through an optical filter that selectively transmits a first wavelength, and converts the reflected light into an electric charge. It is preferable to have a second light receiving element that receives reflected light through an optical filter that selectively transmits two wavelengths and converts the reflected light into electric charges. The first charge storage unit and the second charge storage unit store the charges converted by the first light receiving element. The third charge storage unit stores the charge converted by the second light receiving element.
The image sensor disclosed in the present specification uses first wavelength light and second wavelength light having different wavelengths. For this reason, when the optical filter for selective transmission is not used, in order to prevent the two first wavelength lights received by the light receiving element and the second wavelength light from interfering with each other, the timing of irradiation with the first wavelength light and the first It is necessary to greatly shift the timing of irradiating the two-wavelength light.
On the other hand, when the optical filter is used, it is not necessary to shift the timing of irradiating the first wavelength light and the timing of irradiating the second wavelength light. As a result, the image sensor can acquire image data in a shorter time.

  According to the technique disclosed in this specification, it is possible to acquire both distance image data and part determination image data using a simplified light receiving device. As a result, a more useful image can be synthesized using both the distance image data and the part determination image data.

The outline of a structure of an image sensor is shown. 1 shows an outline of a configuration of an imaging apparatus. An example of the outline of a structure of a light-receiving part is shown. An example of the data memorize | stored in the database is shown. The timing chart in a light source and a charge storage part in the case of acquiring distance image data from a time difference is shown. An explanatory view of a method of calculating distance image data from a time difference is shown. An example of a timing chart in the light source and the charge storage unit when distance image data is acquired from the phase difference is shown. An explanatory view of a method of calculating distance image data from a phase difference is shown. The other example of the outline of a structure of a light-receiving part is shown. The other example of the timing chart in a light source and a charge storage part in the case of acquiring distance image data from a phase difference is shown. The other example of the outline of a structure of a light-receiving part is shown. The other example of the timing chart in a light source and a charge storage part in the case of acquiring distance image data from a phase difference is shown.

First, the techniques disclosed in this specification will be organized.
(First Feature) The irradiation device has a first light source and a second light source. The first light source emits first wavelength light having a first wavelength. The second light source emits second wavelength light having a second wavelength different from the first wavelength.
(Second Feature) The first light source emits pulse-modulated first wavelength light. The second light source emits pulse-modulated second wavelength light.
(Third Feature) The imaging device has a plurality of light receiving units arranged in an array. The light receiving unit includes a light receiving element, a charge storage unit, and a switching element that connects the light receiving element and the charge storage unit.
(Fourth Feature) The switching element between the light receiving element and the charge storage unit is controlled in conjunction with the timing at which the first wavelength light or the second wavelength light is emitted.
(Fifth Feature) The switching element between the light receiving element and the charge storage unit is controlled by a pulse-modulated gate signal in synchronization with the timing at which the first wavelength light or the second wavelength light is emitted.

  Hereinafter, an in-vehicle image sensor will be described with reference to the drawings. In particular, the image sensor described in the following embodiments is used to acquire image data relating to the head of the driver of the vehicle. As shown in FIG. 1, the image sensor 100 includes an irradiation device 20, a light receiving device 80, and a control device 30 that controls the irradiation device 20 and the light receiving device 80. The irradiation device 20, the light receiving device 80, and the control device 30 may be configured as separate devices, or may be configured as an integrated module.

  The irradiation device 20 irradiates the first light source 22 that irradiates the first wavelength light in the first wavelength range toward the surface of the measurement object, and irradiates the second wavelength light in the second wavelength range toward the surface of the measurement object. The second light source 24 is provided. The first light source 22 and the second light source 24 include, for example, LEDs (Light Emitting Diodes). It is desirable that the first light source 22 and the second light source 24 irradiate light in the near infrared region of 800 to 1100 nm. In the present embodiment, the first light source 22 emits the first wavelength light of 870 nm, and the second light source 24 emits the second wavelength light of 970 nm.

  The light receiving device 80 includes an imaging device 40, a storage device 50, an arithmetic device 60, and a database 70. The imaging device 40 accumulates a charge amount according to the first reflected light of the first wavelength light reflected by the measurement object, and charges according to the second reflected light of the second wavelength light reflected by the measurement object. Accumulate. The storage device 50 stores a voltage value corresponding to the amount of charge accumulated in the imaging device 40. The computing device 60 computes the distance image data and the part determination image data using the data stored in the storage device 50 and the database 70. The database 70 stores the reflection characteristic of the first wavelength light at the specific part and the reflection characteristic of the second wavelength light at the specific part.

  FIG. 2 schematically shows the configuration of the imaging device 40. The imaging device 40 includes a plurality of light receiving units 42 arranged in a two-dimensional shape (array shape). The light receiving unit 42 is provided for each pixel. Each of the light receiving units 42 includes a light receiving element (photoelectric conversion element) that converts light into electric charges, and a charge accumulation unit that accumulates charges converted by the light receiving elements. A photodiode is typically used as the light receiving element. A capacitor is typically used for the charge storage unit.

  FIG. 3 shows an example of the light receiving unit 42. The light receiving unit 42 includes four light receiving elements 44a, 44b, 44c, 44d and four charge storage units 46a, 46b, 46c, 46d. The first light receiving element 44a is connected to the first charge storage unit 46a, and the charges converted by the first light receiving element 44a are stored in the first charge storage unit 46a. The second light receiving element 44b is connected to the second charge storage unit 46b, and the charge converted by the second light receiving element 44b is stored in the second charge storage unit 46b. The third light receiving element 44c is connected to the third charge storage unit 46c, and the charge converted by the third light receiving element 44c is stored in the third charge storage unit 46c. The fourth light receiving element 44d is connected to the fourth charge storage unit 46d, and the charge converted by the fourth light receiving element 44d is stored in the fourth charge storage unit 46d.

  The storage device 50 includes a voltage conversion circuit that converts a charge amount stored in the charge storage units 46a, 46b, 46c, and 46d of the light receiving unit 42 into a voltage, and a storage circuit that stores the converted voltage value. The storage circuit of the storage device 50 has a plurality of memories, and stores voltage values corresponding to the respective charge amounts of the charge storage units 46a, 46b, 46c, and 46d of each light receiving unit 42.

  The computing device 60 computes the distance image data based on the voltage value stored in the storage device 50. The computing device 60 further computes part determination image data based on the voltage value stored in the storage device 50 and the reflection characteristics stored in the database 70.

  The database 70 stores data relating to light reflection characteristics. The image sensor 100 of this embodiment is for in-vehicle use, and the measurement target is the driver's head. For this reason, the database 70 stores data relating to the light reflection characteristics at each part of the head. FIG. 4 shows an example of data stored in the database 70. FIG. 4 shows the spectral reflectance of the skin and the spectral reflectance of the hair. A method for calculating the region discrimination image data using the spectral reflectance of the skin and the spectral reflectance of the hair will be described later.

  Next, an example of a method for acquiring the distance image data and the part determination image data by the image sensor 100 of the present embodiment will be described. An example of a method for acquiring distance image data will be described with reference to FIGS. The example shown in FIGS. 5 and 6 is an example of acquiring distance image data by converting the distance to the measurement object from the time difference between the irradiation light irradiated from the irradiation device 20 and the reflected light reflected by the measurement object. is there.

  As shown in FIG. 5, the first light source 22 and the second light source 24 alternately emit light, and irradiate the measurement object alternately with the first wavelength light (870 nm) and the second wavelength light (970 nm). . The control device 30 controls the first light source 22 to irradiate the pulse-modulated first wavelength light (870 nm) toward the measurement object, and controls the second light source 24 to perform the pulse-modulated second wavelength. Light (970 nm) is irradiated toward the measurement object. As shown in FIG. 5, when the first light source 22 is on, the second light source 24 is off, and when the second light source 24 is on, the first light source 22 is off. In addition, a period in which the first light source 22 and the second light source 24 are simultaneously turned off is also provided. In this example, both the first wavelength light (870 nm) of the first light source 22 and the second wavelength light (970 nm) of the second light source 24 have the same pulse period. Note that the type of pulse modulation used is not particularly limited, but in this embodiment, pulse width modulation (PWM) is used.

  The control device 30 is connected to the imaging device 40 and accumulates charges in which charge storage unit of the imaging device 40 in synchronization (synchronization) with the timing at which the first light source 22 and the second light source 24 emit light. Also control. Specifically, in the case of the light receiving unit 42 shown in FIG. 3, switching elements (typically field effect type) are respectively provided between the light receiving elements 44a, 44b, 44c, 44d and the charge storage units 46a, 46b, 46c, 46d. The transistor 30 is connected, and the control device 30 controls the timing for turning on and off the switching elements in conjunction with the timing at which the first light source 22 and the second light source 24 emit light. Therefore, the control device 30 controls on / off of the switching elements between the light receiving elements 44a, 44b, 44c, 44d and the charge storage units 46a, 46b, 46c, 46d using the pulse-modulated gate signal. . As shown in FIG. 5, the control device 30 includes an accumulation timing 1 (phase 0 °) for turning on the switching element in accordance with a timing at which the first light source 22 and the second light source 24 emit light, and the first light source 22 and the first light source 22. There are two types of accumulation timing 2 (phase 180 °) in which the phase is shifted by 180 ° from the timing at which the two light sources 24 emit light and the switching element is turned on. As shown in FIG. 5, the switching elements are turned on and off at the accumulation timing 1 and the accumulation timing 2 repeatedly over a predetermined period.

  The case of the charge storage units 46a, 46b, 46c, and 46d shown in FIG. 3 will be specifically described. At time (1) shown in FIG. 5, the switching element between the first light receiving element 44a and the first charge storage unit 46a in FIG. 3 is turned on, and the charge converted by the first light receiving element 44a is the first. Accumulated in the charge accumulating portion 46a. At time (2) shown in FIG. 5, the switching element between the second light receiving element 44b and the second charge storage section 46b in FIG. 3 is turned on, and the charge converted by the second light receiving element 44b is second. The charge is accumulated in the charge accumulation unit 46b. At time (3) shown in FIG. 5, the switching element between the third light receiving element 44c and the third charge storage unit 46c in FIG. 3 is turned on, and the charge converted by the third light receiving element 44c is third. The charge is accumulated in the charge accumulation unit 46c. At time (4) shown in FIG. 5, the switching element between the fourth light receiving element 44d and the fourth charge storage section 46d in FIG. 3 is turned on, and the charge converted by the fourth light receiving element 44d is the fourth. It is stored in the charge storage section 46d.

  As shown in FIG. 5, the first light source 22 and the second light source 24 emit light repeatedly over a predetermined period, and are converted into four charge storage units 46a, 46b, 46c, and 46d over the predetermined period. Accumulated charge is accumulated.

  FIG. 6 shows a calculation method for acquiring distance image data using a time difference. When the measurement object is irradiated with irradiation light whose light emission period is W, the reflected light reflected by the measurement object is received by the light receiving element with a time difference τ. A light amount E1 corresponding to the reflected light is generated in the light receiving element that has received the reflected light at the accumulation timing 1, and the charge amount E1 is accumulated in the charge accumulation unit connected at the accumulation timing 1. In the light receiving element that receives the reflected light at the accumulation timing 2, a charge amount E2 corresponding to the reflected light is generated, and the charge amount E2 is accumulated in the charge accumulation unit connected at the accumulation timing 2. The time difference τ is obtained by the following formula. Here, d is the distance to the measurement object, and c is the speed of light.

  That is, the voltage value (corresponding to time (1) in FIG. 5) stored in the storage device 50 and the second charge storage unit 46b are stored based on the amount of charge stored in the first charge storage unit 46a. The distance image data can be calculated based on the voltage value (corresponding to the time (2) in FIG. 5) stored in the storage device 50 based on the amount of charge stored. This distance image data is distance image data acquired based on the first wavelength light (870 nm). Similarly, a voltage value (corresponding to time (3) in FIG. 5) stored in the storage device 50 based on the amount of charge stored in the third charge storage unit 46c and stored in the fourth charge storage unit 46d. The distance image data can be calculated based on the voltage value (corresponding to the time (4) in FIG. 5) stored in the storage device 50 based on the amount of charge that has been stored. This distance image data is distance image data acquired based on the second wavelength light (970 nm).

  For the distance image data based on the first wavelength light and the distance image data based on the second wavelength light, for example, an average value thereof may be used as the true distance image data. Alternatively, one of distance image data based on the first wavelength light and distance image data based on the second wavelength light may be used as necessary.

  With reference to FIGS. 7 and 8, another example of a method for acquiring distance image data will be described. The examples shown in FIGS. 7 and 8 are examples in which distance image data is acquired by converting the distance to the measurement object from the phase difference between the irradiation light emitted from the light source and the reflected light reflected by the measurement object. .

  As shown in FIG. 7, in this example, there are four types of timings at which the switching elements of the imaging device 40 are turned on in synchronization (synchronization) with the timings at which the first light source 22 and the second light source 24 emit light. It is said. As shown in FIG. 7, the control device 30 matches the timing at which the first light source 22 and the second light source 24 are turned on, the accumulation timing 1 (phase 0 °) for turning on the switching element, and the first light source 22 and the first light source 22. Switching is performed with the phase shifted by 90 ° from the timing when the first light source 22 and the second light source 24 emit light, and the accumulation timing 2 (phase 180 °) when the switching element is turned on with the phase shifted 180 ° from the timing when the two light sources 24 emit light There are four types, timing 3 when the element is turned on (phase 90 °) and timing 4 when the phase is shifted by 270 ° from the timing when the first light source 22 and the second light source 24 emit light (phase 270 °). To do. As shown in FIG. 7, after the switching elements are turned on / off at the accumulation timing 1 and the accumulation timing 2 for a predetermined period (T1), the switching elements are turned on / off at the accumulation timing 3 and the accumulation timing 4. It repeats over a predetermined period (T2).

  The case of the charge storage units 46a, 46b, 46c, and 46d shown in FIG. 3 will be described as an example. First, in the case of the predetermined period (T1), at the time (1) shown in FIG. 7, the switching element between the first light receiving element 44a and the first charge accumulation unit 46a in FIG. The charges converted by the element 44a are accumulated in the first charge accumulation unit 46a. At time (2) shown in FIG. 7, the switching element between the second light receiving element 44b and the second charge storage section 46b in FIG. 3 is turned on, and the charge converted by the second light receiving element 44b is second. The charge is accumulated in the charge accumulation unit 46b. At time (3) shown in FIG. 7, the switching element between the third light receiving element 44c and the third charge storage unit 46c in FIG. 3 is turned on, and the charge converted by the third light receiving element 44c is third. The charge is accumulated in the charge accumulation unit 46c. At time (4) shown in FIG. 7, the switching element between the fourth light receiving element 44d and the fourth charge storage section 46d in FIG. 3 is turned on, and the charge converted by the fourth light receiving element 44d is the fourth. It is stored in the charge storage section 46d. In the four charge storage units 46a, 46b, 46c, and 46d, charges converted over a predetermined period (T1) are accumulated. When the predetermined period (T1) ends, the amount of charge accumulated in the charge storage units 46a, 46b, 46c, and 46d is converted into a voltage value and stored in the storage device 50. Furthermore, when the predetermined period (T1) ends, the charges accumulated in the charge accumulation units 46a, 46b, 46c, and 46d are erased.

  Next, when the period shifts to the predetermined period (T2), at the time (5) shown in FIG. 7, the switching element between the first light receiving element 44a and the first charge storage unit 46a in FIG. The charges converted by the one light receiving element 44a are accumulated in the first charge accumulation unit 46a. At time (6) shown in FIG. 7, the switching element between the second light receiving element 44b and the second charge storage section 46b in FIG. 3 is turned on, and the charge converted by the second light receiving element 44b is second. The charge is accumulated in the charge accumulation unit 46b. At time (7) shown in FIG. 7, the switching element between the third light receiving element 44c and the third charge storage section 46c in FIG. 3 is turned on, and the charge converted by the third light receiving element 44c is third. The charge is accumulated in the charge accumulation unit 46c. At time (8) shown in FIG. 7, the switching element between the fourth light receiving element 44d and the fourth charge storage section 46d in FIG. 3 is turned on, and the charge converted by the fourth light receiving element 44d is the fourth. It is stored in the charge storage section 46d. In the four charge storage units 46a, 46b, 46c, and 46d, charges converted over a predetermined period (T2) are accumulated. When the predetermined period (T2) ends, the amount of charge accumulated in the charge storage units 46a, 46b, 46c, and 46d is converted into a voltage value and stored in the storage device 50. Furthermore, when the predetermined period (T2) ends, the charges accumulated in the charge accumulation units 46a, 46b, 46c, and 46d are erased.

  FIG. 8 shows a calculation method for acquiring distance image data using a phase difference. When irradiation light whose light emission period is W is irradiated onto the measurement object, the reflected light reflected by the measurement object is received by the light receiving element with a delay of the phase difference φ. A light amount E1 corresponding to the reflected light is generated in the light receiving element that has received the reflected light at the accumulation timing 1, and the charge amount E1 is accumulated in the charge accumulation unit connected at the accumulation timing 1. In the light receiving element that receives the reflected light at the accumulation timing 3, a charge amount E2 corresponding to the reflected light is generated, and the charge amount E2 is accumulated in the charge accumulation unit connected at the accumulation timing 3. A light amount E3 corresponding to the reflected light is generated in the light receiving element that has received the reflected light at the accumulation timing 2, and the charge amount E3 is accumulated in the charge accumulation unit connected at the accumulation timing 2. A light amount E4 corresponding to the reflected light is generated in the light receiving element that receives the reflected light at the accumulation timing 4, and the charge amount E4 is accumulated in the charge accumulation unit connected at the accumulation timing 4. The phase difference φ is obtained by the following equation. Here, d is the distance to the measurement object, c is the speed of light, and f is the repetition frequency.

  That is, the voltage value (corresponding to time (1) in FIG. 7) and the second charge stored in the storage device 50 based on the amount of charge accumulated in the first charge accumulation unit 46a in the predetermined period (T1). The voltage value (corresponding to time (2) in FIG. 7) stored in the storage device 50 based on the amount of charge stored in the storage unit 46b and the first charge storage unit 46a in a predetermined period (T2). The storage device based on the voltage value (corresponding to the time (5) in FIG. 7) stored in the storage device 50 based on the accumulated charge amount and the charge amount accumulated in the second charge accumulation unit 46b. Based on the voltage value stored in 50 (corresponding to time (6) in FIG. 7), the distance image data can be calculated. This distance image data is distance image data acquired based on the first wavelength light (870 nm). Similarly, the voltage value (corresponding to time (3) in FIG. 7) stored in the storage device 50 based on the charge amount stored in the third charge storage section 46c in the predetermined period (T1) and the fourth The voltage value (corresponding to time (4) in FIG. 7) stored in the storage device 50 based on the amount of charge accumulated in the charge accumulation unit 46d and the third charge accumulation unit 46c in a predetermined period (T2). Is stored based on the voltage value stored in the storage device 50 (corresponding to the time (7) in FIG. 7) and the charge stored in the fourth charge storage section 46d. The distance image data can be calculated based on the voltage value stored in the device 50 (corresponding to the time (8) in FIG. 7). This distance image data is distance image data acquired based on the second wavelength light (970 nm).

  For the distance image data based on the first wavelength light and the distance image data based on the second wavelength light, for example, an average value thereof may be used as the true distance image data. Alternatively, one of distance image data based on the first wavelength light and distance image data based on the second wavelength light may be used as necessary.

  Next, an example of a method for acquiring the part determination image data will be described with reference to FIG. As described above, the database 70 stores data relating to the spectral reflectance of the skin and the spectral reflectance of the hair. As shown in FIG. 4, regarding the spectral reflectance of the skin and the spectral reflectance of the hair, the following relationship exists when the spectral reflectances at wavelengths of 870 nm and 970 nm are compared.

-Skin: reflectance at 870 nm> reflectance at 970 nm-Hair: reflectance at 870 nm <reflectance at 970 nm

  For example, when the switching element of the imaging device 40 is operated at the accumulation timing shown in FIG. 5, the voltage value stored in the storage device 50 based on the amount of charge accumulated in the first charge accumulation unit 46 a (FIG. 5). (Which reflects the reflectance of the first wavelength light 870 nm) and the amount of charge stored in the third charge storage unit 46c, the voltage value stored in the storage device 50 Based on (corresponding to the time (3) in FIG. 5 and reflecting the reflectance of the second wavelength light of 970 nm), the region determination image data can be calculated. If necessary, the voltage value stored in the storage device 50 based on the amount of charge stored in the second charge storage section 46b (corresponding to the time (2) in FIG. 5), the first wavelength light 870 nm. Corresponding to the voltage value (time (4) in FIG. 5) stored in the storage device 50 based on the amount of charge stored in the fourth charge storage section 46d and the second charge storage section 46d. Based on the reflectance of the wavelength light of 970 nm, the region determination image data can be calculated.

  Alternatively, when the switching element of the imaging device 40 is operated at the accumulation timing illustrated in FIG. 7, the image is stored in the storage device 50 based on the amount of charge accumulated in the first charge accumulation unit 46a in the predetermined period (T1). Is stored in the storage device 50 based on the voltage value (corresponding to the time (1) in FIG. 7 and reflecting the reflectance of the first wavelength light 870 nm) and the amount of charge accumulated in the third charge accumulation unit 46c. The region determination image data can be calculated based on the stored voltage value (corresponding to the time (3) in FIG. 7 and reflecting the reflectance of the second wavelength light of 970 nm). If necessary, part determination based on voltage values stored corresponding to time (2) and (4), time (5) and (7), and / or time (6) and (8) Image data can also be calculated.

  Thus, it is possible to discriminate the skin and hair of the head region by extracting the driver's head region as a measurement object and comparing the values for each pixel at each wavelength of 870 nm and 970 nm. Become.

  In this embodiment, 870 nm is used for the first wavelength light and 970 nm is used for the second wavelength light. However, light in other wavelength ranges may be used. Preferably, the following conditions are satisfied.

  First, it is desirable to use light in the near infrared region. The reason for this is that when the driver uses it while driving, the driver's face is illuminated with invisible light so as not to distract the driver. This is because it is not desirable.

  Second, it is desirable to use light having a wavelength range of 1100 nm or less. This is because the light receiving element formed of a silicon semiconductor that is excellent in cost, reliability, and responsiveness has high sensitivity to a wavelength of 1100 nm or less.

  Third, it is desirable that the reflectances in the two wavelength regions are different for each of skin, hair, and the like. In addition, it is necessary to consider that when an irradiation apparatus that emits light having a wavelength in the vicinity of the visible region is used, light may leak into the visible region, and that the sensitivity of the light receiving element is hardly present at 1100 nm or more.

  As described above, the image sensor 100 according to the present embodiment can acquire the distance image data and the part determination image data by using the irradiation device 20 and the imaging device 40 that are shared. In the case of the accumulation timing shown in FIG. 5, the voltage value (corresponding to the time (1) in FIG. 5) stored in the storage device 50 based on the amount of charge accumulated in the first charge accumulation unit 46a. This is used for the calculation of both the distance image data and the part determination image data. In the case of the accumulation timing shown in FIG. 7, the voltage value (time (1 in FIG. 7) stored in the storage device 50 based on the amount of charge accumulated in the first charge accumulation unit 46a in the predetermined period (T1). ) Corresponding to () is used for the calculation of both the distance image data and the part determination image data. For this reason, the first charge storage unit 46a is used to acquire distance image data and is also used to acquire part determination image data. By sharing the first charge storage unit 46a of the imaging device 40, the image sensor 100 is simplified. Further, since the charge amount stored in the first charge storage section 46a is used as information for calculating the distance image data and the part determination image data, charges are stored for the calculation of the distance image data and the part determination image data, respectively. Compared with the case where it carries out, the time required for acquisition of distance image data and site | part discrimination | determination image data is shortened.

  FIG. 9 shows another example of the light receiving unit 42 provided in the imaging device 40. The light receiving unit 42 shown in FIG. 9 includes one light receiving element 44 and four charge storage units 46a, 46b, 46c, and 46d. Each of the four charge storage units 46 a, 46 b, 46 c, 46 d is connected to the light receiving element 44. Switching elements are provided between the four charge storage units 46a, 46b, 46c, 46d and the light receiving element 44, respectively. Compared with the light receiving unit 42 shown in FIG. 3, the number of light receiving elements is greatly reduced, so that a more simplified pressure sensor can be realized.

  FIG. 10 shows another example in which the pulse cycle for causing the first light source 22 and the second light source 24 to emit light is changed. This modification is a modification of the distance image data acquisition example based on the phase difference φ shown in FIG. The following technique can also be applied to an example of obtaining distance image data based on the time difference τ shown in FIG.

  As shown in FIG. 10, the second light source 24 is controlled such that the pulse period is modulated twice that of the first light source 22. In conjunction with this, the period of times (3) and (4) for accumulating charges in the third charge accumulation section 46c and the fourth charge accumulation section 46d in the predetermined period (T1) is also adjusted to double. Similarly, the period of times (7) and (8) for accumulating charges in the third charge accumulation unit 46c and the fourth charge accumulation unit 46d in the predetermined period (T2) is also adjusted to double.

  In this example, the distance image data based on the first wavelength light (870 nm) (obtained based on the voltage values stored corresponding to the times (1), (2), (5), and (6)) is obtained. , Used for a distance image of a measurement object existing at a short distance. On the other hand, the distance image data based on the second wavelength light (970 nm) (obtained based on the voltage values stored corresponding to the times (3), (4), (7), and (8)) It is used for a distance image of a measurement object existing at a distance. That is, the first wavelength light (870 nm) modulated with a short pulse period is used to obtain a distance image of a measurement object existing at a short distance, and the second wavelength light (970 nm) modulated with a long pulse period. Is used to acquire a distance image of a measurement object existing at a long distance.

  The time difference and the phase difference between the irradiation light and the reflected light vary according to the distance to the measurement object. When the distance to the measurement object is short (in the case of a short distance), the time difference and phase difference between the irradiation light and the reflected light are small. On the other hand, when the distance to the measurement object is long (in the case of a long distance), the time difference and the phase difference between the irradiation light and the reflected light are large. For this reason, for example, if it is attempted to acquire the distance image data of the measurement object existing at a long distance using the first wavelength light (870 nm) shown in FIG. 10, the reflected light may not be received. There may be situations where charge cannot be stored. Further, when it is attempted to acquire distance image data of a measurement object existing at a short distance using the second wavelength light (970 nm) shown in FIG. 10, the difference in the amount of charge in each phase becomes small, and the accurate distance. There may be a situation where image data cannot be acquired. Therefore, it is desirable to use the first wavelength light (870 nm) modulated with a short pulse period in order to obtain a distance image of the measurement object existing at a short distance, and the second wavelength light modulated with a long pulse period. (970 nm) is preferably used for acquiring a distance image of a measurement object existing at a long distance. As described above, it is possible to acquire accurate and accurate distance image data by properly using the data used for the calculation according to the distance to the measurement object.

  FIG. 11 shows another example of the light receiving unit 42 provided in the imaging device 40. The light receiving unit 42 shown in FIG. 11 includes two light receiving elements 44A and 44B and four charge storage units 46a, 46b, 46c, and 46d. The first charge accumulation unit 46a and the second charge accumulation unit 46b are each connected to the first light receiving element 44A. The third charge accumulation unit 46c and the fourth charge accumulation unit 46d are each connected to the second light receiving element 44B. Switching elements are provided between the first charge storage unit 46a and the first light receiving element 44A, and between the second charge storage unit 46b and the first light receiving element 44A, respectively. Switching elements are also provided between the third charge storage section 46c and the second light receiving element 44B, and between the fourth charge storage section 46d and the second light receiving element 44B.

  The light receiving unit 42 further includes two band pass filters 48A and 48B. The first band pass filter 48A covers the first light receiving element 44A, and the second band pass filter 48B covers the second light receiving element 44B. The first bandpass filter 48A has transmission selectivity with respect to the first wavelength light (870 nm), and the second bandpass filter 48B has transmission selectivity with respect to the second wavelength light (970 nm). Yes. For this reason, the first light receiving element 44A receives only the first wavelength light. The second light receiving element 44B receives only the second wavelength light.

  FIG. 12 shows an example of acquiring distance image data based on the phase difference φ using the light receiving unit 42 of FIG. 11. The following technique can also be applied to an example of obtaining distance image data based on the time difference τ shown in FIG. As can be seen from a comparison with the example of FIG. 7, the first light source 22 and the second light source 24 repeatedly emit light simultaneously. In the case of the predetermined period (T1), at the time (1) shown in FIG. 12, the switching element between the first light receiving element 44A and the first charge accumulation unit 46a in FIG. 11 is turned on, and the first light receiving element 44A. The charges converted in step 1 are stored in the first charge storage section 46a. At time (2) shown in FIG. 11, the switching element between the first light receiving element 44A and the second charge storage unit 46b in FIG. 11 is turned on, and the charge converted by the first light receiving element 44A is second. The charge is accumulated in the charge accumulation unit 46b. At time (3) shown in FIG. 12, the switching element between the second light receiving element 44B and the third charge storage unit 46c in FIG. 11 is turned on, and the charge converted by the second light receiving element 44B is third. The charge is accumulated in the charge accumulation unit 46c. At time (4) shown in FIG. 12, the switching element between the second light receiving element 44B and the fourth charge accumulation unit 46d in FIG. 11 is turned on, and the charge converted by the second light receiving element 44B is the fourth. It is stored in the charge storage section 46d. In the four charge storage units 46a, 46b, 46c, and 46d, charges converted over a predetermined period (T1) are accumulated. When the predetermined period (T1) ends, the amount of charge accumulated in the charge storage units 46a, 46b, 46c, and 46d is converted into a voltage value and stored in the storage device 50. Furthermore, when the predetermined period (T1) ends, the charges accumulated in the charge accumulation units 46a, 46b, 46c, and 46d are erased.

  Next, when the period shifts to the predetermined period (T2), at the time (5) shown in FIG. 12, the switching element between the first light receiving element 44A and the first charge storage unit 46a in FIG. The charges converted by the one light receiving element 44A are stored in the first charge storage section 46a. At time (6) shown in FIG. 12, the switching element between the first light receiving element 44A and the second charge storage unit 46b in FIG. 11 is turned on, and the charge converted by the first light receiving element 44A is second. The charge is accumulated in the charge accumulation unit 46b. At time (7) shown in FIG. 12, the switching element between the second light receiving element 44B and the third charge storage unit 46c in FIG. 11 is turned on, and the charge converted by the second light receiving element 44B is third. The charge is accumulated in the charge accumulation unit 46c. At time (8) shown in FIG. 12, the switching element between the second light receiving element 44B and the fourth charge accumulation unit 46d in FIG. 11 is turned on, and the charge converted by the second light receiving element 44B is the fourth. It is stored in the charge storage section 46d. In the four charge storage units 46a, 46b, 46c, and 46d, charges converted over a predetermined period (T2) are accumulated. When the predetermined period (T2) ends, the amount of charge accumulated in the charge storage units 46a, 46b, 46c, and 46d is converted into a voltage value and stored in the storage device 50. Furthermore, when the predetermined period (T2) ends, the charges accumulated in the charge accumulation units 46a, 46b, 46c, and 46d are erased.

  As shown in FIG. 12, time (1) and (3) are the same, time (2) and (4) are the same, time (5) and (7) are the same, time (6) And (8) are the same. That is, by using the first band-pass filter 48A and the second band-pass filter 48B, the first light receiving element 44A can execute the process of converting only the reflected light of the first wavelength light into the charge, and the second The light receiving element 44B can execute processing for converting only the reflected light of the second wavelength light into electric charges. The process in the first light receiving element 44A and the process in the second light receiving element 44B can be executed in parallel. Thereby, the image sensor 100 can acquire image data in a shorter time.

Specific examples of the present invention have been described in detail above, but these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and changes of the specific examples illustrated above.
The technical elements described in this specification or the drawings exhibit technical usefulness alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the technology exemplified in this specification or the drawings can achieve a plurality of objects at the same time, and has technical usefulness by achieving one of the objects.

20: Irradiation device 22: First light source 24: Second light source 30: Control device 40: Imaging device 42: Light receiving units 44a, 44b, 44c, 44d: Light receiving elements 46a, 46b, 46c, 46d: Charge storage units 48A, 48B : Band pass filter 50: Storage device 60: Computing device 70: Database 80: Light receiving device

Claims (5)

An image sensor for acquiring distance image data and part determination image data,
An irradiation device that irradiates an object with first wavelength light of a first wavelength and second wavelength light of a second wavelength different from the first wavelength;
An imaging device that accumulates a charge amount according to the first reflected light of the first wavelength light reflected by the object and accumulates a charge amount according to the second reflected light of the second wavelength light reflected by the object;
An arithmetic unit that calculates distance image data and part determination image data based on the amount of charge accumulated in the imaging device,
The imaging apparatus includes a first charge accumulation unit that accumulates a charge amount according to the first reflected light in a first time, and a second charge accumulation unit that accumulates a charge amount according to the first reflected light in a second time. And at least a third charge storage section that stores a charge amount according to the second reflected light in the third time period,
The computing device computes distance image data based on a distance from at least the first wavelength light reflected by the object from the irradiation device to reach the imaging device, and a reflection characteristic of the first reflected light on the object. And part determination image data based on the reflection characteristics of the second reflected light on the object ,
The arithmetic device calculates the first distance image data based on the charge amount accumulated in the first charge accumulation unit and the charge amount accumulated in the second charge accumulation unit, and is accumulated in the first charge accumulation unit. Calculating part discrimination image data based on the charge amount and the charge amount accumulated in the third charge accumulation unit;
The image sensor, wherein the first time and the second time are different . The imaging apparatus further includes a fourth charge accumulation unit that accumulates a charge amount according to the second reflected light in the fourth time period,
The arithmetic device calculates the second distance image data based on the charge amount accumulated in the third charge accumulation unit and the charge amount accumulated in the fourth charge accumulation unit,
The image sensor according to claim 1 , wherein the third time and the fourth time are different. One of the first distance image data and the second distance image data is used for distance image data of an object existing at a short distance, and the other is used for distance image data of an object existing at a long distance. The image sensor according to claim 2 . The imaging device receives a reflected light through an optical filter that selectively transmits a first wavelength and converts the reflected light into an electric charge, and an optical filter that selectively transmits a second wavelength. A second light receiving element that receives the reflected light via the light and converts the reflected light into an electric charge,
The first charge storage unit and the second charge storage unit store the charge converted by the first light receiving element,
The image sensor according to any one of claims 1 to 3 , wherein the third charge accumulation unit accumulates the charge converted by the second light receiving element. A light receiving device used for an image sensor that obtains distance image data and part determination image data using wavelength light of different wavelengths,
An imaging device that accumulates a charge amount according to the first reflected light of the first wavelength light reflected by the object and accumulates a charge amount according to the second reflected light of the second wavelength light reflected by the object When,
An arithmetic unit that calculates distance image data and part determination image data based on the amount of charge accumulated in the imaging device,
The imaging apparatus includes a first charge accumulation unit that accumulates a charge amount according to the first reflected light in a first time, and a second charge accumulation unit that accumulates a charge amount according to the first reflected light in a second time. And at least a third charge storage section that stores a charge amount according to the second reflected light in the third time period,
The computing device computes distance image data based on a distance from at least light of the first wavelength reflected by the object from the irradiation device to reach the imaging device, and reflection characteristics of the first reflected light on the object. And the region determination image data based on the reflection characteristics of the second reflected light on the object ,
The arithmetic device calculates distance image data based on the charge amount accumulated in the first charge accumulation unit and the charge amount accumulated in the second charge accumulation unit, and the charge amount accumulated in the first charge accumulation unit. And the region determination image data based on the amount of charge stored in the third charge storage unit,
The light receiving device, wherein the first time and the second time are different .

Images