JPWO2020121705A1 - Imaging device - Google Patents

Abstract

The image pickup device is a solid-state image sensor (106) that exposes reflected light from an object and accumulates it as a signal charge, and a control calculation device that controls irradiation from an infrared light source (103) and exposure of the solid-state image sensor (106). The solid-state image sensor (106) includes a plurality of photoelectric conversion units (4) for converting reflected light from an object into signal charges, and a plurality of charge storage units for accumulating signal charges. The image sensor performs m types of exposure sequences (m is an integer of 4 or more) for controlling irradiation and exposure within one frame period, and exclusively allocates a charge storage unit to the m types of exposure sequences. In at least one type of exposure sequence out of m types of exposure sequences, signal charges obtained from n (n is an integer of 3 or more) photoelectric conversion units (4) are stored in the charge storage unit.

Description

The present disclosure relates to an imaging device that irradiates light from a light source, converts the reflected light into an electric signal, and outputs the light.

In the application of detecting an object such as an obstacle, a TOF (Time Of Flight) method is used in which distance measurement is performed by utilizing the flight time of light reciprocating between a light source and an object. For example, in Patent Document 1, the object is irradiated with pulsed light, the reflected light is measured in two different periods, and the distance to the object is calculated from the ratio of the reflected light. Further, in the fields of biometric measurement and material analysis, a method is used in which an object is irradiated with light and the internal information of the object is acquired in a non-contact manner from the information of the light diffusely reflected inside the object. For example, in Patent Document 2, the object is irradiated with pulsed light, and the reflected light is measured in each of the period in which the reflected light increases and the period in which the reflected light decreases, thereby acquiring the internal information of the shallow part and the deep part of the object. ing.

Generally, the measurement of the reflected light in the above two methods is realized by using an image sensor to expose for a period of Δt to be measured. That is, the reflected light is converted into a signal charge by the photoelectric conversion unit (PD) arranged in a matrix and stored in the charge storage unit. The charge storage unit is composed of a vertical transfer unit (VCCD) in the case of a CCD type image sensor and a floating diffusion capacitance (FD) in the case of a MOS type image sensor. Here, the period Δt determines the moving distance z = c × Δt (c represents the speed of light) of the light to be measured, and determines the distance measurement accuracy in the above two methods and the separation accuracy of the internal information in the shallow part and the deep part. Since it depends, it is required to be a short time of about several ps to a dozen ns. Therefore, in order to obtain a signal charge sufficient for measurement, it is necessary to repeat the combination of pulse light irradiation and exposure a plurality of times in one frame period to accumulate the signal charge. Further, in order to measure the reflected light in a plurality of different periods, a charge storage unit that independently accumulates the signal charge for each measurement is required.

Japanese Patent Publication No. 2002-500567 Japanese Patent No. 608148 JP-A-2018-96988

In the above two methods, it may be desired to perform additional measurement by changing the period for measuring the reflected light and other conditions. For example, in order to cancel the influence of the incident light (background light) regardless of the irradiation of the pulsed light, the measurement may be performed without irradiating the pulsed light, and this may be subtracted from the measurement result at the time of irradiating the pulsed light. Further, it is conceivable to obtain more accurate information by subdividing the measurement period, or to perform measurement according to the property of the measurement target by changing the intensity and wavelength of the pulsed light. However, as already described, the addition of the measurement period or condition causes an increase in the mechanism for independently accumulating the signal charge for each measurement, that is, a decrease in the aperture ratio, and is output from the photoelectric conversion unit. The signal charge is reduced. In addition, since one frame period is divided into a plurality of periods or conditions for exposure, the signal charge in each measurement is significantly reduced as a whole. Therefore, there is a problem that the accuracy of the information calculated from these signal charges is also significantly reduced.

In response to these problems, in Patent Document 3, one of the photoelectric conversion units 4 is obtained by adding signal charges output from two photoelectric conversion units adjacent to each other in the vertical direction or the horizontal direction and accumulating them in the vertical transfer unit. While saving the number of phases of the vertical transfer unit required for hitting, the amount of signal charge accumulated in the divided exposure is increased. Further, by changing the combination of the photoelectric conversion units for adding signal charges between even-numbered columns and odd-numbered columns, or between even-numbered rows and odd-numbered rows, the decrease in spatial resolution due to the addition of signal charges is alleviated. However, when there are four or more types of measurement periods or conditions, the amount of signal charge becomes insufficient.

An object of the present disclosure is to provide an imaging device for accumulating a sufficient amount of signal charge even if there are four or more types of measurement periods or conditions, and for calculating the distance or internal information of an object with high accuracy. There is.

In order to achieve the above object, the image pickup device according to one aspect of the present disclosure includes a light source that irradiates an object with light, and a solid-state image sensor that exposes the reflected light from the object and accumulates it as a signal charge. A control unit that controls irradiation from the light source and exposure of the solid-state image sensor, the solid-state image sensor includes a plurality of photoelectric conversion units that convert light reflected from the object into the signal charge. The image sensor includes a plurality of charge storage units for accumulating the signal charge, and the image sensor performs m types of exposure sequences (m is an integer of 4 or more) for controlling the irradiation and exposure within one frame period. , The charge storage unit is exclusively assigned to the m types of exposure sequences, and in at least one type of exposure sequence among the m types of exposure sequences, n (n is an integer of 3 or more) obtained from the photoelectric conversion unit. The signal charge to be generated is stored in the charge storage unit.

According to the present disclosure, it is possible to realize an imaging device that calculates the distance or internal information of an object with high accuracy.

FIG. 1 is a diagram showing a schematic configuration of a TOF (Time Of Flight) type imaging device according to the first to third embodiments. FIG. 2A is a diagram illustrating an operation timing example and an operation principle in the image pickup apparatus according to the first to second embodiments. FIG. 2B is a diagram illustrating another operation timing example and an operation principle in the image pickup apparatus according to the first to second embodiments. FIG. 2C is a diagram illustrating still another operation timing example and operation principle in the image pickup apparatus according to the first to second embodiments. FIG. 3 is a diagram for explaining the operation timing in the image pickup apparatus according to the first to second embodiments. FIG. 4A is a diagram showing a configuration example of the solid-state image sensor according to the first to fourth embodiments. FIG. 4B is a diagram showing another configuration example of the solid-state image sensor according to the first to fourth embodiments. FIG. 5 is a diagram for explaining the drive timing of the photoelectric conversion unit and the vertical transfer unit according to the first to second embodiments. FIG. 6 is a diagram for explaining the drive timing of the photoelectric conversion unit and the vertical transfer unit according to the third embodiment. 7 (a), (b) and (c) show the long exposure sequence L1 according to the first to second embodiments, the strong exposure sequence K1 according to the third embodiment, and the fourth embodiment. It is a figure explaining the addition and transfer of the signal charge in the 750nm sequence P1 which concerns on. 8 (a), (b) and (c) show the short exposure sequence S1 according to the first to second embodiments, the weak exposure sequence J1 according to the third embodiment, and the fourth embodiment. It is a figure explaining addition and transfer of the signal charge in the 850 nm sequence Q1 which concerns on. 9 (a), (b) and (c) show the long exposure sequence 2 according to the first to second embodiments, the strong exposure sequence K2 according to the third embodiment, and the fourth embodiment. It is a figure explaining the addition and transfer of the signal charge in the 750nm sequence P2 which concerns on. 10 (a), (b) and (c) show the short exposure sequence S2 according to the first to second embodiments, the weak exposure sequence J2 according to the third embodiment, and the fourth embodiment. It is a figure explaining addition and transfer of the signal charge in the 850 nm sequence Q2 which concerns on. 11 (a), (b) and (c) show the long exposure sequence 3 according to the first to second embodiments, the background exposure sequence B0 according to the third embodiment, and the fourth embodiment. It is a figure explaining the addition and transfer of the signal charge in the 750nm sequence P3 which concerns on. 12 (a), (b) and (c) show the addition and transfer of signal charges in the short exposure sequence S3 according to the first to second embodiments and the 850 nm sequence Q3 according to the fourth embodiment. It is a figure explaining. FIG. 13A is a diagram illustrating an operation example of the distance range setting unit according to the second embodiment. FIG. 13B is a diagram illustrating another operation example of the distance range setting unit according to the second embodiment. FIG. 13C is a diagram illustrating a part of another operation example of the distance range setting unit according to the second embodiment. FIG. 14 is a diagram illustrating the operation of the distance correction unit according to the second embodiment. FIG. 15 is a diagram illustrating an observation range of reflected light from a distant object. FIG. 16A is a diagram illustrating an operation timing example and an operation principle in the image pickup apparatus according to the third embodiment. FIG. 16B is a diagram illustrating another operation timing example and an operation principle in the image pickup apparatus according to the third embodiment. FIG. 17 is a diagram illustrating operation timing in the image pickup apparatus according to the third embodiment. FIG. 18 is a diagram illustrating a distance signal selection flow according to the third embodiment. FIG. 19 is a diagram illustrating a schematic configuration of a cerebral blood flow measuring device according to a fourth embodiment. FIG. 20 is a diagram showing surface reflection and internal diffusion of an object. FIG. 21 is a graph of the transmission characteristics of the optical filter of the cerebral blood flow measuring device according to the fourth embodiment. FIG. 22A is a diagram illustrating an operation timing example and an operation principle in the cerebral blood flow measuring device according to the fourth embodiment. FIG. 22B is a diagram illustrating another operation timing example and the operation principle in the cerebral blood flow measuring device according to the fourth embodiment. FIG. 22C is a diagram illustrating still another operation timing example and operation principle in the cerebral blood flow measuring device according to the fourth embodiment. FIG. 23 is a diagram for explaining the operation timing in the cerebral blood flow measuring device according to the fourth embodiment. FIG. 24 is a diagram for explaining the drive timing of the photoelectric conversion unit and the vertical transfer unit according to the fourth embodiment. FIG. 25 is a graph of molecular absorption spectra of oxygenated hemoglobin and deoxygenated hemoglobin in the near infrared region. FIG. 26 is a diagram illustrating an operation example in which each exposure sequence of the first to fourth embodiments is combined.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Although the description will be given with reference to the accompanying drawings, this is for the purpose of exemplification, and the present disclosure is not intended to be limited thereto. Elements representing substantially the same configuration, operation and effect in the drawings are designated by the same reference numerals.

(First Embodiment)
In the first embodiment, an imaging device for accumulating a sufficient amount of signal charge even if there are four or more types of measurement periods or conditions and calculating the distance of an object with high accuracy will be described. Specifically, a configuration example will be described in which the image pickup apparatus expands the distance range that can be measured by using a long exposure period and a short exposure period light, and improves the accuracy in a wide distance range from near to far. ..

FIG. 1 is a diagram schematically showing a configuration of a TOF (Time Of Flight) type ranging device and peripheral objects according to the first to third embodiments. As shown in FIG. 1, in the imaging target space 100, the object 101 is irradiated with the pulsed irradiation light 110 having a wavelength of 850 nm or 940 nm from the infrared light source 103 under the background light illumination 102. The reflected light 111 is received by, for example, a solid-state image sensor 106, which is a CCD image sensor, through an optical lens 104 and an optical filter 105 that transmits a near-infrared wavelength region near 850 nm or 940 nm, and an image is formed. Is converted into a signal charge (hereinafter, this operation is referred to as exposure), and the signal charge amount 114 is output to the control calculation device 107. The control arithmetic unit 107 obtains six types of signal charge amounts 114 from the solid-state image sensor 106 by controlling the irradiation timing 112 of the infrared light source 103 and the exposure timing of the solid-state image sensor 106, and is the target. Two types of distances to the object 101 are calculated and output as a distance signal 115. An infrared light source 103, an optical lens 104, an optical filter 105, a solid-state image sensor 106, and a control arithmetic unit 107 constitute a distance measuring device. The control arithmetic unit 107 is realized by combining, for example, a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an AFE (Analog Front End), and the like.

In the figure, the infrared light source 103 irradiates an object with light. The solid-state image sensor 106 exposes the reflected light from the object and accumulates it as a signal charge. The control arithmetic unit 107 controls irradiation from the infrared light source 103 and exposure of the solid-state image sensor 106.

Hereinafter, the solid-state image sensor 106 and the control arithmetic unit 107 will be further described.

FIG. 4A is a configuration diagram of the solid-state image sensor 106 according to the present embodiment. Here, for the sake of simplification of the drawings, only 4 pixels in the vertical direction and 4 pixels in the horizontal direction are shown. The solid-state image sensor 106 is arranged in a matrix on a semiconductor substrate, and is read from a plurality of photoelectric conversion units (photodiodes) 4 that convert the reflected light 111 from the object 101 into signal charges, and a photoelectric conversion unit 4. A vertical transfer unit 5 that accumulates signal charges and transfers them in the column direction (vertical direction), a horizontal transfer unit 10 that transfers the signal charges transferred by the vertical transfer unit 5 in the row direction (horizontal direction), and a horizontal transfer unit. It includes a charge detection unit 11 that outputs the signal charge transferred by 10.

The solid-state image sensor 106 includes a plurality of photoelectric conversion units 4 that convert light reflected from an object into signal charges, and a plurality of charge storage units that store signal charges. The plurality of charge storage units are signal packets formed as potential wells in the vertical transfer unit 5 in the configuration example of FIG. 4A.

The control arithmetic unit 107 performs m types of exposure sequences (m is an integer of 4 or more) for controlling irradiation and exposure within one frame period. At that time, the charge storage unit is exclusively assigned to the m types of exposure sequences. In at least one type of exposure sequence out of m types of exposure sequences, signal charges obtained from n (n is an integer of 3 or more) photoelectric conversion units are stored in the charge storage unit.

Here, the solid-state image sensor 106 is an interline transfer type CCD. For example, the vertical transfer unit 5 is a 10-phase drive in which 10 gates of vertical transfer electrodes 8 are provided around two photoelectric conversion units 4 adjacent in the vertical direction. Yes, the horizontal transfer unit 10 is a two-phase drive. Of the vertical transfer electrodes 8, φV1 and φV3 also serve as read electrodes for the four photoelectric conversion units 4 connected to the odd-numbered vertical transfer units 5, and φV2 and φV4 are connected to the even-numbered vertical transfer units 5. It is configured to also serve as a readout electrode for the four photoelectric conversion units 4. As a result, the signal charges accumulated in the four photoelectric conversion units 4 are added to the positions represented by, for example, the signal packet 9a of the even-numbered vertical transfer units 5 when a high voltage is applied to φV1 and φV3. When a high voltage is applied to φV2 and φV4, the voltage is added to the position represented by, for example, the signal packet 9b of the even-numbered vertical transfer unit 5 and read out. After that, the signal charge on the vertical transfer unit 5 is transferred in the column direction by applying a voltage to the vertical transfer electrode 8.

Further, the photoelectric conversion unit 4 is provided with a VOD (vertical overflow drain) 12 for sweeping away the signal charge. However, in order to facilitate the understanding of the present disclosure, the VOD 12 is described in the lateral direction of the pixel surface, but is actually configured in the bulk direction of the pixel (the depth direction of the semiconductor substrate). When a high voltage is applied to the electrode φSub connected to the substrate of the VOD 12, the signal charges of all the photoelectric conversion units 4 are collectively discharged to the substrate.

With the above configuration, the solid-state image sensor 106 adds the signal charges accumulated in the four photoelectric conversion units 4 and distributes and stores the signal charges in any of the odd-numbered rows and the even-numbered rows in the patent document. Compared to the solid-state image sensor described in 3, the types of signal charges that can be accumulated while maintaining sensitivity are doubled. Here, since the vertical transfer unit 5 is driven by 10 phases, assuming that signal charges are accumulated in a minimum of 2 phases, 3 types of signal charges are accumulated in each of the odd-numbered row and even-numbered row vertical transfer units 5, for a total of 6 types of signal charges. It is possible.

As shown in FIG. 4B, the photoelectric conversion unit 4 that reads the signal charge to the vertical transfer unit 5 in the odd-numbered row and the photoelectric conversion unit 4 that reads the signal charge to the vertical transfer unit 5 in the even-numbered row are displaced in the column direction. The vertical transfer unit 5 may be arranged in the vertical transfer unit 5. In this case, by performing interpolation processing using the signal charges accumulated in the vertical transfer unit 5 in the odd-numbered rows and the signal charges accumulated in the vertical transfer units 5 in the even-numbered rows, the vertical transfer process is performed as compared with the configuration shown in FIG. 4A. Improvement of spatial resolution in the direction can be expected.

Further, the number of photoelectric conversion units 4 that add and read signal charges may be increased to, for example, 6, to further improve the sensitivity. However, the trade-off between improved sensitivity reduces spatial resolution.

2A to 2C are diagrams for explaining the operation timing and the operation principle of the distance measuring device according to the present embodiment. As shown in FIGS. 2A and 2B, the operation of the distance measuring device has a total of six types of exposure sequences, a long exposure sequence L1 to L3 and a short exposure sequence S1 to S3. These exposure sequences are realized by the control arithmetic unit 107 controlling the irradiation timing 112 of the infrared light source 103 and the exposure timing 113 of the solid-state image sensor 106, and one type of signal charge is applied to each exposure sequence. obtain. Here, the pulse width (irradiation time) of the irradiation light 110 is Tp and Tp'<Tp, the delay between the irradiation light 110 and the reflected light 111 is Δt, and the exposure period width is Te and Te'<Te.

First, as shown in FIG. 2A, in the long exposure sequence L1, the control arithmetic unit 107 controls the irradiation timing 112 and the exposure timing so that the exposure period includes from the rising edge of the irradiation light 110 to the falling edge of the reflected light 111. Control 113 is performed to obtain a signal charge S0 + BG based on the reflected light 111. Next, in the long exposure sequence L2, the control arithmetic unit 107 controls the irradiation timing 112 and the exposure timing 113 so that the exposure period starts from the falling edge of the irradiation light 110, and the signal charge based on the reflected light 111. Obtain S1 + BG. Further, in the long exposure sequence L3, the control arithmetic unit 107 controls the irradiation timing 112 and the exposure timing 113 so as to perform only the exposure without irradiating the irradiation light 110, and obtains the signal charge BG based on the reflected light 111. obtain. After that, the control arithmetic unit 107 extracts the signal charges S0 and S1 independent of the background light by subtracting BG from S0 + BG and S1 + BG.

Similarly, as shown in FIG. 2B, in the short exposure sequence S1, the control arithmetic unit 107 controls the irradiation timing so that the exposure period includes from the rising edge of the irradiation light 110 to the falling edge of the reflected light 111. The 112 and the exposure timing are controlled 113 to obtain the signal charge S0'+ BG' based on the reflected light 111. Next, in the short exposure sequence S2, the control arithmetic unit 107 controls the irradiation timing 112 and the exposure timing 113 so that the exposure period starts after the falling ΔTe'of the irradiation light 110, and the reflected light 111. The signal charge based on obtains S1'+ BG'. Further, in the short exposure sequence S3, the control arithmetic unit 107 controls the irradiation timing 112 and the exposure timing 113 so as to perform only the exposure without irradiating the irradiation light 110, and the signal charge BG'based on the reflected light 111'. To get. After that, the control arithmetic unit 107 extracts the signal charges S0'and S1'independent of the background light by subtracting BG'from S0'+ BG' and S1'+ BG'.

In addition, in FIGS. 2A and 2B, the irradiation light 110 is not irradiated in the long exposure sequence L3 and the short exposure sequence S3, but is included in the reflected light 111 after being irradiated as shown in FIG. 2C. By delaying the start time of the exposure period by the time ΔTbg and ΔTbg'that the irradiation light component decays to a negligible level, only the background light component may be exposed.

Further, in FIGS. 2A and 2B, for convenience of explanation, each exposure sequence is shown only once, but in practice, in order to obtain a sufficient amount of signal charge from the viewpoint of S / N, as shown in FIG. The long exposure sequences L1 to L3 are _{repeated α Long} × β times, and the short exposure sequences S1 to S3 are _{repeated α Short} × β times in one frame period. Here, since the short exposure sequences S1 to S3 have shorter irradiation times and exposure periods than the long exposure sequences L1 to L3, for example, _{by setting α Short} = α _Long × Tp / Tp', S0 and S0'and The sensitivity difference between S1 and S1'is reduced.

When the object to be distanced is a moving object, the exposure sequence is _{repeated α Long} times and α _Short times β times so that the time change of the distance affects each exposure sequence evenly. Again, this is not the case if the object 101 is a stationary body.

Next, the operation timing of the solid-state image sensor 106 in each exposure sequence will be described with reference to FIGS. 5 and 7 to 12. FIG. 5 shows an example of the drive timing of the photoelectric conversion unit 4 and the vertical transfer unit 5 in the six types of exposure sequences constituting one frame period, and FIGS. 7 to 12 show addition and transfer of signal charges. Shows an image. In FIG. 5, for the sake of simplicity, the operation for β = 1 time is extracted, and each exposure sequence is repeated twice (α _Long = α _Short = 2). Further, in FIGS. 7 to 12, the directions of reading and transferring the signal charge are indicated by arrows.

First, in the long exposure sequence L1 of FIG. 5, the voltages of φV1 and φV3 are set to the High level, and the voltages of φV2 and φV4 are set to the Low level. The signal charge can be read from the photoelectric conversion unit 4. Here, by setting the voltage of φSub to the Low level for the exposure period Te in synchronization with the irradiation light, as shown in FIG. 7A, the signal charges 31 from the four photoelectric conversion units 4 are vertically in an odd number of rows. It is stored in the transfer unit 5. The signal charge 31 includes an irradiation light component 31a and a background light component 31b. Further, the signal charge 31 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T1 in FIG. 5, the signal charge 31 is shown in FIG. 7 (b). As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 7C, the signal charge 31 is vertically transferred in the forward direction (downward in the drawing), and is in a state of being accumulated in V2 to V4.

Next, in the short exposure sequence S1 of FIG. 5, by setting the voltages of φV2 and φV4 to the High level and the voltages of φV1 and φV3 to the Low level, 4 connected to V2 and V4 of the even-numbered vertical transfer unit 5 The signal charge can be read from the two photoelectric conversion units 4. Here, by setting the voltage of φSub to the Low level for the exposure period Te'in synchronization with the irradiation light, as shown in FIG. 8A, the signal charges 32 from the four photoelectric conversion units 4 have an even number of rows. It is stored in the vertical transfer unit 5. The signal charge 32 includes an irradiation light component 32a and a background light component 32b. Further, the signal charge 31 and the signal charge 32 are separately stored in V2 and V4 of the vertical transfer unit 5, and the voltages of φV2, φV3, and φV4 are set to the Middle level at the time T2 in FIG. As shown in 8 (b), each packet is one packet. After that, as shown in FIG. 8C, the signal charge 31 and the signal charge 32 are vertically transferred in the forward direction (downward in the drawing) and are stored in V5 and V6.

Next, in the long exposure sequence L2 of FIG. 5, by shifting the start time of the exposure period and performing the same drive as in the long exposure sequence L1, as shown in FIG. 9A, the four photoelectric conversion units 4 The signal charge 33 of the above is stored in the vertical transfer unit 5 of an odd number of rows. At this time, since the signal charge 31 has moved to V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 33. The signal charge 33 includes an irradiation light component 33a and a background light component 33b. Further, the signal charge 33 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T3 of FIG. 5, the signal charge 33 is shown in FIG. 9 (b). As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 9C, the signal charge 33 is vertically transferred in the forward direction (downward in the drawing), and is in a state of being accumulated in V2 to V4.

Next, in the short exposure sequence S2 of FIG. 5, by shifting the start time of the exposure period and performing the same drive as in the long exposure sequence L2, as shown in FIG. 10A, the four photoelectric conversion units 4 The signal charge 34 of the above is stored in the vertical transfer unit 5 of an even number of rows. At this time, since the signal charge 32 is accumulated in V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 34. The signal charge 34 includes an irradiation light component 34a and a background light component 34b. Further, the signal charge 33 and the signal charge 34 are separately stored in V2 and V4 of the vertical transfer unit 5, and the voltages of φV2, φV3, and φV4 are set to the Middle level at the time T4 in FIG. As shown in 10 (b), each packet is one packet. After that, as shown in FIG. 10C, the signal charge 33 and the signal charge 34 are vertically transferred in the forward direction (downward in the drawing), and are stored in V5 and V6.

Next, in the long exposure sequence L3 of FIG. 5, by performing the same drive as the long exposure sequence L1 without irradiating from the infrared light source, as shown in FIG. 11A, the four photoelectric conversion units The signal charge 35 from 4 is accumulated in the vertical transfer unit 5 in an odd number of rows. At this time, since the signal charge 33 has moved to V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 35. Further, the signal charge 35 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T5 in FIG. 5, the signal charge 35 is shown in FIG. 11 (b). As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 11C, the signal charge 35 is vertically transferred in the forward direction (downward in the drawing), and is in a state of being accumulated in V2 to V4.

Finally, in the short exposure sequence S3 of FIG. 5, by performing the same drive as in the short exposure sequence S1 without irradiating from the infrared light source, as shown in FIG. 12A, the four photoelectric conversion units The signal charge 36 from 4 is stored in the even-numbered vertical transfer unit 5. At this time, since the signal charge 34 is accumulated in V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 36. Further, the signal charge 35 and the signal charge 36 are separately stored in V2 and V4 of the vertical transfer unit 5, and the voltages of φV2, φV3, and φV4 are set to the Middle level at the time T6 in FIG. As shown in 10 (b), each packet is one packet. After that, as shown in FIG. 12 (c), the signal charge 36 is vertically transferred from the signal charge 31 in the opposite direction (upward in the drawing), and the signal charge 31 and the signal charge 32 are accumulated in V1 to V3, that is, The long exposure sequence L1 is ready to be executed again.

By repeating the above operation β times, it is possible to accumulate six types of signal charges for the reflected light 111 for one frame period in the vertical transfer unit 5 with high sensitivity and independently.

The signal charge 31, the signal charge 33, and the signal charge 35 are accumulated by the even-numbered column vertical transfer unit 5, and the signal charge 32, the signal charge 34, and the signal charge 36 are accumulated by the odd-numbered column vertical transfer unit 5. You may go. In this case, the reading electrodes for the odd-numbered row photoelectric conversion unit 4 are φV6 and φV8. Further, the signal charge accumulated by the vertical transfer unit 5 in the odd-numbered row and the signal charge accumulated by the vertical transfer unit 5 in the even-numbered row may be exchanged for each frame. In this case, since the signal charges are interlaced in the column direction between the frames, improvement in the horizontal spatial resolution of the distance signal 115 can be expected.

Further, although the solid-state image sensor 106 has been described with the example of the CCD type image sensor, it may be a CMOS type image sensor.

Equations 1 and 2 are equations showing the calculation units of the two types of distances z and z'in the control arithmetic unit 107 according to the present embodiment. Assuming that the speed of light is c, the distance z is calculated from Equation 1 using the signal charges S0 and S1 obtained by removing the influence of the background light obtained from the long exposure sequences L1 to L3. Similarly, from Equation 2, the distance z'is calculated using the signal charges S0'and S1' after removing the influence of the background light obtained from the short exposure sequences S1 to S3. Further, ΔTe'is the lower limit of the distance z', and is a parameter for determining the degree of overlap of the distance z and the distance range of the z'.

z = c × Δt / 2
= (C × Tp / 2) × (Δt / Tp)
= (C × Tp / 2) × (S1 / S0) ・ ・ ・ Equation 1
z'= c × Δt / 2
= (C × Tp'/2) × (Δt / Tp')
= (C x Tp'/ 2) x (S1'/ S0'+ ΔTe'/ Tp')
= (C × Tp ′ / 2) × (S1 ′ / S0 ′) ＋ (c × ΔTe ′ / 2) ・ ・ ・ Equation 2

Here, considering the case of ΔTe'= 0, the distances z and z'are ideally the same value, but in reality, the calculation is performed using the signal charge amount quantized with a finite resolution. There is a difference in accuracy. For example, assuming that S1 / S0 and S1'/ S0'are quantized with Nbit integer accuracy, the unit distance of the distance z = (c × Tp / 2) / 2N, and the unit distance of the distance z'= (c ×). Tp'/ 2) / 2N, and the distance accuracy is higher for the distance z'. On the other hand, the maximum value of the distance z = (c × Tp / 2) × (2N-1) / 2N, and the maximum value of the distance z ′ = (c × Tp ′ / 2) × (2N-1) / 2N. , The distance range is wider at the distance z. Therefore, by calculating the two types of distances z and z', it is possible to achieve both distance measurement over a wide distance range and distance measurement with high distance accuracy.

The irradiation times and exposure periods of the long exposure sequences L1 to L3 and the short exposure sequences S1 to S3 are aligned (that is, Tp = Tp', Te = Te'), and the signal charge is accumulated with ΔTe'= Tp. And, the distance may be calculated. In this case, the distances z and z'are the same distance accuracy, and the distance range due to the combination of the distances z and z'is twice the distance range of the distance z.

By the above operation, six kinds of signal charges can be independently accumulated in the vertical transfer unit in one frame period. Further, since the signal charges of the four photoelectric conversion units are added, the sensitivity is substantially doubled as compared with the conventional method of adding the signal charges of the two photoelectric conversion units, and the type of signal charge to be accumulated. Can be offset by the decrease in each charge due to the increase in. Therefore, it is possible to calculate with high accuracy two types of distances with different distance ranges and distance accuracy from six types of signal charges, and it is possible to achieve both wide range distance measurement and high distance accuracy distance measurement. Become.

As described above, the image sensor according to the first embodiment includes a solid-state image sensor 106 that exposes reflected light from an object and accumulates it as a signal charge, and irradiation from an infrared light source 103 and a solid-state image sensor 106. The solid-state image sensor 106 includes a control calculation device 107 for controlling exposure, a plurality of photoelectric conversion units 4 for converting reflected light from an object into signal charges, a plurality of charge storage units for accumulating signal charges, and a plurality of charge storage units. The image sensor performs m types of exposure sequences (m is an integer of 4 or more) within one frame period for controlling irradiation and exposure, and exclusively allocates a charge storage unit to the m types of exposure sequences. In at least one type of exposure sequence out of m types of exposure sequences, signal charges obtained from n (n is an integer of 3 or more) photoelectric conversion units are stored in the charge storage unit.

Here, the control arithmetic unit 107 may repeat at least one type of exposure sequence within one frame period.

Here, the control arithmetic unit 107 associates the combination of n photoelectric conversion units with one charge storage unit, and stores the signal charges obtained from the n photoelectric conversion units in the corresponding charge storage units. You may.

Here, the control arithmetic unit 107 may add the signal charges obtained from the n photoelectric conversion units and store them in the corresponding charge storage units in at least one type of exposure sequence.

Further, the m types of exposure sequences may differ from each other in at least one of the irradiation intensity from the light source, the irradiation time, the irradiation wavelength, and the exposure period of the image sensor.

The control arithmetic unit 107 may change the combination of n photoelectric conversion units for each exposure sequence of k types (k is an integer of 1 or more).

Here, the image pickup apparatus may set the number of repetitions of the m types of exposure sequences based on the irradiation intensity from the light source, the irradiation time, the irradiation wavelength, and the exposure period of the solid-state image pickup device.

Here, the image pickup apparatus may change the combination of the n photoelectric conversion units that add the signal charges for each frame.

Here, the m is 5 or 6, the plurality of photoelectric conversion units are arranged in a matrix, and the imaging device repeats the exposure sequence consisting of 5 or 6 types a plurality of times in one frame period, respectively, and the above-mentioned. The n photoelectric conversion units may be a combination of the four photoelectric conversion units arranged in 2 rows and 2 columns.

Here, the image pickup device is a TOF (Time Of Flight) type distance measuring device, and the image pickup device is accumulated in the charge storage unit in at least two types of exposure sequences among the m types of exposure sequences. A distance calculation unit that calculates the distance to the object using the signal charge may be provided.

Here, the m types of exposure sequences include a long exposure sequence and a short exposure sequence, and the long exposure sequence has a longer irradiation time from the light source and an exposure period of the solid-state image sensor than the short exposure sequence. Each is set to be long, and the distance calculation unit may calculate at least two types of distances to the object based on the m types of exposure sequences.

Here, the image pickup device is a TOF type distance measuring device, the photoelectric conversion unit is arranged on a matrix, and the m types of exposure sequences are the first to third long exposure sequences and the first to third exposure sequences. Including the third short exposure sequence, the irradiation time from the light source and the exposure period of the solid-state image sensor in the first to third long exposure sequences are longer than the first to third short exposure sequences, respectively. The exposure period in the second long exposure sequence is different from the exposure period in the first long exposure sequence, and the exposure period in the second short exposure sequence is different from the exposure period in the first short exposure sequence. The third long exposure sequence and the third short exposure sequence expose background light that does not contain a reflected light component due to irradiation from the light source, and the first to third long exposure sequences and the third long exposure sequence. The first to third short exposure sequences are repeated a plurality of times within one frame period, and the n photoelectric conversion units are four photoelectric conversion units composed of 2 rows and 2 columns, and the first to third photoelectric conversion units are formed. It is provided with a distance calculation unit that calculates two types of distances to an object by using each of the signal charge accumulated in the long exposure sequence and the signal charge accumulated in the first to third short exposure sequences. You may.

Here, in the third long-distance sequence and the third short-distance sequence, the period after the irradiation from the light source is performed and the reflected light is extinguished may be the exposure period.

(Second Embodiment)
Hereinafter, the image pickup apparatus according to the second embodiment will be described focusing on the differences from the first embodiment. In the second embodiment, a configuration example for suppressing or warning erroneous measurement due to multipath in an environment in which so-called multipath in which a reflected wave as a direct wave and a reflected wave as an indirect wave coexist is likely to occur will be described. Therefore, the image pickup device according to the present embodiment has a schematic configuration of the image pickup device shown in FIG. 1 and uses the solid-state image pickup device shown in FIGS. 4A and 4B, similarly to the image pickup device according to the first embodiment. A distance range determination unit that operates at the operation timings shown in FIGS. 2A to 2C and the drive timing shown in FIG. And a distance correction unit that corrects the distance z using the distance z'. Here, the distance z refers to the distance measured by the long exposure sequences L1 to L3 or the measurable distance range. The distance z'refers to the distance measured by the short exposure sequences S1 to S3 or the measurable distance range.

The distance range determination unit has a full scanning mode in which the distance range of the distance z'is shifted by a fixed amount for each frame, and a selective scanning mode in which the distance range of the distance z'is scanned based on the distribution of the distance z in the past frames. And.

In the full scanning mode, as shown in FIG. 13A, ΔTe'is determined so that the distance range 120 of the distance z'overlaps a part of the distance range of the distance z (from Equation 2, ΔTe' + Tp'≤ Tp). Scanning is performed every frame. For example, in the case of Tp = 4Tp', by setting ΔTe'= 0 in frame 0, the distance range 120 of the distance z'is set in the distance range A, and in frame 1, ΔTe'= Tp', the distance is set. By setting the distance range 120 of z'to the distance range B and setting ΔTe'= 2Tp'in the frame 2, the distance range 120 of the distance z'is set to the distance range C, and ΔTe'= 3Tp'in the frame 3. By setting the distance range 120 of the distance z'to the distance range D and setting ΔTe'= 0 again in the frame 4, the distance range 120 of the distance z'is set to the distance range A, and so on. Repeat the operation of.

By the above operation, in the full scanning mode, the distance range 120 having a distance z'is scanned at a constant cycle, so that highly accurate distance measurement can be performed over a wide distance range.

On the other hand, in the selective scanning mode, as shown in FIGS. 13B and 13C, the histogram 121 of the distance z is calculated at regular intervals, and the distance range 120 of the distance z'is overlapped with the distance range whose frequency is equal to or higher than the threshold value. Is determined, and scanning is performed every frame. For example, when the histogram 121 of the distance z is calculated every four frames as in the example of FIG. 13C, in the histogram 121 of the distance z in the frame 0, the frequency exceeds the threshold value in the distance ranges B and D, so that from frame 1. The distance range of the distance z'in 4 scans only the distance ranges B and D. On the other hand, in the frame 4, since the frequency exceeds the threshold value in the distance ranges A and C, the distance range of the distance z'in the frames 5 to 8 scans only the distance ranges A and C.

By the above operation, in the selective scanning mode, the distance range of the distance z'is selectively scanned based on the distance z of the object, so that the distance measurement is performed with high accuracy in a shorter cycle than in the full scanning mode. Is possible, and the distance measurement followability to a moving object is high.

When the region where the object of interest is imaged is known, the histogram 121 of the distance z is not calculated over the entire imaging region, but is calculated only in the region where the object of interest is imaged. You may. In this case, since the frequency is concentrated near the distance of the object of interest, the distance range 120 of the distance z'can be scanned more efficiently. Further, even when there are a plurality of objects of interest, the same operation can be performed by adding up the histogram 121 of the distance z with respect to the region in which each is imaged.

The distance correction unit includes a high-precision mode in which a part of the distance z is made highly accurate by using the distance z', and a multi-pass correction mode in which the influence of the multi-pass at the distance z is reduced by using the distance z'. .. The distance corrected by the distance correction unit is output as a distance signal 115.

In the high-precision mode, the high-precision distance zh is held over a plurality of frames, a part of the area is updated with the high-precision distance z'for each frame, and the distance changes in the other areas. If there is (or is large), it is updated with a low-precision distance z, and if there is no change (or small) in the distance, the original distance is maintained. For example, as shown in FIG. 14, in frame 0, after initializing the entire region of the distance zh with the distance z (arrow 130), the distance zh is set with the distance z'for the region A within the distance range of the distance z'. Update (arrow 131). Next, in frame 1, the distance zh is updated with the distance z'for the region B within the distance range of the distance z'(arrow 131), and the regions A and C are not updated because there is no difference between the distance zh and the distance z. .. Similarly, in frame 2, the distance zh is updated with the distance z'for the region C within the distance range of z'(arrow 131), and the regions A and B are not updated because there is no difference between the distance zh and the distance z. Further, in the frame 3, the distance zh is updated by the distance z'for the region A within the distance range of the distance z'(arrow 131), and the region B is updated by the distance z because there is a difference between the distance zh and the distance z. (Arrow 132), the area C is not updated because there is no difference between the distance zh and the distance z.

By the above operation, in the high-precision mode, it is possible to perform distance measurement with high distance accuracy for a stationary object while performing distance measurement following a moving object.

On the other hand, in the multipath correction mode, the distance zc in which the influence of the multipath is reduced is calculated. Here, multipath is a multiple reflection of irradiation light, and has the property that a certain object receives and reflects the reflected light from an object further behind, so that the measurement distance shifts to the rear of the actual distance. .. For example, as shown in FIG. 15, assuming that the object A and the object B exist at a distance cΔta / 2 and a distance cΔtb1 / 2 to cΔtb2 / 2, respectively, the reflected light from the object B causes the object A. Consider the case where it is observed via. First, the observation range 140 of the reflected light from the object B in the long exposure sequences L1 and L2 is a distance = 0 to c (Tp + Te) / 2. On the other hand, the observation range 141 of the reflected light from the object B in the short exposure sequences S1 and S2 is the distance = cΔTe ′ / 2 to c (ΔTe ′ + Tp ′ + Te ′) / 2. Here, from Te'<Te and ΔTe'+ Tp'≤ Tp, (ΔTe'+ Tp'+ Te') ≤ (Tp + Te), and excluding the boundary conditions, the observation range 141 is the object compared to the observation range 140. Since the reflected light from B is not included, the distance z'has less deviation due to the influence of multipath than the distance z. Therefore, in the multipath correction mode, it is determined that the region within the distance range of the distance z'is under the influence of the multipath if the difference between the distance z and the distance z'is larger than a predetermined value. The distance zc = z', and conversely, if it is determined that the distance is not under the influence of multipath, the distance zc = z.

By the above operation, in the multipath correction mode, the influence of the multipath is reduced by detecting the area under the influence of the multipath and selectively outputting the distance z'which is less affected by the multipath. be able to.

For the area under the influence of multipath, instead of simply selecting and outputting the distance z', the weighted average of the distance z'and the distance z is taken and output according to the degree of influence of the multipath. May be good. Further, an abnormality notification unit that outputs a signal for notifying an abnormality may be provided in addition to the distance signal 115 when it is determined that the condition is under the influence of multipath.

By combining the two types of distances z and z'with the above distance range determination unit and distance correction unit, it is possible to achieve both wide range distance measurement and high-precision distance measurement. Moreover, by reducing the influence of multipath, more accurate distance measurement can be performed.

As described above, in the imaging apparatus according to the second embodiment, the distance calculated from the long exposure sequence or the first to third long exposure sequences is determined by the short exposure sequence or the first. It is provided with a distance correction unit that corrects using the distance calculated from the first to third short exposure sequences.

Here, a part of the distance calculated from the long exposure sequence or the first to third long exposure sequences and the distance calculated from the short exposure sequence or the first to third short exposure sequences. An abnormality notification unit may be provided to notify an abnormality when the difference between the two and the above exceeds a specified range.

Here, assuming that the exposure period in the short exposure sequence or the first to third short exposure sequences is the short exposure period, the short exposure period is changed for each frame, and for each frame of the short exposure period. Due to the change, the distance range calculated from the short exposure sequence or the first to third short exposure sequences is the distance calculated from the long exposure sequence or the first to third long exposure sequences. It may be scanned after being restricted to overlap a part of the range.

Here, the exposure period in the short exposure sequence or the first to third short exposure sequences may be set based on the distance calculated from the first to third long exposure sequences of the past frames.

(Third Embodiment)
Hereinafter, the image pickup apparatus according to the third embodiment will be described focusing on the differences from the first embodiment. In the third embodiment, a configuration example for expanding the dynamic range of distance measurement by using strong irradiation light and weak irradiation light will be described. Expanding the dynamic range means, for example, extending the range that can be measured from a nearby object with high reflectance to a distant object with low reflectance. Therefore, the image pickup device according to the present embodiment has a schematic configuration of the image pickup device shown in FIG. 1 and uses the solid-state image pickup device shown in FIG. 4, like the image pickup device according to the first embodiment. Unlike the image pickup device according to the embodiment, distance measurement is performed using a plurality of exposure sequences having different irradiation intensities of the infrared light source 103.

FIG. 16 is a diagram illustrating an operation timing and an operation principle in the image pickup apparatus according to the present embodiment. As shown in FIGS. 16A and 16B, the operation of the image pickup apparatus comprises a total of five types of exposure sequences: a strong exposure sequence K1 to K2, a weak exposure sequence J1 to J2, and a background exposure sequence B0. These exposure sequences are realized by the control arithmetic unit 107 controlling the irradiation timing 112 of the infrared light source 103 and the exposure timing 113 of the solid-state image sensor 106, and one type of signal charge is applied to each exposure sequence. obtain. Here, the pulse width (irradiation time) of the irradiation light 110 is Tp, the delay between the irradiation light 110 and the reflected light 111 is Δt, and the exposure period width is Te.

First, as shown in FIG. 16A, in the strong exposure sequence K1, the control calculation device 107 sets the irradiation intensity of the irradiation light 110 to AHigh, and the exposure period includes from the rise of the irradiation light 110 to the fall of the reflected light 111. The irradiation timing control 112 and the exposure timing control 113 are performed to obtain a signal charge S0 + BG based on the reflected light 111. Next, in the strong exposure sequence K2, the control calculation device 107 controls the irradiation timing 112 and the exposure timing 113 so that the irradiation intensity of the irradiation light 110 becomes AHigh and the exposure period starts from the falling edge of the irradiation light 110. To obtain the signal charge S1 + BG based on the reflected light 111. Further, in the background exposure sequence B0, the control arithmetic unit 107 controls the irradiation timing 112 and the exposure timing 113 so as to perform only the exposure without irradiating the irradiation light 110, and obtains the signal charge BG based on the reflected light 111. obtain. After that, the control arithmetic unit 107 extracts the signal charges S0 and S1 independent of the background light by subtracting BG from S0 + BG and S1 + BG.

Similarly, as shown in FIG. 16B, in the weak exposure sequence J1, in the control calculation device 107, the irradiation intensity of the irradiation light 110 is ALo <AHigh, and the exposure period is from the rise of the irradiation light 110 to the fall of the reflected light 111. The irradiation timing control 112 and the exposure timing control 113 are performed so as to include the above, and the signal charge S0'+ BG based on the reflected light 111 is obtained. Next, in the weak exposure sequence J2, the control calculation device 107 controls the irradiation timing 112 and the exposure timing 113 so that the irradiation intensity of the irradiation light 110 becomes ALo and the exposure period starts from the falling edge of the irradiation light 110. Is performed, and the signal charge based on the reflected light 111 obtains S1'+ BG. The operation of the background exposure sequence B0 is the same as that in FIG. 16A. After that, the control arithmetic unit 107 extracts the signal charges S0'and S1'independent of the background light by subtracting BG from S0'+ BG and S1'+ BG.

Further, in FIGS. 16A and 16B, for convenience of explanation, each exposure sequence is shown only once, but in practice, in order to obtain a sufficient amount of signal charge from the viewpoint of S / N, as shown in FIG. Each exposure sequence is repeated α × β times in one frame period.

If it is difficult to adjust AHigh and ALo to the desired intensity due to the characteristics of the infrared light source 103, the strong exposure sequences K1 to K2 are repeated α _High × β times, and the weak exposure sequences J1 to J2. The total amount of light in one frame period may be individually adjusted by repeating α _{Low × β times.} In this case, since the amount of the background light component contained in the signal charge is different between the strong exposure sequences K1 to K2 and the weak exposure sequences J1 to J2, the BG to be subtracted is corrected by the ratio of _{α High} : α _Low. I do.

Next, the operation timing of the solid-state image sensor 106 in each exposure sequence will be described with reference to FIGS. 6 and 7 to 12. FIG. 5 shows an example of the drive timing of the photoelectric conversion unit 4 and the vertical transfer unit 5 in the six types of exposure sequences constituting one frame period, and FIGS. 7 to 12 show addition and transfer of signal charges. Shows an image. In FIG. 5, for the sake of simplicity, the operation for β = 1 time is extracted, and each exposure sequence is repeated twice (α _High = α _Low = 2). Further, in FIGS. 7 to 12, the directions of reading and transferring the signal charge are indicated by arrows.

First, in the strong exposure sequence K1 of FIG. 6, the voltages of φV1 and φV3 are set to the High level, and the voltages of φV2 and φV4 are set to the Low level. The signal charge can be read from the photoelectric conversion unit 4. Here, by setting the voltage of φSub to the Low level for the exposure period Te in synchronization with the irradiation light, as shown in FIG. 7A, the signal charges 31 from the four photoelectric conversion units 4 are vertically in an odd number of rows. It is stored in the transfer unit 5. The signal charge 31 includes an irradiation light component 31a and a background light component 31b. Further, the signal charge 31 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T1 in FIG. As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 7C, the signal charge 31 is vertically transferred in the forward direction (downward in the drawing), and is in a state of being accumulated in V2 to V4.

Next, in the weak exposure sequence J1 of FIG. 6, by setting the voltages of φV2 and φV4 to the High level and the voltages of φV1 and φV3 to the Low level, 4 connected to V2 and V4 of the even-numbered vertical transfer unit 5 The signal charge can be read from the two photoelectric conversion units 4. Here, by setting the voltage of φSub to the Low level for the exposure period Te in synchronization with the irradiation light, as shown in FIG. 8A, the signal charges 32 from the four photoelectric conversion units 4 are vertical in an even number of rows. It is stored in the transfer unit 5. The signal charge 32 includes an irradiation light component 32a and a background light component 32b. Further, the signal charge 31 and the signal charge 32 are separately stored in V2 and V4 of the vertical transfer unit 5, and the voltages of φV2, φV3, and φV4 are set to the Middle level at the time T2 in FIG. As shown in 8 (b), each packet is one packet. After that, as shown in FIG. 8C, the signal charge 31 and the signal charge 32 are vertically transferred in the forward direction (downward in the drawing) and are stored in V5 and V6.

Next, in the strong exposure sequence K2 of FIG. 6, by shifting the start time of the exposure period and performing the same drive as in the strong exposure sequence K1, as shown in FIG. 9A, the four photoelectric conversion units 4 The signal charge 33 of the above is stored in the vertical transfer unit 5 of an odd number of rows. At this time, since the signal charge 31 has moved to V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 33. The signal charge 33 includes an irradiation light component 33a and a background light component 33b. Further, the signal charge 33 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T3 in FIG. As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 9C, the signal charge 33 is vertically transferred in the forward direction (downward in the drawing), and is in a state of being accumulated in V2 to V4.

Next, in the weak exposure sequence J2 of FIG. 6, by shifting the start time of the exposure period and performing the same drive as in the weak exposure sequence J1, as shown in FIG. The signal charge 34 of the above is stored in the vertical transfer unit 5 of an even number of rows. At this time, since the signal charge 32 is accumulated in V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 34. The signal charge 34 includes an irradiation light component 34a and a background light component 34b. Further, the signal charge 33 and the signal charge 34 are separately stored in V2 and V4 of the vertical transfer unit 5, and the voltages of φV2, φV3, and φV4 are set to the Middle level at the time T4 in FIG. As shown in 10 (b), each packet is one packet. After that, as shown in FIG. 10C, the signal charge 33 and the signal charge 34 are vertically transferred in the forward direction (downward in the drawing), and are stored in V5 and V6.

Finally, in the background exposure sequence B0 of FIG. 6, by performing the same drive as the strong exposure sequence K1 without irradiating from the infrared light source, as shown in FIG. 11A, the four photoelectric conversion units The signal charge 35 from 4 is accumulated in the vertical transfer unit 5 in an odd number of rows. At this time, since the signal charge 33 has moved to V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 35. Further, the signal charge 35 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T5 in FIG. As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 12 (c), the signal charge 35 is vertically transferred from the signal charge 31 in the opposite direction (upward in the drawing), and the signal charge 31 and the signal charge 32 are accumulated in V1 to V3, that is, The strong exposure sequence K1 is ready to be executed again. The position of the signal charge 36 is an empty packet and is not used.

By repeating the above operation β times, it is possible to accumulate five types of signal charges for the reflected light 111 for one frame period in the vertical transfer unit 5 with high sensitivity and independently.

From the difference in the irradiation intensity of the irradiation light 110 in the strong exposure sequences K1 to K2 and the weak exposure sequences J1 to J2, the signal charge amounts S0 + BG and S1 + BG, and S0'+ BG and S1'+ BG are the reflectance of the object and Has a sensitivity difference with respect to distance.

Equations 1 and 3 are equations showing the calculation units of the two types of distances z and z'in the control arithmetic unit 107 according to the present embodiment. Assuming that the speed of light is c, the distance z is calculated from Equation 1 using the signal charges S0 and S1 obtained by removing the influence of the background light obtained from the strong exposure sequences K1 and 2 and the background exposure sequence B0. Similarly, from Equation 3, the distance z'is calculated using the signal charges S0'and S1' after removing the influence of the background light obtained from the weak exposure sequences J1 to J2 and the background exposure sequence B0. When S0 = zero level, the distance z cannot be calculated because the division cannot be performed correctly. Further, when S0 + BG and S1 + BG = saturation level, the distance z cannot be calculated because S0 and BG cannot be taken out correctly. Similarly, when S1 = zero level, S0'+ BG and S1'+ BG = saturation level, the distance z'cannot be calculated.

z = (c × Tp / 2) × (S1 / S0) ・ ・ ・ Equation 1 (repost)
z'= (c × Tp / 2) × (S1'/ S0') ・ ・ ・ Equation 3

Therefore, the control arithmetic unit 107 selects which of the two types of distances z and z'should be calculated according to the amounts of the five types of signal charges. For example, as shown in FIG. 18, first, in condition 1, if S0 + BG or S1 + BG is a saturation level, the process proceeds to condition 2, otherwise the process proceeds to condition 3. Next, in condition 2, if S0'+ BG is the saturation level or S0'is the zero level, distance measurement is not possible, and if not, z'is adopted. Further, in condition 3, if S0 is at the zero level, z is adopted, otherwise distance measurement is not possible.

By selecting the distance calculation unit as described above, the signal charge becomes the saturation level and the zero level due to the influence of the reflectance and the distance of the object, and the region where the distance cannot be measured can be minimized.

By the above operation, five kinds of signal charges can be independently accumulated in the vertical transfer unit in one frame period. Further, since the signal charges of the four photoelectric conversion units are added, the sensitivity is substantially doubled as compared with the conventional method of adding the signal charges of the two photoelectric conversion units, and the type of signal charge to be accumulated. Can be offset by the decrease in each charge due to the increase in. Therefore, it is possible to select a distance calculation unit according to the reflectance and distance of the object from the five types of signal charges, and it is possible to realize a distance measuring device that does not select the space to be photographed.

(Fourth Embodiment)
The imaging device of the fourth embodiment targets a light scattering body such as a living body as an object. More specifically, the imaging apparatus of the present embodiment uses two types of infrared light having different peak wavelengths, for example, to observe changes in blood flow in the epidermis and brain of the subject. Detect the distribution and its temporal change. This makes it possible to generate a two-dimensional image of a still image or a moving image showing the distribution. By using the information of the image, for example, the brain activity (for example, concentration ratio or emotion) of the subject can be estimated.

Since the imaging apparatus of the present embodiment can detect the above-mentioned biological information in the same field in a non-contact manner, it is possible to eliminate the troublesomeness associated with the detection, and it is possible to eliminate the troublesomeness associated with the detection. The detection accuracy of blood flow information can be greatly improved.

Hereinafter, the image pickup apparatus of the present embodiment that enables such high-precision detection will be described.

FIG. 19 is a diagram schematically showing a configuration of an imaging device according to the present embodiment. FIG. 19 shows not only the imaging device but also the object to be detected 1901 (human head) and the photographing space 1900.

The present embodiment is different from the first to third embodiments in that two light sources that emit pulsed light having different wavelengths are used. An example of obtaining hemoglobin information of blood flow using the imaging device according to the fourth embodiment will be described.

The image pickup device of the present embodiment includes a first light source 1903, a second light source 1904, an optical system 1905, an optical filter 1906, a solid-state image sensor 1907, and a control calculation device 1908.

The first light source 1903 and the second light source 1904 emit pulsed light in the direction in which the object 1901 is located. In the present embodiment, the first light source 1903 is a laser light source that emits a narrow band pulsed light having a central wavelength of 750 nm, and the second light source is a laser that emits a narrow band pulsed light having a central wavelength of 850 nm. It is a light source. From the object 1901, surface reflection and reflected light component 1912 that is diffused and then reflected inside are incident on the optical system 1905. The optical filter 1906 is arranged between the optical system 1905 and the solid-state image sensor 1907, and mainly transmits only light having a wavelength corresponding to the wavelength of light from the first light source 1903 and the second light source 1904. The control arithmetic unit 1908 is connected to the first light source 1903, the second light source 1904, and the solid-state image sensor 1907, and controls their operations. More specifically, the control arithmetic unit 1908 synchronizes the light emission timing 1913 of the first light source 1903 and the second light source 1904 with the signal accumulation and signal emission timing 1914 of each pixel of the solid-state image sensor 1907. Control.

The control arithmetic unit 1908 is connected to the solid-state image sensor 1907, and generates and outputs image data 1916 (for example, two-dimensional moving image data) based on the electric signal 1915 output from the solid-state image sensor 1907. The generated image data can be sent to, for example, a display (not shown), and an image showing the state of cerebral blood flow can be displayed on the display.

As a result, information on the scalp and cerebral blood flow inside the living body can be detected with high accuracy, and in particular, the amount of change in oxygenated hemoglobin and deoxygenated hemoglobin contained in the blood flow can be detected.

Hereinafter, each component and its operation will be described in more detail.

The first light source 1903 in the present embodiment is a laser pulse light source that emits a narrow band pulsed light having a central wavelength of 750 nm, and the second light source 1904 is a laser pulse light source that emits a narrow band pulsed light having a central wavelength of 750 nm. The first light source 1903 and the second light source 1904 repeatedly emit pulsed light in a predetermined pattern determined by the control arithmetic unit 1908, as will be described later. The pulsed light emitted by the first light source 1903 and the second light source 1904 is, for example, the time from the start of the rise, which is the rise time, to the complete rise, and the time after the start of the fall, which is the fall time. It can be a rectangular wavy light whose time to completely fall is close to zero.

The first light source 1903 and the second light source 1904 are light sources such as a laser diode (LD) in which the rising portion and the falling portion of the pulsed light are close to perpendicular to the time axis (that is, the time response characteristic is rapid). could be. As the first light source 1903 and the second light source 1904, any kind of light source that emits pulsed light, such as a semiconductor laser, a solid-state laser, and a fiber laser, can be used.

In the imaging device of the present embodiment, since the object 1901 is a human body, a first light source 1903 and a second light source 1904 in consideration of the influence on the retina can be used. For example, when a laser light source is used, a light source that satisfies Class 1 of the laser safety standard established in each country can be used. When Class 1 is satisfied, the object 1901 is irradiated with light having a low illuminance such that the exposure limit (AEL) is less than 1 mW. The first light source 1903 and the second light source 1904 themselves may not satisfy the class 1, or the class 1 may be satisfied by the combination with other optical elements. For example, an element such as a diffuser or an ND filter is placed between the first light source 1903 and the second light source 1904 and the object 102, and the light is diffused or attenuated to meet Class 1 of the laser safety standard. May be done.

The wavelengths of light emitted by the first light source 1903 and the second light source 1904 are not limited to 750 nm and 850 nm. For example, light of any wavelength (red light or near-infrared light) included in the wavelength range of 650 nm or more and 950 nm or less can be used. The above wavelength range is called a "window of a living body" and has a property of being relatively difficult to be absorbed by water and skin in the living body. When a living body is targeted for detection, the detection sensitivity can be increased by using light in the above wavelength range. When detecting changes in blood flow in the skin and brain of an object 1901, as in the present embodiment, the light used is considered to be mainly absorbed by oxygenated hemoglobin and oxygenated hemoglobin, and light with respect to each wavelength, respectively. Absorption degree is different. When the blood flow changes, the concentrations of oxygenated hemoglobin and deoxygenated hemoglobin are considered to change, so that the degree of light absorption also changes. Therefore, the amount of light detected changes before and after the change in blood flow.

In the present disclosure, the object 1901 is not limited to a living body. For example, other types of light scatterers such as gases, chemicals, foods, etc. can be objects 1901. The wavelength range of the light emitted by the first light source 1903 and the second light source 1904 is not limited to the near-infrared wavelength range (about 700 nm or more and about 2500 nm or less), for example, the visible light wavelength range (about 400 nm or more and about 700 nm or less). ), The wavelength range of ultraviolet rays (about 10 nm or more and about 400 nm or less) may be used. Depending on the application, electromagnetic waves in the radio range such as mid-infrared rays, far-infrared rays, or terahertz waves or millimeter waves can also be used.

As shown in FIG. 20, the light arriving at the object 1901 from the first light source 1903 and the second light source 1904 is reflected once inside the object 1901 and the surface reflection component I1 reflected on the surface of the object 1901. Alternatively, it is divided into an internal scattering component I2 that scatters or multiple scatters. The surface reflection component I1 includes two components, a direct reflection component and a diffuse reflection component. The direct reflection component is a component that is reflected at a reflection angle equal to the incident angle. The diffuse reflection component is a component that is diffused and reflected due to the uneven shape of the surface.

In the present disclosure, the surface reflection component I1 reflected on the surface of the object 1901 includes these two components. Further, the internal scattering component I2 includes a component that is scattered and reflected by the internal structure near the surface. The traveling direction of the surface reflection component I1 and the internal scattering component I2 changes due to reflection or scattering.

At this time, most of the energy of the light having wavelengths of 750 nm and 850 nm applied to the head of the object 1901 is reflected on the surface of the object 1901. However, a small part of the components reach the deep part of the object 1901 while scattering, and continue to scatter, and a very small amount of energy components contain more internal scattering components and reach the forehead surface of the head again. do. A part of the light passes through the optical system 1905 and the optical filter 1906 and reaches the solid-state image sensor 1907.

Here, since it takes time to reach the deep part of the object 1901 while scattering, the signal that exposes the vicinity of the rising edge of the reflected light contains a lot of surface information of the object 1901 and exposes the vicinity of the falling edge of the reflected light. It is extremely important that the resulting signal contains a large amount of deep information on the object 1901.

The optical system 1905 in the present embodiment is for efficiently forming an image of light on the solid-state image sensor 1907, and may be a combination of a plurality of lenses or a single lens. It may also be a telecentric optical system. A fisheye lens, a wide-angle lens, or a zoom lens may be used to adjust the angle of view of the object. Further, in order to adjust the brightness, pupils may be provided before and after or in the middle of the lens.

FIG. 21 is a graph showing an example of the spectral transmittance of the optical filter 1906. As shown, the optical filter 1906 has a narrow band light having a central wavelength of 750 nm emitted from the first light source 1903 and a narrow band light having a central wavelength of 850 nm emitted from the second light source 1904. It transmits light and blocks light of other wavelengths. By arranging such an optical filter 1906, it is possible to prevent ambient light (for example, background light 1917) from entering the solid-state image sensor 1907.

The solid-state image sensor 1907 receives light emitted from the first light source 1903 and the second light source 1904 and reflected by the object 1901. The solid-state image sensor 1907 has a plurality of pixels arranged in two dimensions on the image pickup surface, and acquires two-dimensional information inside the object 1901. The solid-state image sensor 1907 can be, for example, a CCD image sensor or a CMOS image sensor.

The solid-state image sensor 1907 has an electronic shutter. The electronic shutter has a shutter width corresponding to the length of the exposure period, which is the period of one signal accumulation that converts the received light into an effective electric signal and accumulates it, and the next after the end of one exposure period. It is a circuit that controls the time until the exposure period of the above starts. In this embodiment, the state in which the electronic shutter is exposed is expressed as "OPEN", and the state in which the electronic shutter is stopped from exposure is expressed as "CLOSE". The solid-state image sensor 1907 can adjust the time from the end of one exposure period to the start of the next exposure period by an electronic shutter on a time scale of sub-nanoseconds (for example, 30 ps to 1 ns).

In an application where the object 1901 is, for example, a human forehead and information such as cerebral blood flow is detected, the attenuation rate of light inside the object 1901 is very large, and can be attenuated to, for example, about one millionth. .. Therefore, in order to detect the internal scattering component I2, the amount of light may be insufficient only by irradiating one pulse. In this case, the first light source 1903 and the second light source 1904 may emit pulsed light a plurality of times, and the solid-state image sensor 1907 may be exposed a plurality of times by an electronic shutter accordingly. According to such an operation, the sensitivity can be improved by integrating the detection signals.

FIG. 4A is a block diagram of the solid-state image sensor of this embodiment. The configuration is the same as that of the first embodiment, but the operation is different. Here, for the sake of simplification of the drawings, only 4 pixels in the vertical direction and 4 pixels in the horizontal direction are shown.

This solid-state imaging element is arranged in a matrix on a semiconductor substrate, and accumulates a plurality of photoelectric conversion units (photodiodes) 4 that convert incident light into signal charges and signal charges read from the photoelectric conversion units 4. A vertical transfer unit 5 that transfers in the column direction (vertical direction), a horizontal transfer unit 10 that transfers the signal charge transferred by the vertical transfer unit 5 in the row direction (horizontal direction), and a signal transferred by the horizontal transfer unit 10. It includes a charge detection unit 11 that outputs a charge.

Here, the solid-state image sensor is an interline transfer type CCD. For example, the vertical transfer unit 5 is a 10-phase drive in which 10 gates of vertical transfer electrodes 8 are provided around two photoelectric conversion units 4 adjacent in the vertical direction. , The horizontal transfer unit 10 is a two-phase drive. Of the vertical transfer electrodes 8, φV1 and φV3 also serve as read electrodes for the four photoelectric conversion units 4 connected to the odd-numbered vertical transfer units 5, and φV2 and φV4 are connected to the even-numbered vertical transfer units 5. It is configured to also serve as a readout electrode for the four photoelectric conversion units 4. As a result, the signal charges accumulated in the four photoelectric conversion units 4 are added to the positions represented by, for example, the signal packet 9a of the even-numbered vertical transfer units 5 when a high voltage is applied to φV1 and φV3. When a high voltage is applied to φV2 and φV4, the voltage is added to the position represented by, for example, the signal packet 9b of the even-numbered vertical transfer unit 5 and read out. After that, the signal charge on the vertical transfer unit 5 is transferred in the column direction by applying a voltage to the vertical transfer electrode 8.

Further, the photoelectric conversion unit 4 is provided with a VOD (vertical overflow drain) 12 for sweeping away the signal charge. However, in order to facilitate the understanding of the present disclosure, the VOD 12 is described in the lateral direction of the pixel surface, but is actually configured in the bulk direction of the pixel (the depth direction of the semiconductor substrate). When a high voltage is applied to the electrode φSub connected to the substrate of the VOD 12, the signal charges of all the photoelectric conversion units 4 are collectively discharged to the substrate.

22A and 22B are diagrams for explaining the operation timing and the operation principle in the cerebral blood flow measuring device according to the fourth embodiment. The operation consists of a total of six types of exposure sequences, 750 nm sequences P1 to P3 and 850 nm sequences Q1 to Q3, and two types of signals are calculated based on the amount of signal charge obtained in each exposure sequence. Here, the pulse widths (irradiation time) of the irradiation light 1910 at 750 nm and the irradiation light 1911 at 850 nm are Tp and Tp', respectively. The exposure period width of the 750 nm sequence P1 to P3 is Te, the exposure period of the 850 nm sequence Q1 to Q3 is Te', the background light component contained in the reflected light of the first light source of the 750 nm sequence P1 to P3 is BG, and the background light component is BG, and the 850 nm sequence Q1 to Q3. The background light component included in the reflected light of the second light source is BG'.

First, as shown in FIG. 22A, in the 750 nm sequence P1, the exposure period Te includes the rising vicinity of the reflected light 1912 of the irradiation light 1910 emitted from the first light source 1903, and the signal charge based on the reflected light 1912. The amount of is S0 + BG. On the other hand, in the 750 nm sequence P2, the exposure period Te includes the vicinity of the falling edge of the reflected light 1912 of the irradiation light 1910 emitted from the first light source 1903, and the amount of signal charge based on the reflected light 1912 is S1 + BG. Further, since there is no irradiation light 1910 in the 750 nm sequence P3, the amount of signal charge based on the reflected light 1912 is BG. Therefore, by subtracting BG from S0 + BG and S1 + BG, the signal charges S0 and S1 that do not depend on the background light can be taken out.

Similarly, as shown in FIG. 22B, in the 850 nm sequence Q1, the exposure period Te'includes the rising vicinity of the reflected light 1912 of the irradiation light 1911 emitted from the second light source 1904, and is a signal based on the reflected light 1912. The amount of charge is S0'+ BG'. On the other hand, in the 850 nm sequence Q2, the exposure period Te'includes the vicinity of the falling edge of the reflected light 1912 of the irradiation light 1911 emitted from the second light source 1904, and the amount of signal charge based on the reflected light 1912 is S1'+ BG. '. Further, since there is no irradiation light 1911 in the 850 nm sequence Q3, the amount of signal charge based on the reflected light 1912 is BG'. Therefore, by subtracting BG'from S0'+ BG' and S1'+ BG', the signal charges S0'and S1'independent of the background light can be extracted.

Further, in FIGS. 22A and 22B, in the 750 nm sequence P3 and the 850 nm sequence Q3, the first light source 1903 and the second light source 1904 did not irradiate, but after irradiating as shown in FIG. 22C. By delaying the start time of the exposure period by the time ΔTbg and ΔTbg'that the irradiation light component contained in the reflected light 1912 decays to a negligible level, only the background light component may be exposed.

Further, in FIGS. 22A and 22B, for convenience of explanation, each exposure sequence is shown only once, but in practice, in order to obtain a sufficient amount of signal charge from the viewpoint of S / N, as shown in FIG. 23. The 750 nm sequence P1 to P3 is _{repeated α 750 nm} × β times and the 850 nm sequence Q1 to Q3 is _{repeated α 850 nm} × β times in one frame period. Here, it is also possible to correct the level difference due to the light source power, the sensitivity of the solid-state image sensor, etc. for the 750 nm sequences P1 to P3 and the 850 nm sequences Q1 to Q3 by _{setting α 750 nm} and α _{850 nm independently.} ..

Next, the operation timing of the solid-state image sensor of FIG. 4A will be described with reference to FIGS. 24 and 7 to 12. FIG. 6 shows an example of the drive timing of the photoelectric conversion unit 4 and the vertical transfer unit 5 in the six types of exposure sequences constituting one frame period, and FIGS. 7 to 12 show addition and transfer of signal charges. Shows an image. In FIG. 24, for the sake of simplicity, the operation for β = 1 time is extracted, and each exposure sequence is repeated twice (α _{750 nm} = α _{850 nm} = 2). Further, in FIGS. 7 to 12, the directions of reading and transferring the signal charge are indicated by arrows.

First, in the 750 nm sequence P1 of FIG. 24, the voltages of φV1 and φV3 are set to the High level, and the voltages of φV2 and φV4 are set to the Low level, so that the four photoelectrics connected to V1 and V3 of the vertical transfer unit 5 in the odd number row are set. The signal charge can be read from the conversion unit 4. Here, by setting the voltage of φSub to the Low level for the exposure period Te in synchronization with the irradiation light, as shown in FIG. 7A, the signal charges 31 from the four photoelectric conversion units 4 are vertically in an odd number of rows. It is stored in the transfer unit 5. The signal charge 31 includes an irradiation light component 31a and a background light component 31b. Further, the signal charge 31 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T1 in FIG. 23, FIG. 7 (b) shows. As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 7C, the signal charge 31 is vertically transferred in the forward direction (downward in the drawing), and is in a state of being accumulated in V2 to V4.

Next, in the 850 nm sequence Q1 of FIG. 24, the voltages of φV2 and φV4 are set to the High level, and the voltages of φV1 and φV3 are set to the Low level. The signal charge can be read from the photoelectric conversion unit 4. Here, by setting the voltage of φSub to the Low level for the exposure period Te'in synchronization with the irradiation light, as shown in FIG. 8A, the signal charges 32 from the four photoelectric conversion units 4 have an even number of rows. It is stored in the vertical transfer unit 5. The signal charge 32 includes an irradiation light component 32a and a background light component 32b. Further, the signal charge 31 and the signal charge 32 are separately stored in V2 and V4 of the vertical transfer unit 5, and the voltages of φV2, φV3, and φV4 are set to the Middle level at the time T2 in FIG. As shown in 8 (b), each packet is one packet. After that, as shown in FIG. 8C, the signal charge 31 and the signal charge 32 are vertically transferred in the forward direction (downward in the drawing) and are stored in V5 and V6.

Next, in the 750 nm sequence P2 of FIG. 24, by shifting the start time of the exposure period and performing the same drive as in the 750 nm sequence P1, as shown in FIG. 9A, the signals from the four photoelectric conversion units 4 are displayed. The electric charge 33 is accumulated in the vertical transfer unit 5 in an odd number of rows. At this time, since the signal charge 31 has moved to V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 33. The signal charge 33 includes an irradiation light component 33a and a background light component 33b. Further, the signal charge 33 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T3 in FIG. 24, FIG. 9B is shown. As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 9C, the signal charge 33 is vertically transferred in the forward direction (downward in the drawing), and is in a state of being accumulated in V2 to V4.

Next, in the 850 nm sequence Q2 of FIG. 24, by shifting the start time of the exposure period and performing the same drive as in the 850 nm sequence Q1, as shown in FIG. The electric charge 34 is accumulated in the vertical transfer unit 5 in an even number of rows. At this time, since the signal charge 32 is accumulated in V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 34. The signal charge 34 includes an irradiation light component 34a and a background light component 34b. Further, the signal charge 33 and the signal charge 34 are separately stored in V2 and V4 of the vertical transfer unit 5, and by setting the voltages of φV2, φV3, and φV4 to the Middle level at the time T4 in FIG. 24, FIG. As shown in 10 (b), each packet is one packet. After that, as shown in FIG. 10C, the signal charge 33 and the signal charge 34 are vertically transferred in the forward direction (downward in the drawing), and are stored in V5 and V6.

Next, in the 750 nm sequence P3 of FIG. 24, by performing the same drive as the 750 nm sequence P1 without irradiating from the infrared light source, as shown in FIG. 11 (a), from the four photoelectric conversion units 4 The signal charge 35 of the above is stored in the vertical transfer unit 5 of an odd number of rows. At this time, since the signal charge 33 has moved to V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 35. Further, the signal charge 35 is separately stored in V1 and V3 of the vertical transfer unit 5, and by setting the voltages of φV1, φV2, and φV3 to the Middle level at the time T5 in FIG. As shown, it becomes one packet, and the signal charges from the four photoelectric conversion units 4 are added. After that, as shown in FIG. 11C, the signal charge 35 is vertically transferred in the forward direction (downward in the drawing), and is in a state of being accumulated in V2 to V4.

Finally, in the 850 nm sequence Q3 of FIG. 24, by performing the same drive as in the 850 nm sequence Q1 without irradiating from the infrared light source, as shown in FIG. 12 (a), from the four photoelectric conversion units 4 The signal charge 36 of the above is stored in the vertical transfer unit 5 of an even number of rows. At this time, since the signal charge 34 is accumulated in V5 and V6 by the previous forward transfer, it is not mixed with the signal charge 36. Further, the signal charge 35 and the signal charge 36 are separately stored in V2 and V4 of the vertical transfer unit 5, and by setting the voltages of φV2, φV3, and φV4 to the Middle level at the time T6 in FIG. 24, FIG. As shown in 10 (b), each packet is one packet. After that, as shown in FIG. 12C, the signal charge 36 is vertically transferred from the signal charge 31 in the opposite direction (upward in the drawing), and the signal charge 31 and the signal charge 32 are accumulated in V1 to V3. .. As a result, the 750 nm sequence P1 can be executed again, and the operation of repeating β times is performed.

In this way, highly accurate S0 and S0'that include a large amount of surface information of the object 1901 and highly accurate S1 and S1' that include a large amount of deep information of the object 1901 are output from the solid-state image sensor. ..

In the present embodiment, the example of the CCD type image sensor has been described. However, whether the solid-state image sensor 1907 is a CMOS type or a single photon counting type element, the EMCCD is an amplification type image sensor. It may be an ICCD.

The control arithmetic unit 1908 may be, for example, a combination of a microprocessor and a memory, a hardware logic circuit, or an integrated circuit such as a microcontroller incorporating the processor and the memory.

Further, the early-stage hard logic circuit may be, for example, a programmable logic device (PLD) such as a digital signal processor (DSP) or a field programmable gate array (FPGA).

The control arithmetic unit 1908 also controls blinking instructions to the first light source 1903 and the second light source 1904, imaging instructions to the solid-state image sensor 1907, and calculation processing of signals from the solid-state image sensor.

The signal processing unit of the control calculation device 1908 is a circuit that processes the image signal output from the solid-state image sensor 1907. In the present embodiment, the time of skin and cerebral blood flow is based on the signal output from the solid-state image sensor 1907. Generate moving image data showing changes. Not limited to such moving image data, other information may be generated. For example, biometric information such as blood flow, blood pressure, blood oxygen saturation, or heart rate in the brain may be generated by synchronizing with other devices.

It is known that there is a close relationship between changes in cerebral blood flow or components in the bloodstream (eg, hemoglobin) and human neural activity. For example, changes in nerve cell activity in response to changes in human emotions change cerebral blood flow or components in the blood. Therefore, if biological information such as changes in cerebral blood flow or changes in blood components can be measured, the psychological state of the subject can be estimated. The psychological state of the subject is, for example, mood (eg, pleasant, unpleasant), emotion (eg, relief, anxiety, sadness, resentment, etc.), health status (eg, energy, malaise), temperature sensation (eg, hot). , Cold, sultry) and so on. Derived from this, indicators showing the degree of brain activity, such as proficiency, proficiency, and concentration, are also included in the psychological state. The signal processing unit of the control arithmetic unit 1908 may estimate the psychological state such as the concentration ratio of the subject based on the change of the cerebral blood flow rate and output a signal indicating the estimation result.

The signals S0 and S0'transmitted from the solid-state image sensor 1907 to the control calculation device 1908 and exposing the vicinity of the rising edge of the reflected light accumulated in frame units mainly include information on changes in blood flow in the skin, and the reflected light. The signals S1 and S1'exposing the vicinity of the fall of the light mainly include information on changes in blood flow between the skin and the brain. Therefore, in order to extract only the information on the blood flow change in the brain, the information on the blood flow change in the skin and the information on the blood flow change in the brain may be separated by using these signals.

As shown in FIG. 25, by using a light source having two wavelengths of 750 nm and 850 nm, _{the molecular absorption count of oxygenated hemoglobin (solid line of HbO 2} ) and deoxygenated hemoglobin (dotted line of Hb) at infrared rays is at the boundary of 805 nm. By irradiating two wavelengths around 805 nm and analyzing the reflected light, it is possible to obtain the amount of change between oxygenated hemoglobin and deoxygenated hemoglobin. Figure 25 is taken from "Chapter 3 Light that Contributes to Healthy Living 3 Non-Invasive Biopsy Using Light" on the website of the Ministry of Education, Culture, Sports, Science and Technology.

The present inventors have focused on the fact that both changes in skin blood flow and changes in cerebral blood flow are associated with the expansion and contraction activities of blood vessels or capillaries, and the blood vessel distributions of the skin and the brain are different. It was considered that there is no correlation between the distribution of flow changes and the distribution of cerebral blood flow changes. Based on this idea, the image of the blood flow change in the skin and the blood flow change in the brain are calculated by using the signals projected on the image created from the signal of the rising component of the reflected light and the image created from the signal of the falling component, respectively. Separated from the image of. The details will be described below.

The rising component signal and the falling component signal each contain information on changes in skin blood flow and information on changes in cerebral blood flow in different ratios, and are expressed as the following theoretical formula (1).

Here, a, b, c, d, e, f, g, and h indicate coefficients, and Ss750 nm and Sb750 nm indicate components of changes in skin blood flow and components of changes in cerebral blood flow when irradiated at a wavelength of 750 nm. , Ss850 nm and Sb850 nm indicate a component of skin blood flow change and a component of cerebral blood flow change when irradiated at a wavelength of 850 nm. That is, the left side of the above equation is a known pixel signal value detected by the solid-state image sensor, and the right side is an unknown number.

For example, assuming that the rising component consists of the component of skin blood flow change, a = 1, b = 0, e = 1, f = 0, Ss750nm = S0, Ss850nm = S0'are substituted, and c, d, g And h, Sb750 nm, and Sb850 nm are unknown. At this time, there are many unknown combinations such that the left side and the right side are equal. Here, taking advantage of the fact that there is no correlation between the distribution of changes in skin blood flow and the distribution of changes in cerebral blood flow, there is a phase relationship between all pixel components of changes in skin blood flow Ss750 nm and Ss850 nm and changes in cerebral blood flow Sb750 nm and Sb850 nm. The values of the coefficients c and d, g and h, Sb750 nm, and Sb850 nm so that the number is closest to 0 are extracted.

The signals of Ss750 nm and Sb750 nm thus obtained irradiate the changes in blood flow of the skin and the blood flow of the brain when irradiated with light having a wavelength of 750 nm, respectively, and the signals of Ss850 nm and Sb850 nm irradiate light with a wavelength of 850 nm, respectively. It shows the change in blood flow in the skin and the change in blood flow in the brain.

This time, the explanation is made on the premise that the rising component is equal to the component of the change in skin blood flow, but since it is possible that some cerebral blood flow components are also included, a and b and e and f may be treated as variables.

In equations (1) and (2), the correlation between the distribution of changes in skin blood flow and changes in cerebral blood flow was used to obtain the coefficients, but multivariate analysis such as independent component analysis may also be used. Further, the optimum a, b, c, d and e, f, g, h capable of separating the change in skin blood flow and the change in cerebral blood flow may be obtained in advance using a phantom having optical characteristics similar to those of humans. ..

In addition, changes in skin blood flow and changes in cerebral blood flow are considered to change continuously over time. Therefore, since it is considered that the correlation between frames is high, by obtaining the correlation between frames and the motion vector for each pixel, Ss750 nm and Sb750 nm and Ss750 nm and Sb750 nm that satisfy the equations (1) and (2) can be accurately obtained. You may ask.

By performing such image calculation, the oxygenated hemoglobin change in the skin blood flow and the oxygenated hemoglobin change in the skin blood flow can be obtained as an image from the component of the change in skin blood flow and the component of the change in cerebral blood flow contained in different ratios of the rising component and the falling component. It is possible to separate and output the deoxidized hemoglobin change, the oxygenated hemoglobin change in the cerebral blood flow, and the deoxygenated hemoglobin change.

In this embodiment, light having two kinds of wavelengths is used, but light having three kinds or more wavelengths may be used.

By the above operation, six kinds of signal charges can be independently accumulated in the vertical transfer unit in one frame period. Further, since the signal charges of the four photoelectric conversion units are added, the sensitivity is substantially doubled as compared with the conventional method of adding the signal charges of the two photoelectric conversion units, and the type of signal charge to be accumulated. Can be offset by the decrease in each charge due to the increase in. Therefore, it is possible to separately output the change in skin blood flow and the change in cerebral blood flow within one frame, and it is possible to display it as a moving image if it is operated continuously.

Since this image pickup device captures changes in the scattered light inside the head, there is little need to increase the two-dimensional resolution of the image, but the scattered light inside the head returns to the surface again and reaches the solid-state image sensor. Since extremely weak light detection is required, it can be said that the method of increasing the sensitivity by sacrificing the two-dimensional resolution is an extremely rational method.

As described above, in the image pickup apparatus according to the fourth embodiment, the period from the start of the increase in the intensity of the reflected light from the object to the end of the increase is defined as the rising period, and the period from the object to the end of the increase is defined as a rising period. When the period from the start of the decrease in the intensity of the reflected light to the end of the decrease is defined as the fall period, the exposure period of the solid-state image sensor includes the fall period in the m types of exposure sequences. The period is set to match either the first period including at least a part of the rising period and the second period not including the rising period and including a part of the falling period.

Here, the m types of exposure sequences may include exposure sequences that irradiate light having irradiation wavelengths different from each other.

Here, the imaging device is an observation device inside a light scatterer to which time-resolved imaging is applied, the photoelectric conversion unit is arranged on a matrix, and the m types of exposure sequences have first to third long wavelengths. The irradiation wavelength in the first to third long wavelength sequences includes the sequence and the first to third short wavelength sequences, and is longer than the first to third short wavelength sequences, and the first length is longer. The wavelength sequence and the first short wavelength sequence each have the first period as the exposure period, and the second long wavelength sequence and the second short wavelength sequence each have the second period as the exposure period. The third long-wavelength sequence and the third short-wavelength sequence each expose background light containing no reflected light component due to irradiation from the light source, and the first to third long-wavelength sequences and the third short-wavelength sequence are exposed. The first to third short wavelength sequences are repeated a plurality of times within one frame period, and the n photoelectric conversion units are four photoelectric conversion units composed of 2 rows and 2 columns, and the first to third photoelectric conversion units are formed. Even if it is provided with an arithmetic unit that calculates the structure and state inside the light scatterer by using each of the signal charge accumulated in the long wavelength sequence and the signal charge accumulated in the first to third short wavelength sequences. good.

Here, in the third long wavelength sequence and the third short wavelength sequence, the period after the irradiation from the light source is performed and the reflected light is extinguished may be the exposure period.

Here, the arithmetic unit uses the results of imaging from the first to third long wavelength sequences and the results of imaging from the first to third short wavelength sequences to obtain the structure of the deep part of the light scattering body and the results. The state and the structure and state of the surface layer of the light scattering body may be obtained.

In addition, various exposure sequences in the first to fourth embodiments may be combined. For example, long exposure sequences L1 to L3, short exposure sequences S1 to S3, strong exposure sequences K1 and K2, weak exposure sequences J1 and J2, background exposure sequences B0, 750 nm sequences P1 to P3, and 850 nm sequences Q1 to Q3 all or one. The parts may be combined.

FIG. 26 is a diagram illustrating an operation example in which each exposure sequence of the first to fourth embodiments is combined. In the figure, all of the long exposure sequences L1 to L3, the short exposure sequences S1 to S3, the strong exposure sequences K1 and K2, the weak exposure sequences J1 and J2, the background exposure sequence B0, the 750 nm sequence P1 to P3, and the 850 nm sequence Q1 to Q3. An operation example in which is combined is shown.

Since the present disclosure can efficiently obtain a high-precision distance image, it can be particularly used for an imaging device such as a distance measuring camera. In addition, since it is possible to efficiently obtain internal information of an object without contact, it can be used for biometric measurement, material analysis, and the like.

4 Photoelectric conversion unit (PD)
31-36 Signal charges 31a to 34a Irradiation light components 31b to 34b Background light component 101 Object (subject)
103 Infrared light source 106 Solid-state image sensor 107 Control arithmetic unit

Claims (23)

It is an image pickup device
A light source that irradiates an object with light,
A solid-state image sensor that exposes the reflected light from the object and accumulates it as a signal charge.
A control unit that controls irradiation from the light source and exposure of the solid-state image sensor is provided.
The solid-state image sensor
A plurality of photoelectric conversion units that convert the reflected light from the object into the signal charge, and
A plurality of charge storage units for accumulating the signal charges, and
The imaging device performs m types of exposure sequences (m is an integer of 4 or more) for controlling irradiation and exposure within one frame period.
A charge storage unit is exclusively assigned to the m types of exposure sequences,
An imaging device that stores the signal charges obtained from n (n is an integer of 3 or more) of the photoelectric conversion units in the charge storage unit in at least one of the m types of exposure sequences. The imaging device according to claim 1, wherein the imaging device repeats at least one type of exposure sequence within one frame period. In the imaging device, the combination of the n photoelectric conversion units is associated with one charge storage unit, and the signal charge obtained from the n photoelectric conversion units is converted into the corresponding charge storage unit. The imaging apparatus according to claim 1 or 2, which accumulates in. The imaging apparatus is any one of claims 1 to 3 in which the signal charges obtained from the n photoelectric conversion units are added and stored in the corresponding charge storage units in the at least one type of exposure sequence. The imaging device according to the section. The imaging according to any one of claims 1 to 4, wherein the m types of exposure sequences differ from each other in at least one of the irradiation intensity from the light source, the irradiation time, the irradiation wavelength, and the exposure period of the solid-state image pickup device. Device. The imaging device according to any one of claims 1 to 4, wherein the imaging device changes the combination of the n photoelectric conversion units for each exposure sequence of k types (k is an integer of 1 or more). The image pickup device according to claim 2, wherein the image pickup device sets the number of repetitions of the m types of exposure sequences based on the irradiation intensity from the light source, the irradiation time, the irradiation wavelength, and the exposure period of the solid-state image pickup device. .. The imaging device according to any one of claims 1 to 6, wherein the imaging device changes the combination of the n photoelectric conversion units that add the signal charges for each frame. The m is 5 or 6 and
The plurality of photoelectric conversion units are arranged on a matrix, and the plurality of photoelectric conversion units are arranged in a matrix.
The imaging device repeats the exposure sequence consisting of 5 or 6 types a plurality of times in one frame period, respectively.
The n photoelectric conversion units are a combination of four photoelectric conversion units arranged in 2 rows and 2 columns.
The imaging device according to any one of claims 1 to 8. The imaging device is a TOF (Time Of Flight) type ranging device.
A claim that the imaging device includes a distance calculation unit that calculates a distance to an object using the signal charge accumulated in the charge storage unit in at least two types of exposure sequences among the m types of exposure sequences. 5. The imaging device according to 5. The m types of exposure sequences include a long exposure sequence and a short exposure sequence.
In the long exposure sequence, the irradiation time from the light source and the exposure period of the solid-state image sensor are set longer than the short exposure sequence.
The imaging device according to claim 10, wherein the distance calculation unit calculates at least two types of distances to an object based on the m types of exposure sequences. The image pickup device is a TOF type ranging device, and is
The photoelectric conversion unit is arranged on a matrix and
The m types of exposure sequences include first to third long exposure sequences and first to third short exposure sequences.
The irradiation time from the light source and the exposure period of the solid-state image sensor in the first to third long exposure sequences are longer than those in the first to third short exposure sequences, respectively.
The exposure period in the second long exposure sequence is different from the exposure period in the first long exposure sequence.
The exposure period in the second short exposure sequence is different from the exposure period in the first short exposure sequence.
The third long exposure sequence and the third short exposure sequence expose background light that does not contain a reflected light component due to irradiation from the light source.
The first to third long exposure sequences and the first to third short exposure sequences are repeated a plurality of times within one frame period, respectively.
The n photoelectric conversion units are four photoelectric conversion units having 2 rows and 2 columns.
Two types of distances to the object are calculated using each of the signal charge accumulated in the first to third long exposure sequences and the signal charge additionally accumulated in the first to third short exposure sequences. The imaging apparatus according to any one of claims 1 to 9, further comprising a distance calculation unit. The imaging apparatus according to claim 12, wherein in the third long-distance sequence and the third short-distance sequence, the period after irradiation from the light source is performed and the reflected light is extinguished is defined as the exposure period. A distance that corrects the distance calculated from the long exposure sequence or the first to third long exposure sequences by using the distance calculated from the short exposure sequence or the first to third short exposure sequences. The imaging apparatus according to claim 11 or 12, further comprising a correction unit. A part of the distance calculated from the long exposure sequence or the first to third long exposure sequences,
11. Imaging device. When the exposure period in the short exposure sequence or the first to third short exposure sequences is defined as the short exposure period,
The short exposure period is changed for each frame.
By changing the short exposure period frame by frame, the distance range calculated from the short exposure sequence or the first to third short exposure sequences is the long exposure sequence or the first to third short exposure sequences. The imaging apparatus according to claim 11 or 12, wherein scanning is performed after being restricted so as to overlap a part of a distance range calculated from a long exposure sequence. The 11 or 12 claim, wherein the exposure period in the short exposure sequence or the first to third short exposure sequences is set based on the distance calculated from the first to third long exposure sequences of the past frames. Imaging device. The image pickup device is a TOF type ranging device, and is
The photoelectric conversion unit is arranged on a matrix and
The m-type exposure sequence includes a first strong exposure sequence, a second strong exposure sequence, a first weak exposure sequence, a second weak exposure sequence, and a background exposure sequence.
The irradiation intensity of the light source in the first strong exposure sequence and the second strong exposure sequence is stronger than that of the first weak exposure sequence and the second strong exposure sequence.
The exposure period in the second strong exposure sequence is different from the exposure period in the first strong exposure sequence.
The exposure period in the second weak exposure sequence is different from the exposure period in the first weak exposure sequence.
The background exposure sequence exposes the background light that does not include the reflected light.
The first strong exposure sequence, the second strong exposure sequence, the first weak exposure sequence, the second weak exposure sequence, and the background exposure sequence are repeated a plurality of times within one frame period, respectively.
The n photoelectric conversion units are four photoelectric conversion units having 2 rows and 2 columns.
Either one of the signal charge in the first strong exposure sequence and the second strong exposure sequence and the signal charge in the first weak exposure sequence and the second weak exposure sequence for each of the n photoelectric conversion units. The image pickup apparatus according to any one of claims 1 to 9, further comprising a distance calculation unit that calculates a distance using the signal charge in the background exposure sequence. The period from the start of the increase in the intensity of the reflected light from the object to the end of the increase is defined as the rising period, and the period from the start of the decrease in the intensity of the reflected light from the object to the end of the decrease is set. When the period is the falling period,
In the m types of exposure sequences, the exposure period of the solid-state image sensor is
A first period that does not include the fall period but includes at least a part of the rise period,
The imaging apparatus according to any one of claims 1 to 9, wherein the period is set to match any of the second period including a part of the falling period without including the rising period. The imaging device according to claim 19, wherein the m types of exposure sequences include exposure sequences that irradiate light having irradiation wavelengths different from each other. The image pickup device is an observation device inside a light scatterer to which time-resolved imaging is applied.
The photoelectric conversion unit is arranged on a matrix and
The m types of exposure sequences include first to third long wavelength sequences and first to third short wavelength sequences.
The irradiation wavelength in the first to third long wavelength sequences is longer than that in the first to third short wavelength sequences.
The first long wavelength sequence and the first short wavelength sequence each have the first period as an exposure period.
The second long wavelength sequence and the second short wavelength sequence each have the second period as the exposure period.
The third long wavelength sequence and the third short wavelength sequence each expose background light that does not contain a reflected light component due to irradiation from the light source.
The first to third long wavelength sequences and the first to third short wavelength sequences are repeated a plurality of times within one frame period, respectively.
The n photoelectric conversion units are four photoelectric conversion units having 2 rows and 2 columns.
Using each of the signal charges accumulated in the first to third long wavelength sequences and the signal charges accumulated in the first to third short wavelength sequences, the structure and state inside the light scatterer are calculated by calculation. The imaging apparatus according to claim 19 or 20, further comprising a calculation unit. The imaging apparatus according to claim 21, wherein in the third long wavelength sequence and the third short wavelength sequence, the period after irradiation from the light source is performed and the reflected light is extinguished is defined as the exposure period. The arithmetic unit uses the results of imaging from the first to third long wavelength sequences and the results of imaging from the first to third short wavelength sequences to obtain the structure, state, and light of the deep part of the light scattering body. The imaging apparatus according to claim 21 or 22, wherein the structure and state of the surface layer portion of the scattering body are obtained.

Images