JP7080673B2 - Imaging device, imaging system, moving object - Google Patents

Description

The present invention relates to an image pickup apparatus, an image pickup system, and a moving body.

In recent years, various semiconductor devices have been used. As the operating period of the semiconductor device becomes longer, the function of the semiconductor device may deteriorate.

For example, in the technique described in Patent Document 1, a semiconductor device including a life prediction unit that acquires the degree of deterioration of a functional unit included in the semiconductor device and predicts the life of the semiconductor device from the degree of deterioration is described. Further, in Patent Document 1, when the threshold voltage becomes high due to deterioration of the semiconductor element, the power supply voltage Vdd is increased. This is said to be able to extend the life of the functional unit.

Japanese Unexamined Patent Publication No. 2017-173242

In the technique described in Patent Document 1, no study is made on the life prediction in the image pickup apparatus.

The present invention provides an imaging device capable of predicting the remaining life after performing an actual imaging operation.

The present invention has been made in view of the above problems, and one embodiment thereof is arranged over a plurality of rows and a plurality of columns, each comprising a plurality of pixels having a photoelectric conversion unit, and the plurality of pixels. The image sensor has an image pickup device that reads out each signal of the image sensor and a lifespan prediction unit that acquires the remaining life information of the image pickup element. The image pickup element performs the image pickup operation a plurality of times, and the lifespan prediction unit The image pickup apparatus is characterized in that the remaining life information is acquired after the plurality of image pickup operations are performed.

INDUSTRIAL APPLICABILITY The present invention provides an imaging device capable of predicting the remaining life after performing an actual imaging operation.

The figure which showed the structure of the semiconductor device Flow chart showing the operation of semiconductor devices The figure which showed the relationship between the operating time and the life of a semiconductor device. The figure which showed the structure of the image pickup apparatus Pixel equivalent circuit diagram Block diagram of column circuit The figure which showed the operation of the image pickup apparatus The figure which showed the structure of the image pickup apparatus The figure which showed the structure of the image pickup apparatus The figure which shows the structure of the image pickup system Diagram showing the configuration of a moving body The figure which shows the operation of the image pickup system

Hereinafter, each embodiment will be described with reference to the drawings. In the following description, unless otherwise specified, the transistor is assumed to be an N-type transistor. However, the examples described below are not limited to N-type transistors, and P-type transistors may be used as appropriate. In that case, the gate, source, and drain voltages of the transistor can be appropriately changed with respect to the description in the examples. For example, in the case of a transistor that operates as a switch, the low level and the high level of the voltage supplied to the gate may be reversed with respect to the description in the embodiment.

(Example 1)
First, before the description of the embodiment of the image pickup apparatus, the embodiment of the semiconductor device will be described in order to help the technical understanding regarding the remaining life prediction. In subsequent examples, the embodiment of the image pickup apparatus will be described.

FIG. 1 is a diagram showing an example of a configuration of a semiconductor device according to this embodiment. The semiconductor device 150 includes a semiconductor element 100 and a life prediction unit 109. The semiconductor element 100 is a circuit in which a large number of functions such as light receiving, light emitting, AD conversion processing, signal processing, communication processing, storage unit, and detection unit are integrated. Hereinafter, the configuration and operation of the semiconductor element 100 will be described.

The first circuit block 101, the second circuit block 102, and the Nth circuit block 103 are circuits having a large number of functions such as light reception, light emission, AD conversion processing, signal processing, communication processing, storage unit, and detection unit, respectively. ..

The voltage control unit 104 supplies the voltage value set by the signal from the drive mode selection unit 105, which is the drive control unit, to the first circuit block 101, the second circuit block 102, and the Nth circuit block 103.

When the voltage value supplied by the voltage control unit 104 is zero, all or a part of the first circuit block 101, the second circuit block 102, and the Nth circuit block 103 is turned off.

The frequency control unit 106 sets the cycle in which the signal generation unit 107, which generates the control signals supplied to the first circuit block 101, the second circuit block 102, and the Nth circuit block 103, generates the signal, as the drive mode generation unit. It is changed by the signal from 105.

The remaining life information is input to the drive mode selection unit 105 from the life prediction unit 109, which will be described later, via the life information receiving unit 108. The voltage control unit 104 and the frequency control unit 106 are controlled according to the information. The drive mode generation unit 105 can also generate a control signal capable of stopping all or part of the first circuit block 101, the second circuit block 102, and the Nth circuit block 103.

The life prediction unit 109 executes the life prediction of the semiconductor element 100 based on the information indicating the latest operating state of the semiconductor element 100 or the history information related to the operating state, and obtains the remaining life information of the semiconductor element 100. Here, the operating state represents a state when the semiconductor element 100 is operated, such as an operating voltage and an operating frequency when the semiconductor element 100 is operated, or a state generated as a result of an operation such as temperature. The information indicating the latest operating state of the semiconductor element 100 and the history information related to the operating state are the operating information indicating the operating state of the semiconductor element 100.

Next, an example of the life prediction method of the semiconductor element 100 by the life prediction unit 109 will be described. In general, the life of a semiconductor device is limited by the deterioration of the performance of the transistors, wirings, oxide films, etc. that make up the semiconductor device over time. The main performance degradation mechanisms of semiconductor devices include electromigration (EM) and dielectric breakdown (TDDB). In addition, as other performance degradation mechanisms, there are characteristic fluctuations due to hot carrier injection (HC) and negative bias temperature instability (Negative Bias Temperature Instability; NBTI).

The remaining life time LT (EM) due to the performance deterioration factor being EM is calculated by the following formula (1).
LT (EM) = A × J ⁻ⁿ × exp (Ea / kT) ・ ・ ・ (1)

Here, A is a coefficient depending on the manufacturing process, Ea is activation energy, J is the current density of the wiring, n is the current acceleration coefficient, k is the Boltzmann constant, and T is the wiring temperature.

The remaining life time LT (TDDB) due to TDDB as a factor of performance deterioration is calculated by the following formula (2).
LT (TDDB) = A x exp (-γVg x Vg) x exp (Ea / kT) ... (2)

Here, Vg is the voltage applied to the gate of the transistor, and γVg is the voltage acceleration coefficient.

The remaining life time LT (HC) due to HC as a factor of performance deterioration is calculated by the following formula (3).
LT (HC) = A x Isub- ^m x exp (Ea / kT) ... (3)

Here, Isub is the maximum substrate current flowing through the semiconductor substrate, and m is a coefficient depending on the substrate current.

The remaining life time LT (NBTI) due to NBTI as a factor of performance deterioration of the semiconductor element 100 is calculated by the following formula (4).
LT (NBTI) = A x Vg ⁿ x exp (Ea / kT) ... (4)

The life prediction unit 109 measures all or a part of the coefficients of the equations (1) to (4) and calculates the remaining life time of the semiconductor element 100. The life prediction unit 109 may be inside the semiconductor element 100.

Hereinafter, an example of the operation of the present invention will be described with reference to the flowchart of FIG. 2 and the timing chart of FIG. First, an example of the operation corresponding to the timing chart of FIG. 3A will be described. In FIG. 3A, the relationship between the operating time of the semiconductor element 100 and the remaining life of the semiconductor element 100 is shown by a solid line.

In S200, the drive mode selection unit 105 acquires the first life threshold value L1 and the second life threshold value L2 of the semiconductor element 100. The first life threshold value L1 and the second life threshold value L2 define the threshold value for the remaining life of the semiconductor element 100, and L1> L2> 0.

In S201, the life prediction unit 109 calculates the remaining life Ti of the semiconductor element 100 at time i.

In S202, the drive mode selection unit 105 compares the first life threshold value L1 with the remaining life Ti at time i, and if Ti> L1, the process proceeds to S203, and if Ti ≦ L1, the process proceeds to S204. Proceed.

FIG. 3A is a case where Ti> L1.

In S203, the process of continuing the operation of the semiconductor element 100 is performed for an arbitrary period Δt1, and then the process returns to process 201.

In the above flow, it is shown that the remaining life of the semiconductor element 100 is longer than the first life threshold value L1, and since the remaining life of the semiconductor element 100 has a margin, the driving state of the semiconductor element 100 is changed. Not done.

Next, the case of FIG. 3B, in which the operating time has elapsed from FIG. 3A, will be described.

In FIG. 3B, the relationship between the operating time of the semiconductor element 100 and the remaining life of the semiconductor element 100 is shown by a solid line. Further, at time i, the predicted remaining life of the semiconductor element 100 with respect to the operating time of the semiconductor element 100 predicted by the life prediction unit 109 is represented by a broken line.

In FIG. 3B, it is the case of Ti ≦ L1.

In S204, the life prediction unit 109 predicts the remaining life Lt when the set operation time t of the semiconductor element 100 has elapsed. The set operation time is, for example, the time until the life of the semiconductor element 100 estimated in advance is reached, and if the operation exceeds this time, the operation of the semiconductor element 100 is not guaranteed.

In S205, it is compared whether or not the remaining life Lt when the set operation time t of the semiconductor element 100 has elapsed is larger than 0, and if Lt> 0, the process proceeds to S203, and if Lt ≦ 0, the process is performed. Proceed to S206. In FIG. 3B, it is the case where Lt> 0.

When Lt> 0, it means that the semiconductor element 100 does not reach the end of its life even if the normal drive mode is continued until the set operation time t. Therefore, the drive mode selection unit 105 does not change the drive mode of the semiconductor element 100.
The process proceeds to S203, and after performing the process of continuing the operation of the semiconductor element 100 in the normal drive mode for an arbitrary period Δt1, the process returns to S201.

As described above, in the case of FIG. 3B, the normal drive mode is continued until the drive mode setting operation time t of the semiconductor element 100.

Next, the case of FIG. 3C will be described.

In the case of FIG. 3C, if the operation in the normal drive mode is continued after the time i, the life of the semiconductor element 100 is reached before the set operation time t is reached. That is, in the process of S205, Lt> 0 is determined as No.

In FIG. 3C, the relationship between the operating time of the semiconductor element 100 and the remaining life of the semiconductor element 100 is shown by a solid line. Further, at time i, the predicted remaining life of the semiconductor element 100 with respect to the operating time of the semiconductor element 100 predicted by the life prediction unit 109 is represented by a broken line. Further, at time i, the drive state of the semiconductor element 100 is changed, and the relationship between the remaining life of the semiconductor element 100 and the operating time of the semiconductor element 100 in the changed drive state is represented by a dashed line.

In S205 of FIG. 2, in the case of FIG. 3C, it is determined as No. Therefore, the process proceeds to S206.

In S206 of FIG. 2, the drive mode selection unit 105 controls the operation of the voltage control unit 104 and / or the frequency control unit 106 so that the remaining life of the semiconductor element 100 is longer than the remaining life at time i. Select a life extension drive mode.

Next, an example of a method of selecting the life extension drive mode of the semiconductor element 100 by the drive mode selection unit 105 will be described. As described in the equations (1) to (4) of the first embodiment, the life of the semiconductor device is the current density J of the wiring, the wiring temperature T, the voltage Vg applied to the gate of the transistor, and the maximum substrate current Isub flowing through the semiconductor substrate. Limited by. The drive mode selection unit 105 controls both the voltage control unit 104 and the frequency control unit 106 or one of them so that the remaining life of the semiconductor element 100 is extended. Specifically, the current density J, the wiring temperature T, the voltage Vg applied to the gate of the transistor, the maximum substrate current Isub flowing through the semiconductor substrate, or a part of them are all smaller than the normal drive mode. Change the operating status so that. It can be said that this change in the operating state is a change that suppresses the driving ability for driving the semiconductor element 100. For example, the voltage control unit 104 reduces the absolute value of the power supply voltage supplied to the first circuit block 101, the second circuit block 102, and the Nth circuit block 103 (typically, the power supply voltage is reduced). ). Thereby, the current density J, the wiring temperature T, and the voltage Vg can be lowered. Further, the frequency control unit 106 can reduce the wiring temperature T and the substrate current Isub by delaying the cycle in which the signal generation unit 107 generates a signal. At this time, the drive mode selection unit 105 controls the voltage control unit 104 and the frequency control unit 106 within the range in which the operation of the semiconductor element 100 can be continued.

The normal drive mode is a mode in which the semiconductor element 100 is driven under the first condition. The life extension drive mode is a mode in which the semiconductor element 100 is driven under the second condition of extending the life of the semiconductor element 100 as compared with the case where the semiconductor element 100 is continuously driven in the normal drive mode.

By changing to the life extension drive mode, some functions of the semiconductor element 100 may be stopped and the operating speed of the semiconductor element 100 may be lowered as compared with the normal drive mode. However, in the life extension drive mode, when the operation of the semiconductor element 100 can be continued and the load applied to the circuit of the semiconductor element 100 is reduced as compared with the normal drive mode, the remaining life is continued in the normal drive mode. Can be extended than.

In the case of the life extension drive mode, the processing amount of the signal processing of the semiconductor element 100 may be lower than in the case of the normal drive mode. The processing amount of this signal processing can be said to be, for example, the amount of signal that the semiconductor element 100 performs signal processing per unit time. This example includes an example in which one or a plurality of the operating frequency, drive voltage (power supply voltage), and current supply amount by the current source of the semiconductor element 100 are made smaller in the life extension drive mode than in the normal drive mode. Further, as another example, in the normal drive mode, signal processing is performed in the first period. Then, in the life extension drive mode, the signal processing is performed in the second period shorter than the first period, and the signal processing is not performed in the period included in the first period and not included in the second period. , Or a drive that reduces the processing amount of signal processing as compared with the second period.

The example of FIG. 3C shows a case where the remaining life of the semiconductor element 100 operating in the life extension drive mode is longer than the second life threshold value L2, and further operation is possible in the normal drive mode.

In S207, the life prediction unit 109 calculates the remaining life Lj of the semiconductor element 100 at the time j.

In S208, the drive mode selection unit 105 compares the second life threshold value L2 with the remaining life Lj at the time j, and if Lj> L2, the process proceeds to S209, and if Lj ≦ L2, the process proceeds to the process S210. .. FIG. 3C describes the case where Lj> L2.

The state of Lj> L2 indicates that the remaining life of the semiconductor element 100 operating in the life extension drive mode is shorter than the first life threshold L1 but longer than the second life threshold L2. .. Therefore, with respect to the set operation continuation period, in the life extension drive mode, there is a sufficient margin for the remaining life of the semiconductor element 100. Therefore, the semiconductor element 100 can continue to operate until the set operation continuation period even if it is operated for a further period in the normal drive mode instead of the life extension drive mode. Therefore, the drive state of the semiconductor element 100 is returned to the normal drive mode before changing to the life extension drive mode. Then, the process proceeds to S203, and the operation of the semiconductor element 100 is continued in the normal drive mode for an arbitrary period Δt1. After that, the process returns to S201.

Next, an example of FIG. 3D will be described.

In FIG. 3D, the predicted remaining life of the semiconductor element 100 with respect to the operating time of the semiconductor element 100 predicted by the life prediction unit 109 at time i is represented by a broken line. Further, at time i, the drive state of the semiconductor element 100 is changed, and the relationship between the remaining life of the semiconductor element 100 and the operating time of the semiconductor element 100 in the changed drive state is represented by a dashed line.

In S208, the drive mode selection unit 105 compares the second life threshold value L2 with the remaining life Lj at the time j. FIG. 3D shows the case of Li ≦ L2.

In S210, the life prediction unit 109 calculates the remaining life Lk of the semiconductor element 100 at time k.

In S211 the drive mode selection unit 105 compares the second life threshold value L2 with the remaining life Lk at time k, and if Lk> 0, the process proceeds to S212. On the other hand, if Lk ≦ 0, the process proceeds to S213. In this embodiment, the case where the process proceeds to S212 with Lk> 0 will be described.

The state of Lk> 0 is a case where the remaining life of the semiconductor element 100 operating in the life extension drive mode is shorter than the first life threshold L2. Although the remaining life of the semiconductor element 100 is approaching 0, since it indicates that it is longer than 0, the process proceeds to S212, and the process of continuing the operation of the semiconductor element 100 for an arbitrary period Δt2 is performed. Then, the process returns to S210. At this time, since the remaining life of the semiconductor element 100 is shorter than the second life threshold L2, the period Δt2, which is the interval for obtaining the remaining life Lk, is the remaining life Li when the remaining life is longer than the first life threshold L1. The time is shorter than the period Δt1 which is the interval for obtaining. As a result, it is possible to accurately detect the time during which the normal operation of the semiconductor element 100 is not guaranteed.

Next, an example of FIG. 3 (e) will be described.

FIG. 3 (e) describes the case where Lk ≦ 0.

The state of Lk ≦ 0 indicates that the remaining life Lk of the semiconductor element 100 is 0 or less and the operation is not guaranteed. The process proceeds to S213 to notify the outside that the semiconductor element 100 is in a state where the operation is not guaranteed. As a method of this notification, there is an example of notifying the user of the semiconductor element 100 through a display or the like (not shown in FIG. 1). As another example, there is an example of notifying a monitoring system that monitors the semiconductor element 100 via a communication line. After that, the process proceeds to S214, and the operation of the semiconductor element 100 is stopped.

In the case of FIG. 3 (e), the remaining life of the semiconductor element 100 is 0 or less at time k, and the operation of the semiconductor element 100 is not guaranteed. Therefore, the outside of the semiconductor element 100 is notified that the operation of the semiconductor element 100 is not guaranteed. Then, the operation of the semiconductor element 100 is stopped.

Therefore, even though the remaining life of the semiconductor element 100 is 0 or less, it is possible to prevent the semiconductor element 100 from continuing the operation in a state where the operation is not guaranteed.

In this way, the semiconductor device of this embodiment changes the drive state based on the remaining life information. As a result, the semiconductor element 100 can be driven according to the length of the remaining life. Further, when the remaining life of the semiconductor element 100 falls below the first life threshold value, the normal drive mode is changed to the life extension drive mode. As a result, the life of the semiconductor element 100 can be extended while continuing the operation of the semiconductor element 100. Further, when the remaining life of the semiconductor element 100 is less than the second life threshold, the interval for acquiring the remaining life information is shortened as compared with the case where the remaining life exceeds the first life threshold. As a result, the remaining life of the semiconductor element 100 can be obtained with high accuracy. Further, when the remaining life of the semiconductor element 100 is 0 or less, the notification is sent to the outside of the semiconductor element 100. This makes it possible to prevent the semiconductor element 100 from continuing its operation without being guaranteed.

In this embodiment, the drive mode of the semiconductor element 100 is changed depending on whether or not the remaining life exceeds the first life threshold. The present invention is not limited to this example, and when the remaining life indicates the first length, the normal drive mode is set, and when the remaining life indicates a second length shorter than the first length, the life is extended. The drive mode may be set.

(Example 2)
In this embodiment, a case where the image pickup device is used as the semiconductor device 100 described in the first embodiment will be described.

The image pickup device 450 shown in FIG. 4 has an image pickup device and a life prediction unit 109. The image sensor 400 is a CMOS sensor. A pixel array 402 in which pixels 401 are arranged over a plurality of rows and columns, a vertical scanning circuit 403, a vertical output line 404, and a column circuit 405 are provided. Further, the image sensor 400 includes a reference signal generation circuit 406, a storage unit 407, a counter circuit 408, a horizontal scanning circuit 409, a signal processing circuit 410, an image pickup mode control unit 411, and a life prediction unit 109.

Here, the configuration of the pixel 401 will be described with reference to FIG.

Each pixel 401 has photodiodes 501a and 501b. Light transmitted through one microlens (not shown) is incident on the photodiodes 501a and 501b of one pixel 401. In other words, the photodiodes 501a and 501b receive light in different regions of the exit pupil of the optical system that guides the light to the CMOS sensor. Therefore, by using a signal based on the charge charged by each of the photodiodes 501a and 501b, the focus detection of the phase difference method can be performed.

Further, an image can be generated by using a signal corresponding to the charge obtained by adding the charge photoelectrically converted by the photodiode 501a and the charge photoelectrically converted by the photodiode 501b.

The photodiodes 501a and 501b accumulate electric charges according to the incident light. The vertical scanning circuit 403 shown in FIG. 4 performs vertical scanning for reading a signal in units of pixels 401 arranged over a line. By setting the signal txa output to the transfer gate 502a to the High level, the charge accumulated by the photodiode 501a is transferred to the floating diffusion unit (FD unit) 503.

Further, the vertical scanning circuit 403 sets the signal txb output to the transfer gate 502b to a high level, so that the charge accumulated by the photodiode 501b is transferred to the FD unit 503.

In the FD unit 503, the electric charge transferred from the photodiodes 501a and 501b is converted into a voltage based on the capacity of the FD unit 503. The FD unit 503 is connected to the gate of the amplification transistor 504. The amplification transistor 504 is connected to the vertical output line 404 via the selection transistor 506.

The pixel 401 has a reset transistor 505 connected to the FD unit 503. The reset transistor 505 resets the FD unit 503 when the vertical scanning circuit 403 sets the signal res to the High level. When performing the photodiode reset to reset the charges of the photodiodes 501a and 501b, the vertical scanning circuit sets the signals txa and txb to the high level during the period when the signal res is set to the high level.

Each of the signals res, txa, txb, and sel supplied from the vertical scanning circuit 403 is input in common by the pixel 401 in one row. The output vout of each pixel 401 is input to the corresponding column circuit 405 via the vertical output line 404 arranged corresponding to the column in which the pixel 401 is arranged.

Here, the configuration of the column circuit 405 arranged for each column will be described with reference to FIG.

FIG. 6 is a diagram showing the configuration of the column circuit 405. The column circuit 405 includes a current source 601, a switch 602, a differential amplifier 603, a switch 604, a comparator 605, and a switch 606.

The switch 602 is a switch for switching the current source 601 on and off. The switch 604 is a switch for switching on and off of the differential amplifier 603. The switch 606 is a switch for switching on and off of the comparator 605.

When the switch 602 is on, the vertical scanning circuit 403 sets the signal sel supplied to the pixel 401 in the row to be read to the High level. As a result, a current flows between the amplification transistor 504 and the current source 601 via the selection transistor 506 of the pixel 401 in the row to be read. As a result, the amplification transistor 504 performs a source follower operation. As a result, a signal (pixel signal) based on the voltage of the FD unit 503 is output to the vertical output line 404.

When the switch 604 is on, the differential amplifier 603 is in operation. In this case, the differential amplifier circuit 603 amplifies the pixel signal output from the pixel 401 to the vertical output line 404, and outputs the amplified signal to the comparator 605.

When the switch 605 is on, the comparator 605 is in operation. In this case, the comparator 605 compares the voltage of the amplified signal output from the differential amplifier 603 with the voltage of the lamp signal supplied from the reference signal generation circuit 406. Then, the comparator 605 outputs the result of this comparison to the storage unit 407 shown in FIG. 4 as a signal Cout. The signal level of the signal Cout changes when the magnitude relationship of the signal voltage is inverted. A count signal is input to the storage unit 407 from the counter circuit 408. This count signal changes the count value according to the start timing of the change in the voltage of the lamp signal supplied by the reference signal generation circuit 406. Then, the storage unit 407 latches the count value based on the timing when the signal level of the signal Cout changes. As a result, the storage unit 407 can obtain digital data corresponding to the signal level of the amplified signal. In this way, the row circuit 405 and the storage unit 407 perform AD conversion of the pixel signal.

The digital data stored in the storage unit 407 is sequentially transferred to the signal processing circuit 410 by the horizontal scanning circuit 409 for each column. A series of operations related to reading a pixel signal from these pixels is performed while selecting a pixel row of the pixel array 402 by the vertical scanning circuit 403.

FIG. 7 is a timing diagram showing the operation of the image pickup apparatus shown in FIG. FIG. 7A is a timing diagram when performing an imaging operation. FIG. 7B is a timing diagram when the focus detection operation and the imaging operation are performed. A pixel signal is read out from the pixels 401 included in some of the rows of pixels 401 arranged in the pixel array 402 by the focus detection operation and the imaging operation shown in FIG. 7B. A pixel signal is read out from the pixel 401 included in some of the other rows by the imaging operation shown in FIG. 7 (a). Some of these lines and some of the other lines are allocated according to the area where focus detection is performed. As another example, the pixels 401 in all rows perform the operation shown in FIG. 7A in a certain frame. Then, in another frame, the operation shown in FIG. 7B may be performed. The frame referred to in this case can be, for example, a period from the start of the vertical scan circuit 403 to the start of the next vertical scan. Typically, a control circuit (not shown) sets the signal level of the vertical sync signal indicating the start of vertical scan of the vertical scan circuit 403 to the High level, and then sets the signal level of the vertical sync signal to the High level. Can be the period of. As another example, the signal corresponding to one image generated by the signal processing unit is read from the pixel array 402, and then the signal corresponding to the next image is read from the pixel array 402. It can also be the period until the start of reading. For example, it is assumed that the moving image is shot at 60 fps (fps is an abbreviation for frame per second). In this example, the signal corresponding to the image of one frame out of 60 frames is read from the pixel array 402, and then the signal corresponding to the image of the next one frame is read from the pixel array 402. It can be the period until the start.

First, the operation of FIG. 7A will be described.

At time ta1, the vertical scanning circuit 403 sets the signal sel supplied to the row of the pixel 401 performing the imaging operation to the High level. As a result, the selection transistor 506 of the pixel 401 is turned on.

Then, at time ta2, the vertical scanning circuit 403 sets the signal res to the Low level and turns off the reset transistor 505. As a result, the reset of the FD unit 503 is released.

The amplification transistor 504 outputs a noise signal (N signal) based on the voltage of the FD unit 503 whose reset has been released to the vertical output line 404 via the selection transistor 506. The N signal output to the vertical output line 404 is converted into digital data (N data) by AD conversion by the column circuit 405 and the storage unit 407.

At time ta3, the vertical scanning circuit 403 sets the signals txa and txb to the High level and then to the Low level. By this operation, the electric charge accumulated by the photodiode 501a based on the incident light and the electric charge accumulated by the photodiode 501b based on the incident light are transferred to the FD unit 503. Therefore, the voltage of the FD unit 503 is a voltage corresponding to the charge obtained by adding the charge of the photodiode 501a and the charge of the photodiode 501.

The amplification transistor 504 outputs a signal based on the voltage of the FD unit 503 corresponding to the charge obtained by adding the charge of the photodiode 501a and the charge of the photodiode 501 to the vertical output line 404 via the selection transistor 506. The signal output by the amplification transistor 504 will be described. If the voltage of the FD unit 503 is a voltage based on the electric charge accumulated by the photodiode 501a based on the incident light, the signal output by the amplification transistor 504 is referred to as an A + N signal (sum of the A signal and the N signal). do. When the voltage of the FD unit 503 is a voltage based on the electric charge accumulated by the photodiode 501b based on the incident light, the signal output by the amplification transistor 504 is defined as a B + N signal (sum of the B signal and the N signal). In such a case, the voltage-based signal of the FD unit 503 corresponding to the charge obtained by adding the charge of the photodiode 501a and the charge of the photodiode 501b output by the amplification transistor 504 is the sum of the A + N signal and the B + N signal. Corresponds to a signal. Therefore, the signal output by the amplification transistor 504 is expressed as an A + B + N signal. The A + B + N signal output to the vertical output line 404 is converted into digital data (A + B + N data) by AD conversion by the column circuit 405 and the storage unit 407.

At time ta5, the vertical scan circuit 403 sets the signal res to the High level. As a result, the voltage of the FD unit 503 is reset.

Then, at time ta6, the vertical scanning circuit 403 sets the signal sel to the Low level. As a result, the selection of the pixel 401 in one line is completed. After that, the vertical scanning circuit 403 sets the signal sel of the pixel 401 in the next row to be read to the High level.

The N data and A + B + N data held by the storage unit 407 are sequentially read out from the storage unit 407 to the signal processing circuit 410 by the horizontal scanning circuit 409 for each column. The signal processing circuit 410 outputs A + B data, which is a signal of the difference between the A + B + N data and the N data.

Next, the operation of FIG. 7B will be described.

The operation at time tb1 is the same as the operation at time ta1 in FIG. 7 (a).

The operation at time tb2 is the same as the operation at time ta2 in FIG. 7 (a).

At time tb3, the vertical scanning circuit 403 sets the signal txa to the High level and then to the Low level. As a result, the electric charge accumulated by the photodiode 501a based on the incident light is transferred to the FD unit 503. The amplification transistor 504 outputs an A + N signal to the vertical output line 404 via the selection transistor 506. The A + N signal output to the vertical output line 404 is converted into digital data (A + N data) by AD conversion by the column circuit 405 and the storage unit 407.

Until before time tb6, the FD unit 503 holds the electric charge transferred from the photodiode 501a.

At time tb6, the vertical scanning circuit 403 sets the signal txa and the signal txb to the High level and then to the Low level. As a result, in the FD unit 503, in addition to the electric charge of the photodiode 501a held by the time tb6, the electric charge accumulated by the photodiode 501b based on the incident light and the photodiode 501a during the period from the time tb4 to the time tb7 Retains the accumulated charge. Therefore, the voltage of the FD unit 503 is a voltage corresponding to the charge obtained by adding the charge of the photodiode 501a and the charge of the photodiode 501.

The amplification transistor 504 outputs an A + B + N signal to the vertical output line 404. The A + B + N signal output to the vertical output line 404 is converted into digital data (A + B + N data) by AD conversion by the column circuit 405 and the storage unit 407.

At time tb8, the vertical scan circuit 403 sets the signal res to the High level. As a result, the voltage of the FD unit 503 is reset.

Then, at time tb9, the vertical scanning circuit 403 sets the signal sel to the Low level. As a result, the selection of the pixel 401 in one line is completed. After that, the vertical scanning circuit 403 sets the signal sel of the pixel 401 in the next row to be read to the High level.

The N data, A + N data, and A + B + N data held by the storage unit 407 are sequentially read out from the storage unit 407 to the signal processing circuit 410 by the horizontal scanning circuit 409 for each column. The signal processing circuit 410 outputs a signal of the difference between A + N data and N data (A data) and a signal of the difference between A + B + n data and N data (A + B data).

The signal processing unit (not shown) to which the data output by the image pickup apparatus is input obtains the B data, which is the difference between the A data and the A + B data, and performs focus detection on the A data and the B data. Further, the signal processing unit generates an image using the A + B data.

See FIG. The image pickup mode control unit 411 is a circuit block including the voltage control unit 104, the drive mode selection unit 105, the frequency control unit 106, and the signal generation unit 107 according to the first embodiment. The drive mode control unit 411 receives the remaining life information from the life information receiving unit 108. Then, at least one of changing the voltage value supplied to all or part of the circuit in the image sensor 400 and changing the cycle of the control signal is performed according to the length of the remaining life indicated by the remaining life information. The circuit in the image pickup element 400 is a pixel array 402, a vertical scanning circuit 403, a column circuit 405, a reference signal generation circuit 406, a storage unit 407, a counter circuit 408, a horizontal scanning circuit 409, and a signal processing circuit 410. Alternatively, the operation of a part of the pixel array 402, the vertical scanning circuit 403, the column circuit 405, the reference signal generation circuit 406, the storage unit 407, the counter circuit 408, the horizontal scanning circuit 409, and the signal processing circuit 410 in the image pickup element 400 is stopped. conduct.

As described in the equations (1) to (4) of the first embodiment, the life of the semiconductor device is the current density J of the wiring, the wiring temperature T, the voltage Vg applied to the gate of the transistor, and the maximum substrate current Isub flowing through the semiconductor substrate. Limited by. The image pickup mode control unit 411 has a current density J, a wiring temperature T, a voltage Vg applied to the gate of the transistor, all or one of the maximum substrate current Isub flowing through the semiconductor substrate so that the remaining life of the image pickup element 400 is extended. Change the operating state so that the value of the part becomes smaller.

This causes a decrease in the operating speed of the image pickup device or a partial stop of the function, but the image pickup operation can be continued and the life of the image pickup device can be extended.

In the life extension drive mode by the image pickup mode control unit 411, for example, there is a decrease in the frame rate. Specifically, the period of vertical scanning of the vertical scanning circuit 403 is reduced with respect to the normal drive mode. As the vertical scanning cycle decreases, the operating cycles of the reference signal generation circuit 406, the counter circuit 408, the horizontal scanning circuit 409, and the signal processing circuit 410 also decrease. In other words, the decrease in the frame rate can be said to reduce the number of times the vertical scanning circuit 403 performs vertical scanning per unit time.

Further, as one of the life extension drive modes, a signal may be read from the pixel 401 in a region of only a part of the pixel array 402. That is, the number of pixels to be read is reduced in the life extension drive mode as compared with the normal drive mode. For example, in the normal drive mode, the pixel 401 corresponding to the resolution of 4K2K is read out, and in the life extension drive mode, the pixel 401 corresponding to the VGA resolution is read out. Also in the column circuit 405 and the storage unit 407, the horizontal scanning circuit 409 scans the storage unit 407 of only a part of the columns in response to the decrease in the number of pixels 401 to be read out. This makes it possible to extend the remaining life of the image pickup device while continuing the operation of the image pickup device.

Further, as another example, the number of pixels for performing the focus detection operation in FIG. 7B may be reduced in the life extension drive mode as compared with the normal drive mode. As a result, the number of AD conversions of the A + N signals of the column circuit 405 and the storage unit 407 can be reduced. As a result, the remaining life of the image pickup apparatus can be extended while continuing the image pickup operation. In particular, when the extension of the remaining life is emphasized, the number of pixels for performing the focus detection operation in FIG. 7 (b) is set to zero, and all the pixels 401 are driven for the image pickup operation in FIG. 7 (a). good.

As described in the first embodiment, the life prediction unit 109 predicts the remaining life of the image sensor 400 based on the latest operating state of the image sensor 400 or historical information related to the operating state, thereby predicting the remaining life of the image sensor 400. Obtain the remaining life information.

Here, the operating state represents a state when the image sensor 400 is operated, such as an operating voltage and an operating frequency when the image sensor 400 is operated, or a state generated as a result of an operation such as temperature.

The life prediction unit 109 acquires the remaining life information after performing the imaging operation a plurality of times. The image pickup operation referred to here refers to an operation of photographing a subject under various environments. That is, instead of the image pickup operation of the image pickup element 400 in a known environment such as a test in a manufacturing factory of an image pickup device, the image pickup in an environment in which the remaining life of the image pickup element 400 changes depending on the environment in which the image pickup element 400 is placed. Refers to the operation. Therefore, the image pickup device of this embodiment can predict the remaining life according to the environment and operation in which the image pickup device 400 is actually placed or placed. Since this remaining life prediction is based on the environment and operation in which the image sensor 400 is actually placed or placed, it is possible to predict the remaining life more accurately than the life prediction by simulation. ..

According to the embodiment of the present invention described above, the image sensor 400 is selected to be driven to extend the life of the image sensor 400 based on the remaining life information, so that the image sensor 400 has a faster useful life in simulation. The problem of reaching the end of life is solved.

The image pickup device of this embodiment can be used, for example, for a camera installed in an area where it is difficult to replace the image pickup device. Such cameras include, for example, cameras installed in remote areas (disaster monitoring (volcanic disasters, meteorological disasters), spacecraft, artificial satellites (meteorological observation satellites, exploration satellites, space stations, etc.), space exploration robots, etc.). There are surveillance cameras installed in conflict areas. While it is not easy to replace the image pickup device at the end of its life, it is desired that these cameras continue to operate as the image pickup device. Therefore, when the predicted remaining life falls below the first life threshold, the operation of the image pickup device 400 is changed from the normal drive mode to the life extension drive mode. As a result, the remaining life of the operation of the image pickup apparatus can be extended while continuing the image pickup operation.

Further, the image pickup device of this embodiment can be used for an in-vehicle camera as an example of an image pickup device mounted on a moving body. In an in-vehicle camera, even if the remaining life is less than the first life threshold, it is not easy to replace the image sensor 400 while driving. In such a case, it is useful to change the image pickup device 400 from the normal drive mode to the life extension drive mode to continue the image pickup operation and extend the life of the image pickup device 400.

Further, the life prediction unit 109 may acquire the remaining life information in parallel with the imaging operation. For example, in the case of a camera such as a camera installed in a remote place or a surveillance camera that continuously shoots a moving image, the remaining life information is acquired by the life prediction unit 109 during the period during which the imaging operation is performed. To. This makes it possible to predict the remaining life of the image sensor 400 while continuing to shoot moving images. In other words, the life prediction unit 109 may acquire the remaining life information during the period during which the vertical scanning is performed. Further, the life prediction unit 109 may acquire the remaining life information during the period in which the photoelectric conversion unit accumulates the electric charge based on the incident light. Further, the life prediction unit 109 may acquire the remaining life information during the period when the column circuit 405 and the storage unit 407 are processing the pixel signal (for example, the period during which the AD conversion is performed). That is, after performing the imaging operation a plurality of times, the life prediction unit 109 may acquire the remaining life information in parallel with the period during which the imaging operation is performed.

As described above, the image pickup device of the present embodiment has an effect that the life of the image pickup device can be extended while continuing the operation of the image pickup device even when the replacement of the image pickup element is not easy.

(Example 3)
The image pickup apparatus of this embodiment will be described focusing on the differences from the second embodiment. The image pickup apparatus of this embodiment can accurately obtain information on the remaining life of the image pickup device.

FIG. 8 is a diagram showing the image pickup apparatus 800 of this embodiment.

The image pickup device 800 of the present embodiment is different from the image pickup device 450 shown in FIG. 5 in that it includes a state acquisition unit 812 and a life prediction unit 813 connected to the state acquisition unit 812.

A case where hot carrier injection (HC) is dominant as a factor of performance deterioration of the image pickup device 400 will be described. In this case, the remaining life of the image pickup device 400 is evaluated by the following equation (5), which is the ratio of the time change amount ΔId of the drain current to the drain current Id of the transistor when the image pickup device 400 starts operation as a new product. To.
Id _RATIO = ΔId / Id ... (5)

The state acquisition unit 812 measures the drain current Id of the transistor of the image pickup device 400. The measurement result is output to the life prediction unit 813. The life prediction unit 813 calculates the ratio of the change amount of the drain current Id of the known initial MOS transistor and the drain current Id of the latest MOS transistor.

Here, it is assumed that the time when the Id _RATIO becomes 0.1, that is, 10% in the equation (5) is set as the life time of the image sensor 400. In this case, 10% is set as the threshold value of Id _RATIO in the life prediction unit 813. The life prediction unit 813 predicts the remaining life of the image sensor 400 by comparing the threshold value of 10% with the actual Id _RATIO .

As an example of the method of measuring the drain current Id of the transistor, a resistance element is provided in the state acquisition unit 812, one end of the resistance element is connected to the power supply voltage, and the other end of the resistance element is connected to the drain of the transistor of the image pickup element 400. do. Then, the voltage values of both terminals of this resistance element are measured. Thereby, the drain current can be measured by using the equation of V = IR (V is the voltage at both terminals of the resistance element, R is the resistance value of the resistance element, and I is the current value flowing through the resistance element). Then, by obtaining the time change rate of Id _RATIO , it is possible to predict the period until Id _RATIO reaches 10%, that is, the remaining life.

As another method for acquiring the remaining life information, the change in the current consumption value consumed by the image sensor 400 may be measured. By measuring the change between the current consumption value consumed by the image pickup element 400 when it is new and the current consumption value at the time of measurement, the drain current Id of the transistor can be indirectly measured. Then, the remaining life can be predicted by obtaining the amount of time change of the current consumption value.

(Example 4)
The image pickup apparatus of this embodiment will be described focusing on the differences from the third embodiment. The configuration of the image pickup apparatus can be the same as that shown in FIG.

In the image pickup apparatus of Example 3, the remaining life prediction was obtained by the change of the drain current Id. In this embodiment, other prediction methods will be described.

In the semiconductor element and the image pickup element, as described in the first embodiment, the performance is deteriorated by HC, EM, TDDB, and negative NBTI. This deterioration in performance determines the life of the semiconductor element and the image pickup element.

The life prediction regarding the performance deterioration due to HC is as described in Example 3.

The life time LT (EM) due to EM as a factor of performance deterioration is obtained by the formula (1) described in Example 1.

The life time LT (TDDB) due to TDDB as a factor of performance deterioration is obtained by the formula (2) described in Example 1.

The life time LT (NBTI) due to NBTI as a factor of performance deterioration can be calculated by the formula (4) described in Example 1.

Further, the life time LT (HC) due to HC as a factor of performance deterioration is obtained by the formula (3) described in Example 1. It can also be obtained by the formula (5) described in Example 3.

The state acquisition unit 812 measures all or part of the equations (1) to (5) and calculates the life of the image sensor 400.

In order to calculate the life time of the image pickup element 400 using the equations (1) to (5), the state acquisition unit 812 has a current density J of a current flowing through a transistor or a resistance element provided in the image pickup element 400, and a wiring temperature T. , The voltage value Vg applied to the gate of the transistor is measured.

The method for obtaining the current density J can be the same as that in the third embodiment.

As an example, the method for acquiring the wiring temperature T measures the current value flowing through a diode element (not shown) of the state acquisition unit 812. Since the current value flowing through the diode element generally has a temperature dependence, the temperature T can be obtained by measuring the current value.

The voltage value Vg can be a gate voltage applied to the transistor whose life is predicted. As for the voltage value Vg, the design value may be used when the difference between the measured value and the design value is expected to be substantially negligible.

In this way, the state acquisition unit 812 acquires each value used in the equations (1) to (5). The state acquisition unit 812 outputs each acquired value to the life prediction unit 813. That is, the state acquisition unit 812 acquires the temperature and the amount of current consumption of the image pickup device 400. The state acquisition unit 812 may further obtain information on the integrated operation time and the frame rate of the image pickup device 400.

The life prediction unit 813 obtains the life of the image pickup device 400 by using all or a part of the equations (1) to (5). When the life prediction unit 813 predicts the life using a plurality of the formulas (1) to (5), the shortest life among the formulas used is set as the life of the image sensor 400.

As described above, the life prediction unit 813 of this embodiment predicts the life based on the actual operating state of the image sensor 400. This makes it possible to predict the life with high accuracy.

The life of the image pickup device 400 varies greatly depending on the usage environment. It varies depending on the temperature at which the image sensor 400 is placed, the number of images taken, and the applied voltage. For example, in the case of an in-vehicle camera, a low latitude area is placed at a high temperature of 80 ° C. or higher for a longer time than a high latitude area. In addition, the high latitude area takes longer to be placed in a low temperature of 0 ° C. or lower than the low latitude area. Therefore, the actual life of the image sensor 400 may be shorter than the design life. Therefore, by predicting the life based on the actual operation of the image sensor 400, it is possible to prevent the sudden stop of the operation of the image sensor 400 from occurring.

In this embodiment, the state acquisition unit 812 and the life prediction unit 813 are provided outside the image sensor 400, but the state acquisition unit 812 and the life prediction unit 813 may be inside the image sensor 400. In particular, when the state acquisition unit 812 is provided on the same semiconductor substrate as the image pickup device 400, it is placed in the same operating environment as the image pickup device 400, so that more accurate life prediction can be performed.

(Example 5)
The image pickup apparatus of this embodiment will be described focusing on the differences from the fourth embodiment.

FIG. 9 is a diagram showing the configuration of the image pickup apparatus of this embodiment. The image pickup apparatus of this embodiment further has a memory unit 920 with respect to the image pickup apparatus of Example 4.

The memory unit 920 holds history information related to the operating state.

The life prediction unit 913 of this embodiment predicts the remaining life using the history information held by the memory unit 920 and the information output from the state acquisition unit 812, and generates the remaining life information. The history information is, for example, the integrated operation time of the image pickup device and the history of the image pickup mode of the image pickup device. The image pickup mode is a cycle (frame rate) at which the image pickup element 400 outputs an image signal, an image size, whether or not the focus detection function is executed, and the like.

As an example of the operation, the history information that the memory unit 920 holds the remaining life predicted by the life prediction unit 913 using all or a part of the equations (1) to (5) as described in the fourth embodiment is used. Use to correct.

Further, as an example of another operation, the memory unit 920 holds the Id _RATIO obtained by using the equation (5) each time the remaining life is predicted. Then, the life prediction unit 913 obtains the change over time of the Id _RATIO from the Id _RATIO at each time held in the memory unit 920. From this change over time of Id _RATIO , the time when Id _RATIO becomes 10% or more is predicted (that is, the remaining life is predicted). This makes it possible to predict the remaining life with higher accuracy.

Further, the value stored in the memory unit 920 is not limited to the drain current Id of the transistor, and may be the value of all or a part of the equations (1) to (5).

The memory unit 920 may be outside the image pickup apparatus 800. Further, it is desirable that the memory unit 920 is a non-volatile memory. When the memory unit 920 is provided outside the image pickup device 800, the manufacturing process of the image pickup device 800 and the memory section 920 can be separated. As a result, the restrictions on the manufacturing process of the image pickup apparatus 800 and the memory unit 920 can be reduced.

As described above, the image pickup apparatus of this embodiment includes a memory unit 920 that holds history information related to the operating state. By predicting the remaining life using the history information held by the memory unit 920, the remaining life can be predicted with high accuracy.

The image pickup apparatus described in the examples so far can be a laminated sensor in which a plurality of chips are laminated. For example, the pixel array 402 is arranged on the first chip, and the column circuit 405, the reference signal generation circuit 406, the storage unit 407, the counter circuit 408, the horizontal scanning circuit 409, the signal processing circuit 410, and the image pickup mode control unit 411 are arranged on the second chip. May be provided. In this case, the second chip may further include a life prediction unit 109. Further, the third chip provided with the signal processing unit for generating an image may be further laminated. In this case, the life prediction unit 109 may be provided on the third chip.

(Example 6)
FIG. 10 is a block diagram showing the configuration of the imaging system 500 according to the present embodiment. The image pickup system 500 of this embodiment includes an image pickup device 200 to which any of the configurations of the image pickup devices described in each of the above-described embodiments is applied. Specific examples of the image pickup system 500 include a digital still camera, a digital camcorder, a surveillance camera, and the like. FIG. 10 shows a configuration example of a digital still camera to which the image pickup device of any of the above-described embodiments is applied as the image pickup device 200.

The image pickup system 500 illustrated in FIG. 10 is for protecting the image pickup device 200, the lens 5020 for forming an optical image of a subject on the image pickup device 200, the aperture 504 for varying the amount of light passing through the lens 5020, and the lens 5020. Has a barrier 506. The lens 5020 and the aperture 504 are optical systems that collect light on the image pickup apparatus 200.

The image pickup system 500 also has a signal processing unit 5080 that processes an output signal output from the image pickup apparatus 200. The signal processing unit 5080 performs a signal processing operation of performing various corrections and compressions on the input signal as necessary and outputting the signal. The signal processing unit 5080 may have a function of performing AD conversion processing on the output signal output from the image pickup apparatus 200. In this case, it is not always necessary to have an AD conversion circuit inside the image pickup apparatus 200.

The image pickup system 500 further includes a buffer memory unit 510 for temporarily storing image data, and an external interface unit (external I / F unit) 512 for communicating with an external computer or the like. Further, the imaging system 500 includes a recording medium 514 such as a semiconductor memory for recording or reading imaging data, and a recording medium control interface unit (recording medium control I / F unit) 516 for recording or reading on the recording medium 514. Has. The recording medium 514 may be built in the image pickup system 500 or may be detachable.

Further, the image pickup system 500 has an overall control / calculation unit 518 that performs various calculations and controls the entire digital still camera, and a timing generation unit 520 that outputs various timing signals to the image pickup device 200 and the signal processing unit 5080. Here, a timing signal or the like may be input from the outside, and the image pickup system 500 may have at least an image pickup device 200 and a signal processing unit 5080 that processes an output signal output from the image pickup device 200. The overall control / calculation unit 518 and the timing generation unit 520 may be configured to perform a part or all of the control functions of the image pickup apparatus 200.

The image pickup apparatus 200 outputs an image signal to the signal processing unit 5080. The signal processing unit 5080 performs predetermined signal processing on the image signal output from the image pickup apparatus 200, and outputs the image data. Further, the signal processing unit 5080 uses the image signal to generate an image.

By configuring the image pickup system by using the image pickup device by the image pickup device of each of the above-described embodiments, it is possible to realize an image pickup system capable of acquiring a higher quality image.

(Example 7)
The imaging system and the moving body of this embodiment will be described with reference to FIGS. 11 and 12.

FIG. 11 is a schematic view showing a configuration example of an imaging system and a moving body according to the present embodiment. FIG. 12 is a flow chart showing the operation of the imaging system according to the present embodiment.

In this embodiment, an example of an imaging system related to an in-vehicle camera is shown. FIG. 11 shows an example of a vehicle system and an imaging system mounted on the vehicle system. The image pickup system 701 includes an image pickup device 702, an image preprocessing unit 715, an integrated circuit 703, and an optical system 714. The optical system 714 forms an optical image of the subject on the image pickup apparatus 702. The image pickup apparatus 702 converts the optical image of the subject imaged by the optical system 714 into an electric signal. The image pickup device 702 is an image pickup device according to any one of the above-described embodiments. The image preprocessing unit 715 performs predetermined signal processing on the signal output from the image pickup apparatus 702. The function of the image preprocessing unit 715 may be incorporated in the image pickup apparatus 702. The image pickup system 701 is provided with at least two sets of an optical system 714, an image pickup device 702, and an image preprocessing unit 715 so that the output from each set of image preprocessing units 715 is input to the integrated circuit 703. It has become.

The integrated circuit 703 is an integrated circuit for an imaging system application, and includes an image processing unit 704 including a memory 705, an optical ranging unit 706, a parallax calculation unit 707, an object recognition unit 708, and an abnormality detection unit 709. The image processing unit 704 performs image processing such as development processing and defect correction on the output signal of the image preprocessing unit 715. The memory 705 stores the primary storage of the captured image and the defect position of the captured pixel. The optical ranging unit 706 focuses the subject and measures the distance. The parallax calculation unit 707 calculates the parallax (phase difference of the parallax image) from the plurality of image data acquired by the plurality of image pickup devices 702. The object recognition unit 708 recognizes a subject such as a car, a road, a sign, or a person. When the abnormality detection unit 709 detects an abnormality in the image pickup apparatus 702, the abnormality detection unit 709 notifies the main control unit 713 of the abnormality.

The integrated circuit 703 may be realized by specially designed hardware, may be realized by a software module, or may be realized by a combination thereof. Further, it may be realized by FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), or the like, or may be realized by a combination thereof.

The main control unit 713 controls and controls the operations of the image pickup system 701, the vehicle sensor 710, the control unit 720, and the like. A method in which the image pickup system 701, the vehicle sensor 710, and the control unit 720 each have a communication interface individually without having a main control unit 713, and each of them sends and receives a control signal via a communication network (for example, CAN standard). Can also be taken.

The integrated circuit 703 has a function of receiving a control signal from the main control unit 713 or transmitting a control signal and a set value to the image pickup apparatus 702 by its own control unit. For example, the integrated circuit 703 transmits a setting for pulse-driving the voltage switch 13 in the image pickup apparatus 702, a setting for switching the voltage switch 13 for each frame, and the like.

The image pickup system 701 is connected to the vehicle sensor 710, and can detect the traveling state of the own vehicle such as the vehicle speed, the yaw rate, and the steering angle, the environment outside the own vehicle, and the state of other vehicles / obstacles. The vehicle sensor 710 is also a distance information acquisition means for acquiring distance information from the parallax image to the object. Further, the image pickup system 701 is connected to a driving support control unit 711 that provides various driving support such as automatic steering, automatic cruising, and collision prevention function. In particular, regarding the collision determination function, collision estimation / collision presence / absence with other vehicles / obstacles is determined based on the detection results of the image pickup system 701 and the vehicle sensor 710. As a result, avoidance control when a collision is estimated and safety device activation in the event of a collision are performed.

The image pickup system 701 is also connected to an alarm device 712 that issues an alarm to the driver based on the determination result of the collision determination unit. For example, when the possibility of a collision is high as the judgment result of the collision detection unit, the main control unit 713 controls the vehicle to avoid the collision and reduce the damage by applying the brake, releasing the accelerator, suppressing the engine output, and the like. conduct. The alarm device 712 warns the user by sounding an alarm such as a sound, displaying alarm information on a display unit screen of a car navigation system, a meter panel, or the like, or giving vibration to a seat belt or steering.

In this embodiment, the surroundings of the vehicle, for example, the front or the rear, are photographed by the image pickup system 701. FIG. 11B shows an arrangement example of the image pickup system 701 when the image pickup system 701 images the front of the vehicle.

The two image pickup devices 702 are arranged in front of the vehicle 700. Specifically, when the center line with respect to the advancing / retreating direction or the outer shape (for example, the vehicle width) of the vehicle 700 is regarded as an axis of symmetry, and the two image pickup devices 702 are arranged line-symmetrically with respect to the axis of symmetry, the vehicle 700 and the cover are covered. It is preferable for acquiring distance information with the object to be photographed and determining the possibility of collision. Further, it is preferable that the image pickup apparatus 702 is arranged so as not to obstruct the driver's field of view when the driver visually recognizes the situation outside the vehicle 700 from the driver's seat. The alarm device 712 is preferably arranged so that it can be easily seen by the driver.

Next, the failure detection operation of the image pickup apparatus 702 in the image pickup system 701 will be described with reference to FIG. The failure detection operation of the image pickup apparatus 702 is performed according to steps S810 to S880 shown in FIG.

Step S810 is a step for setting the start-up of the image pickup apparatus 702. That is, the setting for the operation of the image pickup device 702 is transmitted from the outside of the image pickup system 701 (for example, the main control unit 713) or the inside of the image pickup system 701, and the image pickup operation and the failure detection operation of the image pickup device 702 are started.

Next, in step S820, a pixel signal is acquired from the effective pixel. Further, in step S830, the output value from the failure detection pixel provided for failure detection is acquired. The failure detection pixel includes a photoelectric conversion unit like the effective pixel. A predetermined voltage is written in this photoelectric conversion unit. The failure detection pixel outputs a signal corresponding to the voltage written in the photoelectric conversion unit. The steps S820 and S830 may be reversed.

Next, in step S840, the non-determination of the expected output value of the failure detection pixel and the output value from the actual failure detection pixel is performed.

If the expected output value and the actual output value match as a result of the non-determination in step S840, the process proceeds to step S850, it is determined that the imaging operation is normally performed, and the processing step proceeds to step S860. And migrate. In step S860, the pixel signal of the scanning line is transmitted to the memory 705 and primarily stored. After that, the process returns to step S820 and the failure detection operation is continued.

On the other hand, if the expected output value and the actual output value do not match as a result of the non-determination in step S840, the processing step proceeds to step S870. In step S870, it is determined that there is an abnormality in the imaging operation, and an alarm is issued to the main control unit 713 or the alarm device 712. The alarm device 712 displays on the display unit that an abnormality has been detected. After that, in step S880, the image pickup apparatus 702 is stopped, and the operation of the image pickup system 701 is terminated.

In this embodiment, an example in which the flowchart is looped for each line is illustrated, but the flowchart may be looped for each of a plurality of lines, or the failure detection operation may be performed for each frame.

The alarm in step S870 may be notified to the outside of the vehicle via the wireless network.

Further, in this embodiment, the control that does not collide with other vehicles has been described, but it can also be applied to the control that automatically drives following other vehicles and the control that automatically drives so as not to go out of the lane. .. Further, the image pickup system 701 can be applied not only to a vehicle such as a own vehicle but also to a moving body (moving device) such as a ship, an aircraft, or an industrial robot. In addition, it can be applied not only to moving objects but also to devices that widely use object recognition, such as intelligent transportation systems (ITS).

[Modification Example]
The present invention is not limited to the above embodiment, and various modifications are possible.

For example, an example in which a partial configuration of any of the examples is added to another embodiment or an example in which a partial configuration of another embodiment is replaced is also an example of the present invention.

In addition, the above-mentioned examples are merely examples of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these examples. That is, the present invention can be carried out in various embodiments without departing from the technical idea or its main features.

100 Semiconductor element 101-103 Circuit block 105 Drive mode selection unit 109, 813, 913 Life prediction unit 150 Semiconductor device 400 Image sensor 402 Pixel array 411 Image sensor mode control unit 450 Image sensor 812 State acquisition unit 920 Memory unit

Claims (24)

An image pickup device arranged over a plurality of rows and a plurality of columns, each having a plurality of pixels each having a photoelectric conversion unit, and performing an image pickup operation for reading a signal of each of the plurality of pixels, and information on the remaining life of the image pickup device. It has a life prediction unit to acquire,
The image sensor performs the image pickup operation a plurality of times, and the image sensor performs the image pickup operation a plurality of times.
The life prediction unit is an imaging device characterized in that the remaining life information is acquired after the plurality of imaging operations. The life prediction unit acquires the first information, which is the remaining life information, at the first time included in the period during which the plurality of imaging operations are performed.
The life prediction unit acquires the second information, which is the remaining life information, at the second time after the first time in the period.
The image pickup apparatus according to claim 1, wherein the life prediction unit predicts the remaining life of the image pickup device from the first information and the second information. The life prediction unit acquires operation information indicating the operation state of the image sensor, and obtains operation information.
The imaging device according to claim 1 or 2, wherein the life prediction unit acquires the remaining life information using the operation information. The image pickup apparatus according to claim 3, wherein the operation information is the temperature of the image pickup element. The image pickup apparatus according to claim 3, wherein the operation information is the integrated operation time of the image pickup element. The image pickup apparatus according to claim 3, wherein the operation information is a current consumption amount of the image pickup element. The image pickup apparatus according to claim 3, wherein the operation information is the frame rate of the image pickup element. The image pickup device has a memory unit that holds history information indicating the history of operation of the image pickup element.
The image pickup apparatus according to any one of claims 1 to 7, wherein the life prediction unit corrects the remaining life information by using the history information. The image pickup apparatus according to claim 8, wherein the history information is information indicating an integrated operation time of the image pickup apparatus. The imaging apparatus according to any one of claims 1 to 9, wherein the remaining life information is acquired by using the following equation (1).
LT = A × J ⁻ⁿ × exp (Ea / kT) ・ ・ ・ (1)
LT: Remaining life information, A: Coefficient, Ea: Activation energy, J: Current density of wiring, n: Current acceleration coefficient, k: Boltzmann constant, T: Temperature The imaging apparatus according to any one of claims 1 to 9, wherein the remaining life information is acquired by using the following equation (2).
LT = A x exp (-γVg x Vg) x exp (Ea / kT) ... (2)
LT: Remaining life information, Vg: Voltage applied to the gate of the transistor, γVg: Voltage acceleration coefficient The imaging apparatus according to any one of claims 1 to 9, wherein the remaining life information is acquired by using the following equation (3).
LT = A × Isub ^−m × exp (Ea / kT) ・ ・ ・ (3)
LT: Remaining life information, Isub: Current flowing through the semiconductor substrate, m: Substrate current-dependent coefficient, Ea: Activation energy, k: Boltzmann constant, T: Temperature The imaging apparatus according to any one of claims 1 to 9, wherein the remaining life information is acquired by using the following equation (4).
LT = A × Vg ⁿ × exp (Ea / kT) ・ ・ ・ (4)
LT: Remaining life information, A: Coefficient, Ea: Activation energy, k: Boltzmann constant, T: Temperature, n: Current acceleration coefficient The image pickup device includes a transistor and has a transistor.
The image pickup apparatus according to any one of claims 1 to 9, wherein the life prediction unit acquires the remaining life information based on a time change amount of a drain current flowing through the transistor. The image pickup apparatus according to any one of claims 1 to 14, wherein the life prediction unit acquires the remaining life information during a period in which the photoelectric conversion unit accumulates a charge based on incident light. .. It has a processing unit that processes signals from the pixels, and has a processing unit.
The imaging according to any one of claims 1 to 15, wherein the life prediction unit acquires the remaining life information during a period in which the processing unit processes a signal from the pixel. Device. The image pickup apparatus according to claim 16, wherein the processing by the processing unit is a processing for AD-converting a signal from the pixel. Further, a drive control unit for controlling the drive of the image pickup device is provided.
When the remaining life information indicates the first length, the drive control unit drives the image pickup device under the first condition.
When the remaining life information indicates a second length shorter than the first length, the drive control unit has a longer remaining life of the image pickup device than when the image pickup device is driven under the first condition. The image pickup apparatus according to any one of claims 1 to 17, wherein the image pickup device is driven under the second condition. The image pickup device includes a scanning circuit that performs vertical scanning for scanning the plurality of pixels row by row.
The first condition is a condition for performing the vertical scanning the first number of times per unit time.
The image pickup apparatus according to claim 18, wherein the second condition is a condition for performing the vertical scanning a second number of times less than the first number of times per unit time. When the drive control unit drives the image sensor under the first condition, the life prediction unit acquires the remaining life information at the first interval.
When the drive control unit drives the image pickup device under the second condition, the life prediction unit is characterized in that it acquires the remaining life information at a second interval shorter than the first interval. The image pickup apparatus according to claim 18 or 19. The first condition is a condition in which the operating frequency of the image pickup device is set to the first frequency.
The imaging according to any one of claims 18 to 20, wherein the second condition is a condition in which the operating frequency of the image pickup device is a second frequency lower than the first frequency. Device. The first condition is a condition in which the power supply voltage supplied to the image pickup device is the first voltage.
One of claims 18 to 21, wherein the second condition is a condition in which the power supply voltage supplied to the image pickup device is a second voltage having an absolute value smaller than that of the first voltage. The image pickup device according to the section. An image pickup system comprising the image pickup apparatus according to any one of claims 1 to 22 and a signal processing unit for processing a signal output by the image pickup apparatus. A moving body having the image pickup apparatus according to any one of claims 1 to 22.
A moving body having a control unit for controlling the movement of the moving body.

Images