WO2019135411A1

WO2019135411A1 - Detection device and sensor

Info

Publication number: WO2019135411A1
Application number: PCT/JP2019/000037
Authority: WO
Inventors: 友希平田
Original assignee: 株式会社ニコン
Priority date: 2018-01-05
Filing date: 2019-01-04
Publication date: 2019-07-11
Also published as: TW201937917A; JP2021044599A

Abstract

Provided is a detection device, comprising: an event detection unit which, in a pixel block composed of one or a plurality of pixels which each have at least one photoelectric conversion element, detects an event indicating that the value of photocurrent generated by the photoelectric conversion element exceeds a predetermined event detection threshold, and which outputs an event detection signal; a read control unit which identifies a region of interest on the basis of the event detection signal; and a brightness detection unit which detects, in the region of interest, a brightness signal in which is stored the photocurrent generated by the photoelectric conversion element.

Description

Detector and sensor

The present invention relates to a detection device and a sensor.

Conventionally, a detection device for identifying a region of interest (ROI) from a captured original image is known (see, for example, Patent Document 1).
Patent Document 1: Japanese Patent Application Publication No. 2006-157615

General disclosure

However, it is desirable to identify the ROI with less data processing.

In the first aspect of the present invention, in a pixel block constituted by one or a plurality of pixels each having at least one photoelectric conversion element, an event detection in which the value of photocurrent generated in the photoelectric conversion element is predetermined An event detection unit that detects an event indicating exceeding a threshold and outputs an event detection signal, a readout control unit that identifies a region of interest based on the event detection signal, and light generated by the photoelectric conversion element in the region of interest Provided is a detection device including: a luminance detection unit that detects a luminance signal in which current is accumulated.

A second aspect of the invention provides a sensor comprising one or more pixels and a detection device according to the first aspect of the invention.

The above summary of the invention does not enumerate all of the features of the present invention. In addition, a subcombination of these feature groups can also be an invention.

The outline of composition of sensor 300 is shown. An example of composition of sensor 300 is shown. 5 is an example of a flowchart showing a detection operation by the detection device 100. An example of the ROI identification method by detection apparatus 100 concerning an example is shown. An example of the ROI identification method by the detecting device concerning a comparative example is shown. An example of embodiment of the misdetection prevention by the detection apparatus 100 is shown. 5 shows an example of an embodiment of data reduction by the detection device 100. FIG. An example of composition of sensor 300 is shown. An example of composition of sensor 300 is shown. An example of composition of sensor 300 is shown. An example of composition of sensor 300 is shown. An example of the structure of the first substrate 110 is shown. An example of a structure of the 1st board | substrate 110 is shown. An example of a structure of the 1st board | substrate 110 is shown.

Hereinafter, the present invention will be described through the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Moreover, not all combinations of features described in the embodiments are essential to the solution of the invention.

FIG. 1 shows an outline of the configuration of the sensor 300. The sensor 300 includes the pixel unit 10 and the detection device 100.

The pixel unit 10 includes one or more pixels 11. The pixel unit 10 of this example includes a plurality of pixels 11 arranged in a two-dimensional manner. The pixel unit 10 in this example has M × N pixels 11. The pixel 11 has at least one photoelectric conversion element. The pixel unit 10 includes a pixel block 12 configured by one or more pixels 11. The pixel block 12 is an area including m × n (m and n are natural numbers) pixels 11. The photoelectric conversion element is, for example, a photodiode (PD).

The detection device 100 includes an event detection unit 20, an event processing unit 30, a luminance detection unit 40, and a signal processing unit 50. The detection device 100 is electrically connected to the pixel unit 10. The detection apparatus 100 detects an event and acquires an image of a region of interest (ROI) according to the detected event. By specifying the ROI, the detection apparatus 100 can detect the motion of the detection target.

The event detection unit 20 detects an event. In the present specification, an event indicates that the amount of change in photocurrent generated in the photoelectric conversion element exceeds a predetermined threshold (event detection threshold). Here, the event detection threshold value is the amount of change from the photocurrent of the photoelectric conversion element detected at the previous time, and a relative value from the previous time may be defined. A value may be defined. When the photocurrent changes to the event detection threshold value or more, there is a high possibility that there is any change in the imaging target imaged by the pixel 11 including the photoelectric conversion element. The event detection unit 20 generates an event detection signal S _ID based on the photocurrent from the pixel unit 10. For example, the event detection signal S _ID is a luminance value corresponding to the pixel 11. The event detection unit 20 outputs the generated event detection signal S _ID to the event processing unit 30. The event detection unit 20 may be provided in the pixel unit 10.

The event processing unit 30 processes the event detection signal S _ID output from the event detection unit 20. The event processing unit 30 specifies an ROI for reading based on the event detection signal S _ID . The event processing unit 30 of this example specifies whether or not it is an ROI in units of pixel blocks 12. The event processing unit 30 outputs information on the identified ROI to the luminance detection unit 40.

Luminance detector 40 detects the luminance signal S _L accumulated light current generated in the photoelectric conversion element. Luminance detector 40 of this example detects the luminance signal S _L of the photoelectric conversion elements corresponding to the ROI to the event processing section 30 has identified. Brightness detection unit 40, of the luminance signal S _L accumulated photocurrent in all the pixels of one or more pixels 11, and outputs the luminance signal S _L detected by the ROI to the signal processing unit 50. In other words, the brightness detection unit 40 does not need to output the luminance signal S _L in the region other than the ROI in the signal processing unit 50. However, the luminance detection unit 40 may detect the luminance signal S _L from an area other than the ROI for the purpose other than acquisition of an image, for example, measurement of automatic exposure (AE) or autofocus (AF). . Although the luminance detection unit 40 of this example is provided in the detection device 100, it may be provided in the pixel unit 10.

The signal processing unit 50 processes the luminance signal S _L to the brightness detection unit 40 has detected. Thereby, the signal processing unit 50 acquires an image of the subject. For example, the signal processing unit 50, by processing the luminance signal _{S L,} to acquire an image of ROI. The signal processing unit 50 may function as a data output unit that outputs the acquired image to the outside.

The detection apparatus 100 of this example processes only the luminance signal S _L detected in the ROI among the luminance signals S _L in which the photocurrent is accumulated in all the pixels of one or a plurality of pixels 11. Accordingly, the detection device 100, than the case of outputting the luminance signal S _L of all the pixels to the signal processor 50, power consumption can be reduced.

The detection apparatus 100 may detect an event at any timing using the event detection unit 20. The event detection unit 20 adjusts an event detection threshold for detecting an event of the pixel block 12 according to the luminance of the ROI. For example, when the amount of change in luminance of the ROI becomes large, the event detection unit 20 reduces the amount of change in photocurrent, which is an event detection threshold. On the other hand, when the amount of change in luminance of the ROI decreases, the event detection unit 20 increases the amount of change in photocurrent, which is an event detection threshold. Specifically, when the change amount per unit time of the luminance of the ROI is equal to or greater than a predetermined threshold, the event detection threshold (the change amount of the photocurrent) is set as a first value, and the event is less than the predetermined threshold. The detection threshold value is set to a second value larger than the first value. Thus, the detection device 100 can easily follow the change in the detection target when the change in the luminance of the ROI is large, and it is difficult to detect the event detection signal S _ID when the change in the luminance of the ROI does not change so much. As a whole, low power consumption can be realized. Note that the predetermined threshold and the event detection threshold adjusted accordingly may be multistages more than two stages.

Also, the size of the pixel block 12 may be any size. The event detection unit 20 adjusts the size of the pixel block 12 according to the luminance of the ROI. For example, the event detection unit 20 reduces the pixel block 12 when the amount of change in luminance of the ROI increases. On the other hand, the event detection unit 20 enlarges the pixel block 12 when the amount of change in luminance of the ROI becomes small.

Specifically, when the amount of change in luminance of the ROI per unit time is equal to or larger than a predetermined threshold, the size of the pixel block 12 is set to a first size, and when smaller than the predetermined threshold, the size of the pixel block 12 To a second magnitude that is greater than the first value. Thus, the detection device 100 can easily follow the change in the detection target when the change in the luminance of the ROI is large, and it is difficult to detect the event detection signal S _ID when the change in the luminance of the ROI does not change so much. As a whole, low power consumption can be realized. Note that the predetermined threshold value and the size of the pixel block 12 adjusted accordingly may be multistages more than two stages or more.

FIG. 2 shows an example of the configuration of the sensor 300. The pixel unit 10 of this example includes an event detection unit 20 and a luminance detection unit 40 in addition to the photoelectric conversion element. The sensor 300 is, for example, a CMOS image sensor.

The pixel 11 includes a first photoelectric conversion element PD1, a second photoelectric conversion element PD2, an event detection unit 20, and a luminance detection unit 40. That is, the pixel 11 of this example includes both of the first photoelectric conversion element PD1 for event detection and the second photoelectric conversion element PD2 for luminance detection. The first photoelectric conversion element PD1 and the second photoelectric conversion element PD2 in the pixel 11 are independent of each other. Moreover, when the pixel 11 is provided with two or more, each pixel 11 is independent.

The first photoelectric conversion element PD1 is a photoelectric conversion element for event detection. The first photoelectric conversion element PD1 outputs the photocurrent generated in response to light reception to the event detection unit 20.

The second photoelectric conversion element PD2 is a photoelectric conversion element for luminance detection. The second photoelectric conversion element PD <b> 2 accumulates the photocurrent generated in response to the light reception and outputs the accumulated photocurrent to the luminance detection unit 40.

The event processing unit 30 includes a space density processing unit 32 and a read control unit 34. The event processing unit 30 of this example is provided outside the pixel unit 10.

The spatial density processing unit 32 receives the event detection signal S _ID detected from the event detection unit 20. The spatial density processing unit 32 calculates the spatial density of the event detection signal S _ID for each pixel block 12.

Readout control unit 34 controls the reading of the luminance signal S _L detected of the luminance detector 40. In one example, the read control unit 34 specifies the ROI based on the spatial density of the event detection signal S _ID calculated by the spatial density processing unit 32. The ROI is an image acquisition area where the sensor 300 acquires an image. The read control unit 34 of this example determines whether or not each pixel block 12 is an ROI. Thereby, the read control unit 34 can reduce the required number of data as compared with the case where the ROI is determined for each pixel 11.

For example, when the spatial density of the event detection signal S _ID is higher than a predetermined threshold, the read control unit 34 specifies it as the ROI. Further, the read control unit 34 determines that it is not the ROI when the spatial density of the event detection signal S _ID is lower than a predetermined threshold. The read control unit 34 causes the brightness signal _SL to be read only from the brightness detection unit 40 corresponding to the ROI.

The read control unit 34 may drive each pixel of the plurality of pixels 11 asynchronously. For example, the read control unit 34 controls the driving of the event detection unit 20 and the luminance detection unit 40. The read control unit 34 can drive the first photoelectric conversion element PD1 for event detection and the second photoelectric conversion element PD2 for luminance detection independently of each other without synchronizing them. Therefore, the read control unit 34 may drive the first photoelectric conversion element PD1 asynchronously with the imaging frame while driving the second photoelectric conversion element PD2 corresponding to the imaging frame. In this case, in the second photoelectric conversion element PD2, accumulation and reset of photocurrent are repeated for each imaging frame, and the ROI is set to the ROI at the timing when the event is detected in the first photoelectric conversion element PD1. luminance signal S _L from the second photoelectric conversion element PD2 of the pixel 11 is output. Therefore, the detection apparatus 100 of the present example can detect a change in an event faster than an imaging frame because there is no concept of an imaging frame with respect to detection of an event. The event detection method of the present example is particularly effective when the change of the event is large or when the event is detected at the time of moving image shooting. The imaging frame indicates a frame rate at the time of imaging, and is, for example, 30 fps or 60 fps.

FIG. 3 is an example of a flowchart showing a detection operation by the detection device 100. The detection apparatus 100 detects the luminance signal of the ROI according to the detected event by executing steps S100 to S104. In the pixel block 12 composed of one or a plurality of pixels 11 each having at least one photoelectric conversion element, the detection device 100 sets an event detection threshold value in which the amount of change in photocurrent generated by the photoelectric conversion elements is predetermined. An exceeding event is detected (S100). The detection apparatus 100 specifies an ROI for reading based on the detection of an event (S102). Then, in the ROI, the detection apparatus 100 detects a luminance signal in which the photocurrent generated by the photoelectric conversion element is accumulated (S104). Steps S100 to S104 may be performed simultaneously for all pixel blocks 12, or may be performed separately for each pixel block 12.

FIG. 4 shows an example of a method of specifying an ROI by the detection apparatus 100 according to the embodiment. The detection apparatus 100 of the present example identifies an ROI corresponding to the event detection target 200, and partially acquires an image of the event detection target 200. For example, when a human being an event detection target 200 is riding a bicycle and is traveling on a road, the detecting device 100 acquires only an image of the human.

(A) of FIG. 4 shows the entire screen including a person and a road. The detection device 100 of this example acquires an event detection signal S _ID from a part of the pixels 11 among all the pixels 11 corresponding to the entire screen. For example, the detection apparatus 100 acquires an event detection signal S _ID from a pixel 11 whose change amount of luminance value is larger than a predetermined threshold value, and plots the pixel 11 in white. On the other hand, the detection device 100 plots, in black, pixels whose luminance variation is smaller than a predetermined threshold, that is, pixels 11 which do not acquire the event detection signal S _ID . That is, as shown in (b) of FIG. 4, the detection apparatus 100 acquires binarized data on the entire screen according to whether or not an event is detected. Therefore, the detection apparatus 100 does not have to acquire an image of the entire screen as shown in (a) of FIG. 4 when specifying the ROI. The detection apparatus 100 of this example divides the entire screen into 6 × 7 = 42 pixel blocks 12.

Next, as shown in (c) of FIG. 4, the detection apparatus 100 extracts an ROI from the spatial density of the event detection signal S _ID . The spatial density of the event detection signal S _ID is a ratio of the pixels 11 in which an event is detected among all the pixels 11 included in a certain pixel block 12. The detection apparatus 100 specifies the pixel block 12 as the ROI when the spatial density of the event detection signal S _ID is larger than a predetermined threshold value in a certain pixel block 12.

The event detection unit 20 may set the upper limit value and the lower limit value of the spatial density of the event detection signal S _ID . In this case, the read control unit 34 specifies an area in which the spatial density of the event detection signal S _ID is included in the range between the upper limit value and the lower limit value as the ROI. Also, the event detection unit 20 may set the upper limit value and the lower limit value of the spatial density of the event detection signal S _ID according to the luminance of the ROI.

For example, even though the size of the pixel block 12 is a certain size or more (for example, 1⁄2 or 1⁄4 of the entire screen, etc.), the spatial density of the event detection signal S _ID exceeds the upper limit value. There are cases where noise is included in the whole pixel block 12 due to an external factor such as disturbance light. By setting the upper limit value of the spatial density of the event detection signal S _ID , it is possible to prevent the erroneous identification of the ROI when the entire pixel block 12 includes noise. In addition, when the spatial density of the event detection signal S _ID exceeds the lower limit value, specifying as the ROI is the same as the case where the event detection threshold is defined by one value.

The detection apparatus 100 according to the present embodiment does not have to acquire a large volume of image data in the acquisition stage of the event detection signal S _ID , and may obtain binarized low volume image data. Thus, the detection device 100 can compress the amount of data needed to identify the ROI.

FIG. 5 shows an example of a method of specifying an ROI by a detection device according to a comparative example. The detection device of the comparative example acquires an image of the entire screen including a person and a road, which is an event detection target 200 ((a) in FIG. 5). Next, the detection device of the comparative example specifies an ROI including the event detection target 200 from the image of the entire screen as post processing ((b) in FIG. 5). And the detection apparatus of a comparative example extracts only the image of ROI from the image of the full screen acquired by (b) ((c) of FIG. 5). That is, in the detection device of the comparative example, since the image of the entire screen is acquired to specify the ROI, it is necessary to handle a large volume of data. Furthermore, the ROI can only be detected at intervals of imaging frames.

The read control unit 34 in this example calculates the event detection signal S _ID for each pixel 11 to specify the ROI. However, as described later, the readout control unit 34 determines a representative event detection result for each of the plurality of pixels 11 to specify the ROI based on the event detection signal S _ID representative of the plurality of pixels 11. You may

Here, a representative event detection result is a plurality of event detection signals corresponding to each of a plurality of pixels 11 constituting a set (a set smaller than the pixel block 12) consisting of a certain pixel 11 and its peripheral pixels 11 of S _ID, an event detection signal _{S ID} representative of the set. The read control unit 34 can reduce the data amount of the event detection signal S _ID by using a representative event detection result. Further, the readout control unit 34 can prevent erroneous detection when noise is included in some of the pixels 11 of the plurality of pixels 11 by using a representative event detection result.

FIG. 6 shows an example of an embodiment of the erroneous detection prevention by the detection device 100. The detection apparatus 100 of this example reduces false detection of an event by applying a filter process to the event detection signal S _ID . The peripheral pixel area 15 is an area including a × b (a and b are natural numbers) pixels 11. The peripheral pixel area 15 of this example includes a × b = 3 × 3 = 9 pixels 11. The event detection unit 20 outputs “1” when an event is detected, and outputs “0” when an event is not detected. In this example, although the case where the event detection unit 20 binarizes is described, it is not limited thereto. In FIG. 6, an arbitrary pixel 11 is indicated by (X, Y), with the upper left pixel 11 as the origin (0, 0), the right direction as the positive X direction, and the lower direction as the positive Y direction.

In the example illustrated in FIG. 6, the event detection unit 20 detects an event in the pixel 11 of (a, b) = (2, 2). On the other hand, the event detection unit 20 does not detect an event in the remaining eight pixels 11. In this case, in the pixel 11 of (a, b) = (2, 2), there is a high possibility that an event is erroneously detected. For example, false detection of an event is caused by noise such as light shot noise or fluctuation of a light source. The event detection unit 20 of this example changes the pixel 11 of (a, b) = (2, 2) to “0” by filter processing using an AND circuit with peripheral pixels. Therefore, in the peripheral pixel area 15 of this example, “0” is a representative event detection result.

Moreover, although the event detection part 20 used AND circuit as a filter circuit applied to the peripheral pixel area 15, the kind of filter circuit is not limited to this. For example, the detection apparatus 100 may calculate a representative event detection result using the peripheral pixel area 15 by using another filter such as a median filter circuit. As described above, the filter circuit of the event detection unit 20 only needs to be able to perform filter processing to remove noise.

The readout control unit 34 filters a representative event detection result representing the a × b event detection results from the event detection results of the peripheral pixel area 15, and based on the representative event detection result, the ROI is calculated. Identify.

Thus, the detection device 100 of the present example reduces false detection of an event by filtering the peripheral pixel region 15. Thus, the detection device 100 can improve the image quality of the captured image.

The event detection unit 20 of this example compares and outputs the event detection signal S _ID input in a predetermined period in the peripheral pixel area 15. In other words, the event detection unit 20 may compare the event detection signal S _ID input in a different period for each pixel 11 in the peripheral pixel region 15. That is, the event detection signal S _ID compared in the peripheral pixel area 15 is not limited to one included in the same image. This is because the plurality of pixels 11 in the peripheral pixel area 15 are not completely identical because the event detection unit 20 detects an event independently. However, it is preferable that the event detection unit 20 detects an event in the peripheral pixel region 15 using a common event detection threshold.

Here, the pixel block 12 is a pixel area larger than the peripheral pixel area 15. That is, in other words, the pixel block 12 may be configured by a plurality of peripheral pixel areas 15. In this case, the detection apparatus 100 can regard the peripheral pixel area 15 as one pixel by calculating an event detection result representing the peripheral pixel area 15. In this case, although the resolution of data to be binarized on the entire screen in order to specify the ROI region is low, the data amount of the event detection signal S _ID can be reduced.

The event detection unit 20 has one or more event detection thresholds in order to determine the presence or absence of an event. In one example, the event detection unit 20 has one event detection threshold, and determines the presence or absence of an event with the same event detection threshold for all the pixels 11 included in the peripheral pixel area 15. Further, the event detection unit 20 may have a plurality of event detection thresholds, and may determine the presence or absence of an event with respect to the pixels 11 included in the peripheral pixel area 15 using different event detection thresholds. For example, the event detection unit 20 detects an event of the first pixel 11 included in the peripheral pixel area 15 based on comparison with the first event detection threshold. In addition, the event detection unit 20 may detect an event for the second pixel 11 different from the first pixel 11 based on comparison between the first event detection threshold and a second event detection threshold different from the first event detection threshold.

FIG. 7 shows another example of an embodiment of data reduction by the detection device 100. The detection device 100 of this example reduces the number of data by filtering the event detection signal S _ID . For example, consider a × b = 3 × 3 = 9 pixels 11. When an event is detected in all pixels, a typical event detection result remains with the pixel 11 of (a, b) = (2, 2) being "1" by filter processing using an AND circuit with peripheral pixels I assume. If the event detection signal S _ID is output for all pixels, the number of data is nine. On the other hand, the number of data can be reduced to 1 by executing the filtering process on the event detection signal S _ID of nine pixels of a × b = 3 × 3. The detection device 100 of this example can be treated as one piece of data because the filter size is known.

In addition, the size of the peripheral pixel area 15 may be any size. In one example, the event detection unit 20 adjusts the size of the peripheral pixel area 15 according to the luminance of the ROI. For example, the event detection unit 20 sets the peripheral pixel region 15 to the first size when the amount of change in luminance of the ROI per unit time is equal to or greater than a predetermined threshold. On the other hand, when the amount of change in luminance of the ROI falls below a predetermined threshold, the event detection unit 20 sets the peripheral pixel area 15 to a second size larger than the first size. Thus, the detection apparatus 100 can increase the data reduction effect when the change in the event is small, and can improve the event detection accuracy when the change in the event is large. Note that the predetermined threshold value and the size of the peripheral pixel area 15 adjusted accordingly may be multistages more than two stages.

As described above, the detection device 100 of this example reduces the data amount of the peripheral pixel area 15 to specify the ROI. Thereby, the detection apparatus 100 can realize high-speed reading and reduction of power consumption. The detection apparatus 100 may identify the ROI and reduce the data amount based on at least one of the sum of events, the density, and the change.

FIG. 8 shows an example of the configuration of the sensor 300. The sensor 300 of this example comprises a first substrate 110 and a second substrate 120.

The first substrate 110 has a pixel block array 112. The pixel block array 112 includes a first photoelectric conversion element PD1, a second photoelectric conversion element PD2, an event detection unit 20, and a luminance detection unit 40 for each of the plurality of pixels 11. That is, the pixel 11 of this example includes both the first photoelectric conversion element PD1 for event detection and the second photoelectric conversion element PD2 for luminance detection.

The second substrate 120 has a processing block array 122 and an imaging control unit 60. The second substrate 120 includes a space density processing unit 32, a read control unit 34, and a signal processing unit 50. The second substrate 120 is stacked on the first substrate 110.

The imaging control unit 60 controls imaging conditions of the detection device 100. In one example, the imaging control unit 60 controls the conditions of auto focus (AF) and automatic exposure (AE) according to the ROI detected by the read control unit 34. For example, the imaging control unit 60 executes optimum AF and AE in accordance with the luminance and the like in the ROI. Thereby, the detection apparatus 100 can appropriately capture an image of the ROI.

FIG. 9 shows an example of the configuration of the sensor 300. The sensor 300 of this example comprises a first substrate 110 and a second substrate 120. The first substrate 110 and the second substrate 120 are provided to be stacked on each other. The sensor 300 is, for example, a stacked CMOS image sensor.

The first substrate 110 has M × N pixels 11. Further, the first substrate 110 has a pixel block 12 composed of m × n pixels 11. The first substrate 110 may have a plurality of pixel blocks 12.

Each pixel 11 includes a first photoelectric conversion element PD1, a second photoelectric conversion element PD2, an event detection unit 20, and a luminance detection unit 40. As described above, the event detection unit 20 and the luminance detection unit 40 of the present example are provided corresponding to the first photoelectric conversion element PD1 and the second photoelectric conversion element PD2, respectively.

The second substrate 120 has an event processing unit 30 provided corresponding to the pixel block 12. The event processing unit 30 has a space density processing unit 32 and a read control unit 34, respectively. The event processing unit 30 and the corresponding pixel block 12 are electrically connected. The event processing unit 30 of this example executes event processing of the pixel blocks 12 provided correspondingly. The event processing unit 30 is preferably provided immediately below the corresponding pixel block 12. As a result, the wiring connecting the corresponding pixel block 12 and the event processing unit 30 becomes short, so that the processing speed of the detection device 100 is improved. Further, the set of the pixel block 12 and the event processing unit 30 is independent of the set of the adjacent pixel block 12 and the event processing unit 30. In the sensor 300 of this example, the event processing unit 30 is configured on the second substrate 120, whereby the size of the photoelectric conversion element PD can be increased, and a highly sensitive sensor can be realized.

FIG. 10 shows an example of the configuration of the sensor 300. In the sensor 300 of this example, the event detection unit 20 and the luminance detection unit 40 are provided on a substrate different from the photoelectric conversion element PD. The sensor 300 comprises a first substrate 110, a second substrate 120 and a third substrate 130. The first substrate 110, the second substrate 120, and the third substrate 130 are provided stacked on one another.

The first substrate 110 has a pixel block array 112. The pixel block array 112 in this example has one or more pixels 11. A photoelectric conversion element PD1 and a second photoelectric conversion element PD2 are provided in the pixel 11 of this example.

The second substrate 120 has a processing block array 122. The processing block array 122 of this example includes an event detection unit 20 and a luminance detection unit 40. It is preferable that the event detection unit 20 and the luminance detection unit 40 be provided to face the corresponding pixels 11 respectively. As a result, the wiring connecting the pixel block array 112 and the processing block array 122 is shortened.

The third substrate 130 has an output block array 132. The output block array 132 of this example includes a space density processing unit 32, a read control unit 34, and a signal processing unit 50. Since the sensor 300 of this example does not have the event detection unit 20 and the luminance detection unit 40 on the first substrate 110, the size of the photoelectric conversion element PD can be increased, and a highly sensitive sensor can be realized. .

FIG. 11 shows an example of the configuration of the sensor 300. The basic configuration of the sensor 300 is the same as that of the sensor 300 of FIG. 10, and shows a specific configuration for realizing the function shown in FIG.

The first substrate 110 has pixel blocks 12 of m × n size. The pixel 11 of this example has a first photoelectric conversion element PD1 and a second photoelectric conversion element PD2.

The second substrate 120 has an imaging control unit 60 provided corresponding to the pixel block 12. The imaging control unit 60 controls driving of the pixels 11 provided correspondingly. The imaging control unit 60 includes an event detection unit 20 and a luminance detection unit 40. The event detection unit 20 is provided corresponding to the first photoelectric conversion element PD1. The luminance detection unit 40 is provided corresponding to the second photoelectric conversion element PD2.

The third substrate 130 has an event processing unit 30 provided corresponding to the imaging control unit 60. The event processing unit 30 receives event information and luminance information. The event processing unit 30 specifies the ROI based on the event information, and outputs the luminance information of the ROI to the signal processing unit 50.

FIG. 12 shows an example of the configuration of the first substrate 110. The first substrate 110 in this example includes a first pixel group 13 and a second pixel group 14.

The first pixel group 13 includes a third photoelectric conversion element PD3, an event detection unit 20, and a luminance detection unit 40, and is shown in white in the drawing. The third photoelectric conversion element PD3 is connected to the event detection unit 20 and the luminance detection unit 40.

The third photoelectric conversion element PD3 doubles as the functions of the first photoelectric conversion element PD1 and the second photoelectric conversion element PD2. That is, the third photoelectric conversion element PD3 functions as a photoelectric conversion element for event detection and a photoelectric conversion element for luminance detection. Thereby, the event detection unit 20 detects an event according to the signal from the third photoelectric conversion element PD3, and the luminance detection unit 40 detects luminance according to the signal from the third photoelectric conversion element PD3. it can. For example, the third photoelectric conversion element PD3 switches and executes an event detection function and a luminance detection function.

The second pixel group 14 includes the pixels 11 that detect only the luminance without detecting an event. The second pixel group 14 has a second photoelectric conversion element PD2 and a luminance detection unit 40, and is shown by hatching in the figure. Thereby, the second pixel group 14 detects the luminance according to the signal from the second photoelectric conversion element PD2. Since the second pixel group 14 is used as a pixel dedicated to the luminance detection unit 40, the second pixel group 14 is driven at an arbitrary timing not limited to the drive timing of the event detection unit 20.

The first pixel group 13 of this example is disposed discontinuously across the second pixel group 14 in a predetermined direction. That is, the event detection unit 20 does not have to be provided in all the pixels 11. The first pixel group 13 may be thinned out so as to have the density necessary to specify the ROI. Thus, the circuit configuration of the first substrate 110 can be simplified and power consumption can be reduced.

The first pixel group 13 of this example is provided continuously in the column direction in the pixels 11 of M × N size. In addition, the second pixel group 14 of this example is continuously provided in the column direction in the pixels 11 of the M × N size. The first pixel group 13 and the second pixel group 14 are alternately arranged in the row direction. However, the first pixel group 13 and the second pixel group 14 are not limited to the arrangement method of this example as long as they are arranged with the distribution necessary for processing the event detection signal S _ID . For example, the first pixel group 13 and the second pixel group 14 may be arranged mutually in a staggered manner.

FIG. 13 shows an example of the configuration of the first substrate 110. The first substrate 110 in this example includes a first pixel group 13 and a second pixel group 14.

The first pixel group 13 includes a first photoelectric conversion element PD1, a second photoelectric conversion element PD2, an event detection unit 20, and a luminance detection unit 40. That is, the first pixel group 13 of this example includes both of the first photoelectric conversion element PD1 for event detection and the second photoelectric conversion element PD2 for luminance detection. The event detection unit 20 is connected to the first photoelectric conversion element PD1. Further, the event processing unit 30 is connected to the second photoelectric conversion element PD2.

The second pixel group 14 includes a second photoelectric conversion element PD2 and a luminance detection unit 40. The luminance detection unit 40 is connected to the second photoelectric conversion element PD2. The second pixel group 14 is two-dimensionally arranged with the first pixel group 13.

The first pixel group 13 of this example is provided continuously in the column direction in the pixels 11 of M × N size. In addition, the second pixel group 14 of this example is continuously provided in the column direction in the pixels 11 of the M × N size. The first pixel group 13 and the second pixel group 14 are alternately arranged in the row direction. However, as long as the first pixel group 13 and the second pixel group 14 are arranged in a distribution necessary for processing of the event detection signal S _ID , the arrangement method of the present embodiment is not limited to that shown in FIG. It is the same as the example.

FIG. 14 shows an example of the configuration of the first substrate 110. The first substrate 110 of this example is different from the case of FIG. 13 in the method of arranging the first pixel group 13 and the second pixel group 14.

The first pixel group 13 in the present example is disposed more around the first substrate 110 than inside the first substrate 110. That is, the first pixel group 13 is densely arranged around the pixels 11 arranged in a two-dimensional manner than the centers of the plurality of pixels 11 arranged in a two-dimensional manner. Therefore, the detection device 100 can acquire more event detection signals S _ID than in the pixel unit 10 on the outer periphery of the pixel unit 10. Thus, the detection apparatus 100 can easily detect an event on the outer periphery of the first substrate 110.

Here, an event occurs outside the pixel unit 10 when, for example, an imaging target comes in from the outside of the image. The detection device 100 of the present example has high detection sensitivity of an event on the outer periphery of the pixel unit 10, so that it becomes easy to find an imaging target. Therefore, when the imaging target object intrudes into the imaging region, the detection apparatus 100 can promptly detect an event and capture only the imaging target object.

As described above, in the detection device 100, the spatial density processing unit 32 determines the ROI based on the spatial density of the event detection signal S _ID . Therefore, the detection apparatus 100 does not have to specify the ROI in advance. In addition, the sensor 300 can appropriately update the ROI in response to the occurrence of an event. The sensor 300 can also follow unexpected behavior. As a result, when adding or changing a region during ROI operation, the sensor 300 does not require post-processing calculation or additional image acquisition, and can suppress increase in power and deterioration in latency. Furthermore, the sensor 300 can automatically select a target area of auto focus (AF) or auto exposure (AE).

As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. For example, the first substrate 110 as shown in FIGS. 12 to 14 can be applied to any of the stacked sensors as shown in FIGS. 7 to 10. It is apparent to those skilled in the art that various changes or modifications can be added to the above embodiment. It is also apparent from the scope of the claims that the embodiments added with such alterations or improvements can be included in the technical scope of the present invention.

The order of execution of each process such as operations, procedures, steps, and steps in the apparatuses, systems, programs, and methods shown in the claims, the specification, and the drawings is particularly "before", "before" It should be noted that it can be realized in any order, unless explicitly stated as etc., and unless the output of the previous process is used in the later process. With regard to the operation flow in the claims, the specification, and the drawings, even if it is described using “first,” “next,” etc. for convenience, it means that it is essential to carry out in this order. is not.

DESCRIPTION OF SYMBOLS 10 ... pixel part, 11 ... pixel, 12 ... pixel block, 13 ... 1st pixel group, 14 ... 2nd pixel group, 15 ... periphery pixel area, 20 ... · · · Event detection unit, 30 ... Event processing unit, 32 ... Space density processing unit, 34 ... Readout control unit, 40 ... Brightness detection unit, 50 ... Signal processing unit, 60 · · · Imaging control unit 100 Detection device 110 First substrate 112 Pixel block array 120 Second substrate 122 Processing block array 130 Third substrate, 132: Output block array, 200: Event detection target, 300: Sensor

Claims

In a pixel block composed of one or more pixels each having at least one photoelectric conversion element, an event indicating that the value of photocurrent generated by the photoelectric conversion element exceeds a predetermined event detection threshold is detected An event detection unit that outputs an event detection signal,
A readout control unit that identifies a region of interest based on the event detection signal;
A detection unit that detects a luminance signal in which the photocurrent generated by the photoelectric conversion element is accumulated in the region of interest.
The signal processing unit further includes a signal processing unit that processes a signal detected by the luminance detection unit.
The luminance detection unit outputs, to the signal processing unit, the luminance signal detected in the region of interest among the luminance signals in which the photocurrent is accumulated in all the pixels of the one or a plurality of pixels. Detection device as described.
The detection apparatus according to claim 1, wherein the readout control unit specifies the region of interest according to a spatial density of the event detection signal output from the event detection unit.
The event detection unit sets an upper limit value and a lower limit value of the spatial density of the event detection signal according to the luminance of the region of interest.
The detection device according to claim 2, wherein the read control unit specifies a region in which a space density of the event detection signal is included in the range of the upper limit value and the lower limit value as the region of interest.
The readout control unit filters a representative event detection result from an event detection result of m × n pixels (m and n are natural numbers), and the region of interest is filtered based on the representative event detection result. The detection device according to claim 1 or 2.
The detection device according to claim 5, wherein the event detection unit detects an event using the common event detection threshold in the m × n pixels.
The pixel block is a pixel area larger than the m × n pixels,
The event detection unit detects an event of a first pixel included in the pixel block based on comparison with a first event detection threshold, and the second pixel different from the first pixel detects the event. The detection device according to claim 5, wherein an event is detected based on a comparison between the one event detection threshold and a second event detection threshold different from one another.
The detection apparatus according to any one of claims 1 to 7, wherein the event detection threshold is defined as a range of a change amount of photocurrent generated in the photoelectric conversion element.
The detection apparatus according to any one of claims 1 to 7, wherein the event detection threshold is set as an absolute value of the photocurrent.
The detection device according to any one of claims 1 to 9, wherein each of the one or more pixels includes a first photoelectric conversion element for event detection and a second photoelectric conversion element for luminance detection.
The one or more pixels are
A first pixel group including a first photoelectric conversion element for event detection and a second photoelectric conversion element for luminance detection;
The detection device according to any one of claims 1 to 9, further comprising: a second pixel group that is two-dimensionally arranged with the first pixel group and includes a second photoelectric conversion element for luminance detection.
The detection device according to claim 11, wherein the first pixel group is disposed discontinuously across the second pixel group in a predetermined direction.
The detection device according to claim 11, wherein the first pixel group is densely arranged in the periphery of the two-dimensionally arranged pixels than the center of the two-dimensionally arranged pixels.
The detection device according to any one of claims 1 to 9, wherein the one or more pixels include a third photoelectric conversion element serving also as an event detection and a luminance detection.
The detection device according to any one of claims 1 to 14, wherein the event detection unit adjusts the event detection threshold according to the luminance of the region of interest.
The detection device according to any one of claims 1 to 15, wherein the event detection unit adjusts the size of the pixel block in accordance with the luminance of the region of interest.
The detection device according to any one of claims 1 to 16, wherein the read control unit asynchronously drives each pixel of the plurality of pixels.
One or more pixels,
A sensor comprising the detection device according to any one of claims 1 to 17.
A first substrate provided with the one or more pixels, the event detection unit, and the luminance detection unit;
The sensor according to claim 18, further comprising: a second substrate stacked on the first substrate and provided with the read control unit.
A first substrate provided with the one or more pixels;
A second substrate stacked on the first substrate and provided with the event detection unit and the luminance detection unit;
The sensor according to claim 18, further comprising: a third substrate stacked on the second substrate and provided with the read control unit.
In a pixel block composed of one or more pixels each having at least one photoelectric conversion element, an event indicating that the value of photocurrent generated by the photoelectric conversion element exceeds a predetermined event detection threshold is detected And
Identifying a region of interest based on the detection of the event;
Detecting the luminance signal in which the photocurrent generated by the photoelectric conversion element is accumulated in the region of interest.