JP7357477B2 - Imaging device and its manufacturing method - Google Patents

Description

The present invention relates to an imaging device and a method of manufacturing the same.

In recent years, with improvements in packaging technology, a stacked image sensor in which an image sensor (hereinafter referred to as the first layer) and an LSI (hereinafter referred to as the second layer) are stacked has been developed, and digital cameras etc. equipped with such sensors have been developed. The product is on the market. The first layer contains pixels that receive light and generate pixel signals, and the vertical lines that connect the pixels in the vertical direction (Y direction) on the first layer, and the second layer performs data processing based on image signals. A processing circuit can be provided respectively. For example, Patent Document 1 discloses that a processing circuit provided in the second layer performs exposure correction, white balance correction, dark correction, and the like.

JP 2018-19227 Publication

Depending on the application, various image sensors with different numbers of pixels and vertical lines are required. Generally, when designing a stacked image sensor, the number of pixels and the number of vertical lines are selected depending on the application, and the hardware configuration of the second layer is designed according to the number of pixels and the number of vertical lines. On the other hand, there is also a need to reduce the manufacturing cost of image sensors.

In a stacked image sensor, the present invention enables a common second layer having a processing circuit to be stacked on image sensors (first layer) having different numbers of pixels or vertical lines. The purpose is to reduce development costs or manufacturing costs.

In order to achieve the object of the present invention, an imaging device according to an embodiment of the present invention has the following configuration. That is,
a plurality of pixels provided in the first layer and generating pixel signals;
a plurality of vertical lines provided in the first layer and connecting the plurality of pixels in the vertical direction of the first layer;
Data processing based on pixel signals obtained from the first layer, which is provided in a second layer stacked on the first layer, and whose processing parameters can be controlled according to the number of pixels or the number of vertical lines. A processing circuit that performs
In a first period, pixel signals from a plurality of pixel rows are transmitted in parallel from the first layer to the second layer.

In a stacked image sensor, for example, it is possible to stack a common second layer having a processing circuit on image sensors (first layer) having different numbers of pixels or vertical lines, thereby reducing the development cost of the stacked image sensor. Manufacturing costs can be reduced.

FIG. 1 is a diagram illustrating a configuration example of an image sensor according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of a data processing unit according to an embodiment. FIG. 3 is a diagram illustrating a configuration example of a black level correction section according to an embodiment. FIG. 3 is a diagram showing variations of the first layer and examples of processing parameters. 5 is a timing chart of data processing according to an embodiment. 5 is a timing chart of data processing according to an embodiment. FIG. 3 is a diagram illustrating a configuration example of a black level correction section according to an embodiment. 5 is a timing chart of data processing according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

An imaging device according to an embodiment of the present invention is a stacked image sensor, and includes a first layer and a second layer stacked on the first layer. The first layer is provided with a plurality of pixels that generate pixel signals. Further, the second layer is provided with a processing circuit that performs data processing based on the pixel signals acquired from the first layer. Further, the imaging device according to an embodiment of the present invention may be an image generation device such as a digital camera or a digital video camera that includes such a stacked image sensor as an imaging element.

As described above, in a stacked image sensor, by configuring the second layer so that it can be stacked on various first layers, the development cost or manufacturing cost of the stacked image sensor can be reduced. To this end, in one embodiment of the invention, the processing parameters of the processing circuit can be controlled depending on the number of pixels or the number of vertical lines provided in the first layer.

FIG. 1 shows an example of a stacked image sensor. The stacked image sensor shown in FIG. 1 includes a first layer 100 and a second layer 110. Here, the first layer 100 is provided with a plurality of pixels that generate pixel signals. In the example of FIG. 1, the pixels are arranged in a Bayer arrangement. Further, it is assumed that the stacked image sensor shown in FIG. 1 has 2K vertical lines. Such a plurality of vertical lines are provided in the first layer 100 and connect the plurality of pixels in the vertical direction of the first layer. Each vertical line can transfer pixel signals from different pixels in the same pixel column in parallel. Pixel signals from different pixels in the same pixel column can be transmitted to the second layer 110 through a plurality of vertical lines and the same number of wires connecting the first and second layers. Thus, pixel signals from multiple pixel rows can be transmitted in parallel from the first layer 100 to the second layer 110 in a first period.

Although a pixel signal output from a Gb pixel will be described below, a similar configuration can be adopted for R, B, and Gr pixels as well. In the following, when pixel signals from pixel rows including Gb pixels are transferred using 2K vertical lines, it is assumed that pixel signals from Gb pixels are transferred using K vertical lines.

The second layer 110 is provided with an ADC section (analog-to-digital converter) 210, a data processing section 220, and a data output section 230. The second layer 110 is provided with four sets of an ADC section 210, a data processing section 220, and a data output section 230, and each set is one of four types of pixels: Gb, Gr, R, and B. Corresponds to crab. Although a set corresponding to Gb will be described below, sets corresponding to other types of pixels can have similar configurations.

The ADC unit 210 A/D converts the pixel signal transferred via the vertical line into a digital signal. The ADC unit 210 can have a read column memory of a predetermined capacity, and can sequentially read pixel data (A/D conversion results) from this memory and send it to the data processing unit 220.

The data processing section 220 receives data from the ADC section 210, performs predetermined data processing on the received data, and sends the processed data to the data output section 230. The data output unit 230 outputs the data received from the data processing unit 220 to the outside of the stacked image sensor.

FIG. 2 is a diagram showing an example of the configuration of the data processing section 220. The black level correction section 250 performs black level correction processing on the pixel data received from the ADC section 210. Next, the gain processing unit 260 performs gain processing on the pixel data after the black level correction processing. Further, the offset processing unit 270 performs offset processing on the pixel data after the gain processing. The data processing unit 220 can perform these processes in parallel on pixel data from a plurality of pixels.

Hereinafter, one horizontal synchronization period (ADC drive unit) will be referred to as 1H. During the 1H period, the ADC section 210 supplies pixel data for K rows. In this case, the data processing unit 220 processes pixel data for K rows in a period of 1H. Therefore, when stacking a common second layer 110 on a first layer 110 having various numbers (for example, number of columns) of pixels, data processing is performed so that pixel data for K rows can be processed in a period of 1H. 220 is configured.

In one embodiment, a common second layer 200 is used even if the first layer 100 has a different configuration (eg, number of pixels or number of vertical lines). For this reason, the data processing unit 220 is configured to be able to process pixel data of different numbers (K) of pixel rows in a period of 1H. In one embodiment, the data processing unit 220 performs data processing based on pixel signals from a first group of pixel rows among a plurality of pixel rows in a first sub-period, and Data processing is performed based on pixel signals from a second pixel row group of the rows. In this way, by repeating data processing for different pixel row groups within 1H, it becomes easy for the data processing unit 220 to respond to changes in the number of vertical lines K. In addition, by having such a configuration, the data can be processed faster than when simultaneously realizing both the number of parallel processing rows corresponding to the largest number of vertical lines K and the number of parallel processing columns corresponding to the largest number of pixels. The circuit scale and power of the processing unit 220 can be reduced.

Although specific embodiments will be described later, a method for setting processing parameters of the data processing unit 220 according to the configuration of the first layer 100 will be outlined here. The 1H period can be divided into S pieces. In this specification, each of the S divided sections is referred to as a small period or a divided period. Further, the length of one divided period is called sH. In this case, the length of sH can be set to H/S.

The data processing unit 220 can process multiple rows of pixel data in parallel in each divided period. For example, the data processing unit 220 can process pixel data for T rows in parallel in one divided period. In this case, the data processing unit 220 can process pixel data for K rows by processing pixel data for T rows S times in a period of 1H (K=T×S).

In such a configuration, the data processing unit 220 completes processing of T rows of pixel data within one divided period. In the following, the throughput of the data output unit 230 is assumed to be T×U (T rows and U columns) pixels per clock cycle, and this is constant regardless of the configuration of the first layer 100 (for example, the number of pixels or the number of vertical lines). . The output throughput of the data output unit 230 is determined according to the specifications, and the value of T×U is set to be less than or equal to the output throughput of the data output unit 230. By setting U to the value obtained by dividing the output throughput of the data output section 230 by T, the processing efficiency of the data processing section 220 is improved. That is, the data processing unit 220 is capable of processing T×U pixels per clock cycle so that the pixel data after data processing can be supplied to the data output unit 230 in accordance with the throughput of the data output unit 230. can. For example, the data processing unit 220 may include T×U arithmetic elements, each of which performs data processing on one pixel data of T×U pixels. Then, the data processing unit 220 processes T rows of pixel data within one divided period by repeating data processing for T×U pixels.

Here, when the number of horizontal pixels (the number of pixel columns) is W, the length of the divided period (H/S) is set to be greater than the number of clock cycles within one divided period (W/U). can do. In this case, the data processing unit 220 can process pixel data for T rows within one divided period.

In this case, the processing parameters of the data processing unit 220 can be set as follows. In this example, the processing parameters define the timing at which the data processing unit 220 processes each of K rows of pixel data within a period of 1H. In particular, in the following example, the above-mentioned division number S or the number of processing lines T per clock cycle (referred to as the number of parallel processing lines) is controlled as a processing parameter. However, the number U of processing columns per clock cycle (referred to as the number of parallel processing columns) may be controlled as a processing parameter.

For example, when the number W of horizontal pixels of the first layer 100 is small and the number K of vertical lines is large, the processing parameters of the data processing unit 220 can be set so that the number S of divisions becomes large. By increasing the number of divisions S, it is possible to accommodate a larger number of vertical lines K even if the values of T and U are constant. In this case, although the length of the division period sH is shortened, since the number W of horizontal pixels is small, data processing for the pixels in column W can be completed in the division period sH. On the other hand, when the number W of horizontal pixels is large and the number K of vertical lines is small, the number S of divisions can be made small.

Further, when the number W of horizontal pixels of the first layer 100 is small and the number K of vertical lines is large, the processing parameters of the data processing unit 220 can be set so that the number T of parallel processing lines becomes large. By increasing the number of parallel processing lines T, even if the number of divisions S is constant, it is possible to handle a larger number of vertical lines K. On the other hand, the computing power of the data processing unit 220 is limited. For example, the data processing unit 220 may have a fixed number of arithmetic elements that each perform data processing on one pixel data of T×U pixels every clock cycle. In such a case, when the number T of parallel processing rows is increased, the number U of parallel processing columns becomes smaller. However, in this case, since the number W of horizontal pixels is small, data processing for pixels in column W can be completed in the division period sH. On the other hand, when the number W of horizontal pixels is large and the number K of vertical lines is small, the number T of parallel processing lines can be reduced.

Note that, depending on the number of pixels or the number of vertical lines, both the number of divisions S and the number of parallel processing lines T may be set, or only one of them may be set. For example, by setting S and T such that the product of the number of divisions S and the number of lines processed per division period T matches the number of vertical lines K, the processing efficiency by the data processing unit 220 can be improved. can. However, the value of T that can be set depends on the specific configuration of the data processing section 220. Therefore, if the value of S is kept constant, there is a possibility that the value of T cannot be set so that the value of S×T matches the number of vertical lines K.

On the other hand, if the value of S increases, the period during which data processing is not performed (for example, the HBLANK period) becomes longer, and the processing efficiency by the data processing unit 220 may decrease. Further, if the value of S is not an integer, there is a possibility that processing using pixel data transferred in the 1H period will not be completed in the next 1H period. The value of S may be limited to an integer if the second layer 110 does not support values of S that are not integers, such as if the second layer 110 has a memory that holds pixel data for a period of 2H.

Furthermore, depending on the processing performed after the data output section 230, the data processing section 220 may be required to simultaneously output a predetermined number of rows of pixel data, and in this case, the value of T may be limited. do not have. Further, the data processing unit 220 may be required to simultaneously process multiple rows of pixel data. For example, when performing processing where there is dependence between input data, the data processing unit 220 can simultaneously perform data processing on a rectangular pixel area of multiple rows and multiple columns, thereby improving the processing efficiency of the data processing unit 220. I can do it. An example of processing in which there is such a dependence between input data is the black level correction processing performed by the black level correction section 250. Even in such cases, the value of T may be limited.

In this way, at least one of the number of divisions S and the number of parallel processing rows T can be set according to the specifications of the second layer 110 or the requirements for the data processing unit 220. By configuring the data processing unit 220 so that both the number of divisions S and the number of parallel processing lines T can be set, it is easier to set the data processing unit 220 according to the number of pixels or the number of vertical lines. becomes.

Note that the value of T does not have to be an integer value. For example, in Embodiment 3, which will be described later, a case where T=2.5 will be described. Further, the lengths of the divided periods do not need to be the same, and the lengths of the divided periods may be different from each other as in the second embodiment. Furthermore, as in the second embodiment, the values of T and U may change for each divided period.

[Embodiment 1]
Hereinafter, an example of a specific method of setting processing parameters of the data processing unit 220 according to the configuration of the first layer 100 will be described. Further, a specific configuration example of the data processing unit 220 that can change processing parameters will also be described. In the first to third embodiments described below, the internal operation of the data processing unit 220 changes in accordance with changes in processing parameters, as described with reference to FIGS. 5, 6, and 8.

In the first embodiment, the data processing unit 220 corresponds to various configurations of the first layer 100 by keeping the values of T and U constant and changing the value of S. FIGS. 5A and 5B show examples of timing charts of processing in the first embodiment. As shown in FIG. 5, in a horizontal synchronization period 510 (first period), pixel signals for K rows are transferred from the first layer 100 to the second layer 110. Furthermore, the horizontal synchronization period 520 (second period) that follows the horizontal synchronization period 510 includes divided periods 521 and 561 (first sub-periods). The data processing unit 220 performs data processing on pixel signals from some pixel row groups among the K pixel rows in the divided periods 521 and 561. Further, the data processing unit 220 performs data processing on pixel signals from one pixel row over one divided period 521 and 561. Here, the length of the division period 1sH can be set according to the number of pixels (for example, the number W of horizontal pixels), as described above. Furthermore, in the horizontal synchronization period 520, the data processing unit 220 repeats data processing based on pixel signals from some pixel row groups among the plurality of pixel rows. Perform data processing. Here, the number of divisions S corresponding to the number of repetitions of data processing can be set according to the number K of vertical lines, as described above.

Below, the case of T=2 and U=4 will be explained. That is, the data processing unit 220 processes pixel signals from a total of eight pixels in a rectangular area of 2 rows and 4 columns per clock cycle. The read column memory of the ADC section 210 stores at least K rows of pixel data corresponding to the pixel signals transferred from the first layer 100 within the 1H period. The data processing unit 220 repeats every clock cycle to read out pixel data of pixels in 2 rows and 4 columns arranged in the upper left from pixel data for K rows and perform data processing. The data processing unit 220 can perform data processing for two rows within one divided period by performing horizontal scanning while shifting four pixels toward the right. Therefore, the length of the divided period 1sH is set so that the number of cycles included in one divided period is greater than the number of horizontal pixels/4. Furthermore, the data processing unit 220 performs data processing for two lines adjacent below in the next divided period. By repeating such data processing for every two rows S times, the data processing unit 220 performs data processing on K rows of pixel data within the horizontal synchronization period 1H. The data processing unit 220 performs data processing on pixel data for K rows adjacent below in the next horizontal synchronization period.

The processing performed by the data processing unit 220 in this embodiment will be described in detail. The data processing unit 220 has a pipeline type arithmetic unit. Pixel data input to the data processing section 220 is outputted to the data output section 230 at any time after a predetermined pipeline delay. As described above, pixel data regarding a plurality of pixels is input to the data processing unit 220 in parallel. Further, the black level correction section 250, the gain processing section 260, and the offset processing section 270 perform data processing on each pixel, and output processed pixel data on each pixel. For example, the gain processing unit 260 performs a process of multiplying the pixel data for each pixel input from the black level correction unit 250 by a predetermined value. Further, the offset processing section 270 performs a process of adding a predetermined value to the pixel data for each pixel input from the gain processing section 260. In this way, the calculations of the gain processing section 260 and the offset processing section 270 are independent for each input (pixel). Therefore, the gain processing section 260 and the offset processing section 270 perform various processes corresponding to the first layer 100 by performing settings such as switching the above-mentioned predetermined values according to the specifications of the first layer 100. be able to.

On the other hand, there is dependence between the input data of the black level correction section 250. An example of the hardware configuration of the black level correction section 250 will be described with reference to FIG. 3. Pixel data for four consecutive pixels in the horizontal direction (for example, pixel data in the first row) is input to the input 300 every clock cycle. Furthermore, pixel data for four consecutive pixels in the horizontal direction (for example, pixel data on the second row) is input to the input 305 every clock cycle. In this way, pixel data for 2 rows and 4 columns is input to the black level correction section 250 in one clock cycle. Pixel data input to input 300 and input 305 are sent to accumulators 310 and 315, respectively, and are accumulated in accumulators 310 and 315. Here, the accumulators 310 and 315 may integrate only pixel data within a predetermined black level acquisition window. For example, the accumulators 310 and 315 may accumulate pixel data for a predetermined number of pixels from the left end of one pixel row. The black level acquisition window is, for example, a region of the first layer 100 where no light enters, and may be an edge region of a region of the first layer 100 where pixels are provided.

The integrated values obtained by accumulators 310 and 315 are input to dividers 340 and 345 via selectors 330 and 335, respectively. The dividers 340 and 345 can calculate the black level by dividing the input integrated value by the number of integrated data. In this way, the average value of pixel data within the black level acquisition window can be calculated as the black level. The subtracters 350 to 353 each subtract the black level from each of the pixel data input to the input 300, thereby generating and outputting pixel data after black level correction. Similarly, the subtracters 355 to 358 each subtract the black level from each of the pixel data input to the input 305, thereby generating and outputting pixel data after black level correction.

In this embodiment, adder 320 is not essential. However, by using the adder 320, the average value of two or more adjacent rows of pixel data within the black level acquisition window can be calculated as the black level. That is, the adder 320 can add the integrated values obtained by the accumulators 310 and 315 and output the sum to the dividers 340 and 345 via the selectors 330 and 335. The dividers 340 and 345 can calculate the black level by dividing the input integrated value by the number of integrated data. For example, each of dividers 340 and 345 can calculate the black level using both pixel data input to input 300 and input 305. Note that the accumulators 310 and 315 may integrate pixel data for each division period, or may integrate pixel data independently of the division period (for example, for all pixels within the black level acquisition window). Good too.

Further, in this embodiment, the selectors 330 and 335 are not essential. On the other hand, the selectors 330 and 335 can input data selected from the data from the accumulators 310 and 315 and the data from the adder 320 to the dividers 340 and 345. According to such a configuration, when calculating the black level, it is possible to select whether to use the integrated value of the accumulator 310 or the total value of the integrated values of the accumulators 310 and 315.

Note that in the example shown in FIG. 3, the arithmetic mean value of pixel data is used as the black level. However, the black level may be calculated using other methods such as filtering pixel data. In this case, the data processing unit 220 is configured such that multiple lines of pixel data (for example, both pixel data input to inputs 300 and 350) are input to a filter calculator for calculating the black level. Good too. According to such a configuration, filter processing that refers to two or more rows of pixel data can be performed. On the other hand, if such a configuration is adopted, the value of T will be set to 2 or more.

Here, consider stacking first layers 100 with different specifications on a common second layer 110. FIG. 4A shows an example of a variation in the specifications of the first layer 100 and the corresponding parameters of the second layer 110. In the following, only Gb pixels will be explained unless otherwise specified. Further, the combination of the first layer specifications and the corresponding second layer parameters is called a set. Two sets are shown in FIG. 4A, set a and set b, respectively. Of course, the first layer 100 may have three or more specifications.

In set a, the number of vertical lines Ka is 4 (8 if you include those other than Gb), and in set b, the number of vertical lines Ka is 5 (10 if you include those other than Gb). On the other hand, in set a, the number of horizontal pixels is 2500, and in set b, the number of horizontal pixels is 2000. Furthermore, as mentioned above, the output throughput of the second layer 110 is constant. Further, as described above, in the first embodiment, the number T of parallel processing rows and the number U of processing columns are constant. That is, in set a, the number of processed lines Ta=2 and the number of processed lines Ua=4, and in set b, the number of processed lines Tb=2 and the number of processed lines Ub=4. Therefore, for set a, the number of divisions Sa is set to 2 (Ka/Ta). Further, for set b, the number of divisions Sb is set to 2.5 (Kb/Tb). The number of divisions Sb being 2.5 means that 2H is divided into five division periods.

FIGS. 5A and 5B are timing charts of processing in this embodiment. In the figure, from the top, rows in which A/D conversion is being performed by the ADC unit 210, rows in which pixel data is stored in the read column memory of the ADC unit 210, and rows in which the data processing unit 220 is processing , and the row where the data output unit 230 is currently outputting data are shown. The numbers in circles indicate line numbers. The horizontal direction is the time axis, which is divided into horizontal synchronization periods of 1H. Further, the horizontal synchronization period 1H is further divided into S divided periods sH. Note that the divided period shH does not need to be divided equally. For example, 1H may be divided into 0.4H and 0.6H.

FIG. 5A is a timing chart of processing for set a. In the first horizontal synchronization period 510, A/D conversion is performed for the first to fourth rows, and the pixel data obtained as the conversion result is transferred to the read column memory. In the next horizontal synchronization period 520, A/D conversion is similarly performed for the 5th to 8th rows. At this time, the pixel data in the 1st to 4th rows stored in the read column memory can be overwritten with the pixel data in the 5th to 8th rows. Furthermore, in the first divided period 521 of the horizontal synchronization period 520, the pixel data of the first to second rows are sequentially input from the read column memory to the data processing section 220. The pixel data of the first and second rows after data processing are output from the data output unit 230 after a predetermined pipeline delay. Thereafter, in the second divided period 522, the pixel data of the third to fourth rows are similarly input to the data processing section 220, and the pixel data after data processing is output. After the horizontal synchronization period 530, the same process is performed while shifting the row numbers by four.

FIG. 5(B) is a timing chart of processing for set b. In the same manner as set a, A/D conversion is performed for the 1st to 5th rows in the first horizontal synchronization period 550, and A/D conversion is performed for the 6th to 10th rows in the next horizontal synchronization period 560, Each is transferred to the read column memory. However, for set b, data processing for 5 rows of pixel data (Kb=5) transferred during the horizontal synchronization period 1H requires 3 divided periods sH, and therefore cannot be completed within the 1H period. do not have. Therefore, in addition to the 6th to 10th rows, the read column memory continues to hold the pixel data of the 5th row. In this configuration, the read column memory has a capacity to store pixel data for at least six rows.

Thus, in one embodiment, the second layer 110 is provided with a memory for storing pixel signals transmitted from the first layer 100. This memory stores at least a portion (for example, the fifth row) of the pixel signals from the plurality of pixel rows (for example, the first to fifth rows). This memory further stores pixel signals from another plurality of pixel rows (rows 6 to 10) transmitted in parallel from the first layer 100 to the second layer 110 during the horizontal synchronization period 520.

In the example of FIG. 5B, in the first divided period 561 of the horizontal synchronization period 560, the pixel data of the first and second rows are sequentially input from the read column memory to the data processing unit 220, and set. Data processing is performed in the same manner as in a. In this example, the first divided period 561 starts after a predetermined wait period (corresponding to approximately 1H/5) from the start of the horizontal synchronization period 560. After that, in each of the divided period 562, divided period 571, divided period 572, and divided period 573, data processing of the 3rd to 4th lines, 5th to 6th lines, 7th to 8th lines, and 9th to 10th lines is performed. It will be done. By providing a wait period and delaying the start of data processing in this way, it is possible to simultaneously perform data processing on the 5th and 6th rows in the divided period 571 in synchronization with the start of the horizontal synchronization period 570.

Thus, in one embodiment, from the end of the horizontal synchronization period 510, at least some of the pixel signals from a plurality of pixel rows (eg, rows 1 to 5) are first input to the data processing unit 220. A wait period is provided until the start of the divided period 561. On the other hand, when at least some of the pixel signals from a plurality of pixel rows (for example, the 6th to 10th rows) are first input to the data processing unit 220 after the end of the horizontal synchronization period 520 (the division period 571 No wait period is required until the start of the process. That is, the time from the end of the horizontal synchronization period 510 to the start of the divided period 561 is different from the time from the end of the horizontal synchronization period 520 to the start of the divided period 571.

In the example of FIG. 5B, pixel signals from multiple pixel rows are input to the data processing unit 220 alone (for example, the first and second rows), or pixels from another multiple pixel rows are input to the data processing unit 220. It is switched whether the signal is input to the data processing unit 220 together with the signal (for example, in the 5th and 6th lines). Thus, in one embodiment, in addition to at least some of the pixel signals from the plurality of pixel rows, at least some of the pixel signals from another plurality of pixel rows are simultaneously applied to the processing circuit. It is possible to control whether to input or not.

According to these configurations, even when the number of divisions S is not an integer value as in set b, the waiting time of the data processing unit 220 can be reduced and efficient processing can be realized.

Note that in the above example, the capacity of the read column memory included in the ADC unit 210 is increased, but a buffer column memory may be provided at another location. For example, a column memory may be added between the data processing section 220 and the data output section 230. This column memory may be configured to temporarily hold data (for example, during a wait period) and wait for the fifth and sixth rows.

As described above, according to the present embodiment, by setting the division number S, which is one of the control parameters of the second layer 110, the second layer 110 can correspond to the first layer 100 having multiple specifications. . In particular, according to this embodiment, data processing for different pixel row groups is repeated within 1H. For this reason, for example, when T=2, the number of vertical lines is 4 (S=2), 6 (S=3), and 8 (S=4), the second layer 110 is possible. Therefore, since a common second layer 110 can be used for a plurality of types of first layers 100, it is expected that the development cost or manufacturing cost of the imaging device can be reduced.

Furthermore, by further adopting the configuration described with reference to FIG. 5(B), the second layer 110 can handle cases where the number of divisions S is not an integer value, and achieves efficient processing. be able to. For example, when T = 2, the number of vertical lines is 3 (S = 1.5), 5 (S = 2.5), and 7 (S = 35), the second layer 110 is applicable. According to such a configuration, the common second layer 110 can be used for more types of first layers 100, so it is expected that the development cost or manufacturing cost of the imaging device can be further reduced. Ru.

[Embodiment 2]
In the second embodiment, the data processing unit 220 corresponds to various configurations of the first layer 100 by changing both the value of S and the value of T. Also in the second embodiment, the data processing unit 220 corresponds to both set a and set b shown in FIG. 4(A). The configurations of the imaging device and the data processing unit 220 according to the second embodiment are the same as those of the first embodiment, and the description thereof will be omitted. Further, the processing for set a is the same as in the first embodiment, and is performed as shown in FIG. 5(A). The processing for set b will be described below with reference to the timing chart of FIG.

As shown in FIG. 6, in a horizontal synchronization period 650 (first period), pixel signals from a plurality of pixel rows (for example, 1st to 5th rows) are transferred from the first layer 100 to the second layer 110. Ru. Then, in the divided period 662 (first sub-period), data processing is performed based on pixel signals from the first pixel row group (for example, the third and fourth rows) among the plurality of pixel rows. Furthermore, in the divided period 663 (second sub-period), data processing is performed based on pixel signals from a second pixel row group (for example, the fifth row) among the plurality of pixel rows. Here, the number of pixel rows included in the first pixel row group and the number of pixel rows included in the second pixel row group are different.

To explain in more detail, in this embodiment, in order to correspond to the number of divisions S=2.5, the 1H period is divided into 2.5 division periods, that is, two first division periods and two first division periods. The second divided period is divided into one, which is shorter than the divided period. In the first divided period, T=2 and U=4 are used, and in the second divided period, T=1 and U=8 are used. In the example of FIG. 6, in the first divided periods 661 and 662, data processing can be performed in the same manner as in the first embodiment, except that no wait period is required. On the other hand, in the second division periods 663 and 673, the pixel data in the 5th and 10th rows are processed every 1 clock cycle by 1 row and 8 columns (equivalent to normal double speed). Therefore, the data processing for the 5th and 10th lines is completed in the horizontal synchronization periods 660 and 670, respectively.

Such a configuration can be realized by using the data processing section 220 having a first arithmetic unit and a second arithmetic unit. For example, in FIG. 3, the data processing unit 220 includes a first arithmetic unit (subtractors 350 to 353 and a divider 340) and a second arithmetic unit (subtractors 355 to 358 and a divider 345). It is shown. Here, in the first division period 662, each of the first arithmetic unit and the second arithmetic unit has a different pixel row (for example, the third row or the fourth row) included in the first pixel row group. A pixel signal from is input. The processing in this case is the same as in the first embodiment.

On the other hand, in the second divided period 663, pixel signals from the same pixel row (for example, the fifth row) included in the second pixel row group are input to the first arithmetic unit and the second arithmetic unit. Ru. In other words, half of the pixel data in the 1st row and 8th column (pixel data in the 1st row and 4th column) is input to the input 300, and the remaining half (pixel data in the 1st row and 4th column) is input to the input 305. At this time, since each of the accumulators 310 and 315 integrates a part of the pixel data in the fifth row, the sum of the integrated values by the accumulators 310 and 315 becomes the integrated value of the pixel data in the fifth row. Therefore, selectors 330 and 335 select the output from adder 320 indicating the sum of the integrated values from accumulators 310 and 315, and supply it to dividers 340 and 345. Note that in this embodiment, the arrangement of data output from the data output unit 230 changes for each row. Therefore, on the data receiving side, the memory location for storing the received data can be adaptively changed or the data can be rearranged.

As shown in FIG. 6, in the second divided period, high-speed data processing is performed on pixel data with a smaller number of rows than in the first divided period. That is, over a period of a predetermined length within the first divided period 662, the data processing unit 220 processes the first number of one pixel row (for example, the third row) included in the first pixel row group. Data processing is performed on pixel signals from pixels. Furthermore, over a period of the same predetermined length within the second divided period, the data processing unit 220 calculates the first number of one pixel row (for example, the fifth row) included in the second pixel row group. Data processing is performed on pixel signals from a second larger number of pixels.

Also in this embodiment, by setting the number of divisions S and the number of parallel processing lines T, which are one of the control parameters of the second layer 110, the second layer 110 can correspond to the first layer 100 having multiple specifications. . In the first embodiment, the read column memory of the ADC section 210 can store at least six rows of pixel data, but in this embodiment, the read column memory of the ADC section 210 can store five rows of pixel data. A memory capable of storing data can be used. In this way, by further adopting the configuration explained with reference to FIG. 6, even if the number of divisions S is not an integer value, efficient processing can be achieved while suppressing an increase in circuit scale. can do.

[Embodiment 3]
In the third embodiment, the data processing unit 220 is compatible with various configurations of the first layer 100 by changing the value of T while fixing the value of S. FIGS. 5A and 8 show timing charts of processing in the third embodiment. As shown in FIG. 5(A), the horizontal synchronization period 520, 860 (second period) that follows the horizontal synchronization period 510, 850 includes a divided period 521, 861 (first sub-period). . The data processing unit 220 performs data processing on pixel signals from some pixel row groups (corresponding to the value of T) among the K pixel rows in parallel over the division periods 521 and 861. Here, the number of parallel processing lines T can be changed according to the number of pixels or the number of vertical lines, as described above.

In the following, a case will be described in which the data processing unit 220 corresponds to both set a and set b shown in FIG. 4(B) (unless otherwise specified, only Gb pixels will be described). The configuration of the imaging device according to Embodiment 3 is similar to Embodiment 1, and different parts will be described below. For example, as the imaging device according to the third embodiment, the imaging device shown in FIG. 1 can be used, and as the circuit configuration of the second layer 110, the configuration shown in FIG. 2 can be adopted.

The processing for set a is the same as in the first embodiment, and the data processing unit 220 can perform the processing as shown in FIG. 5(A). However, in this example, as shown in FIG. 4B, the output throughput of the second layer 110 is 10, and the data processing unit 220 processes a total of 10 pixels in 2 rows and 5 columns every clock cycle. Process pixel data.

FIG. 8 shows a timing chart of processing for set b. As shown in FIG. 6, data processing based on pixel signals from the first pixel row group (for example, the first and second rows) among the plurality of pixel rows is performed over a divided period 861 (first sub-period). It is being done. Further, data processing based on pixel signals from a second pixel row group (for example, the third row) among the plurality of pixel rows is performed over the divided periods 861 to 862 (second sub-period).

More specifically, in the division period 861, the data processing unit 220 processes four pixels in the first row, four pixels in the second row, and two pixels in the first half of the third row per clock cycle. Data processing is performed for a total of 10 pixels. In this way, the data processing unit 220 processes one pixel row (for example, the first row) included in the first pixel row group over a predetermined length period within the divided period 861 (first sub-period). Data processing can be performed on pixel signals from a third number of pixels (for example, four pixels). On the other hand, the data processing unit 220 calculates, over a period of the same predetermined length within the divided period 861, a number smaller than the third of one pixel row (for example, the third row) included in the second pixel row group. Data processing can be performed on pixel signals from a fourth number of pixels (for example, two pixels).

In this way, in the example of FIG. 8, the processing for the third row progresses more slowly than the processing for the first and second rows. Therefore, the processing for the third row is not completed during the divided period 861, and the data processing unit 220 continues the processing for the third row during the divided period 862. In the division period 862, the data processing unit 220 performs data processing on a total of 10 pixels, 2 pixels in the second half of the 3rd row, 4 pixels in the 4th row, and 4 pixels in the 5th row, per clock cycle. conduct. The data output unit 230 outputs pixel data after data processing after a predetermined pipeline delay in synchronization with the data processing unit 220.

As described above, the gain processing section 260 and the offset processing section 270 can perform independent processing on each of the input pixel data of 10 pixels using, for example, 10 same arithmetic units. . Therefore, the gain processing section 260 and the offset processing section 270 can perform the same processing on the input pixel data for both set a and set b. On the other hand, the black level correction unit 250 performs data processing in which dependence exists between data. To this end, in this embodiment, the black level correction unit 250 operates in a first mode for set a and a second mode for set b, depending on the number of pixels or the number of vertical lines. It is possible to switch between them.

FIG. 7 shows an example of the configuration of the black level correction section 250 in this embodiment. Pixel data for a total of 10 pixels is input to inputs 900, 901, 905, and 906 every clock cycle. In the case of set a, pixel data of the 1st row and 5th column (for example, the 1st row or 3rd row) is input to the inputs 900 and 901, and the pixel data of the 1st row and 5th column (for example, the 2nd row or the 3rd row) is input to the inputs 905 and 906. The pixel data of the fourth line) is input. On the other hand, in the case of set b, the input 900 has pixel data in the 1st row and 4th column (for example, the 1st row or the 4th row), and the input 905 has the pixel data in the 1st row and 4th column (for example, the 2nd row or the 5th row). Pixel data of the first row and second column (for example, the third row) is input to inputs 901 and 906, respectively.

In this way, the data processing unit 220 includes a first arithmetic unit (for example, subtracters 950 to 953) and a second arithmetic unit (for example, subtractor 954). In the case of set a, pixel signals from the same pixel row (for example, the first row) among the plurality of pixel rows are input to the first arithmetic unit and the second arithmetic unit. In the case of set b, each of the first arithmetic unit and the second arithmetic unit receives pixel signals from mutually different pixel rows (for example, the first and third rows) among the plurality of pixel rows. is input. According to such a configuration, the data processing unit 220 can change the number T of rows to be processed in parallel. Note that in the example of FIG. 8, in the case of set b, the first arithmetic unit performs data processing on pixel signals from a pixel row (for example, the first row) that is processed more quickly. On the other hand, the second arithmetic unit performs data processing on pixel signals from a pixel row (for example, the third row) that is processed more slowly.

More specifically, in the case of set a, the pixel data input to inputs 900 and 905 are sent to accumulators 910 and 915, respectively, and accumulated. Furthermore, the pixel data input to the input 901 is sent to the accumulator 910 by the switch 960 and is integrated. Furthermore, the pixel data input to the input 906 is sent to the accumulator 915 by the switch 965 and is integrated. In this case, 0 is input to accumulator 919.

Adder 920 can select whether to add up the three integrated values. Selectors 930, 935, and 939 can output values selected from the integrated values output from accumulators 910, 915, and 919, and the output value of adder 920. The roles of the adder 920 and selectors 930, 935, and 939 are the same as those of the adder 320 and selectors 330 and 335 in the first and second embodiments, so the explanation will be omitted. Further, as in the first embodiment, the adder 920 and selectors 930, 935, and 939 can be omitted.

Dividers 940 and 945 calculate the black level using the integrated values output from accumulators 910 and 915, similarly to dividers 340 and 345 in the first and second embodiments. The subtracters 950 to 959 correct the black level by subtracting the black level from the pixel data input to the inputs 900, 901, 905, and 906, similarly to the subtracters 350 to 353 and 355 to 358 of the first embodiment. Generate and output subsequent pixel data. In the case of set a, the black level output from dividers 940 and 945 is subtracted from the pixel data input to inputs 901 and 906, respectively. Therefore, selector 970 selects the output of divider 940 and supplies it to subtracter 954, and selector 975 selects the output of divider 945 and supplies it to subtracter 959.

On the other hand, in the case of set b, the pixel data input to inputs 900 and 905 are sent to accumulators 910 and 915, respectively, and accumulated. Furthermore, the pixel data input to the input 901 is sent to the accumulator 919 by the switch 960 and is integrated. Furthermore, the pixel data input to the input 906 is sent to the accumulator 919 by the switch 965 and is integrated.

In this case, dividers 940, 945, and 949 calculate the black level using the integrated values output from accumulators 910, 915, and 919, respectively. Calculation of the black level can be performed in the same way as for set a, but the divisor used to calculate the arithmetic mean value varies depending on the set or the number of pixel data accumulated in each accumulator. ,be changed.

Further, subtracters 950 to 959 subtract the black level from the pixel data input to inputs 900, 901, 905, and 906. In the case of set b, the black level output from the divider 949 is subtracted from the pixel data input to each of the inputs 901 and 906. Therefore, selectors 970 and 975 select the output of divider 949 and supply it to subtracters 954 and 959.

With the above configuration, the black level correction unit 250 can obtain the black level for each row and perform black level correction processing in both set a and set b.

As described above, in this embodiment, by setting the number of parallel processing lines T, which is one of the control parameters of the second layer 110, the second layer 110 can correspond to the first layer 100 having multiple specifications. . Further, such a configuration can be realized by adding an accumulator 919, a divider 949, and some selectors shown in FIG. 7, for example, and achieves efficient processing while suppressing an increase in circuit scale. can do.

[Other embodiments]
In the first embodiment, set a (Sa=2, Ta=2) and set b (Sb=2.5, Tb=2) were switched. Further, in the second embodiment, in order to realize Sb=2.5, Tb=1 and Tb=2 were switched during the processing of set b. Furthermore, in the third embodiment, set a (Sa=2, Ta=5) and set b (Sb=2.5, Tb=4) were switched. However, these parameters such as S and T may have other values.

For example, as in the third embodiment, the data processing unit 220 can handle any number of parallel processing rows T (Ta<Tb), number of divisions S (Sa=Sb), and number of parallel processing columns U (Ua>Ub). can be realized as follows. Here, Ka=Sa×Ta and Kb=Sb×Tb. That is, in addition to an arithmetic unit that can process Ta rows in each divided period (equivalent to 910, 915, 940, 945), an arithmetic unit that can process (Tb-Ta) rows in 1H (equivalent to 919, 949). A data processing unit 220 having the following can be used. Here, the arithmetic unit that can process row Ta for each division period refers to an arithmetic unit that can supply pixel data of row Ta and column Ua to the data output unit 230 every clock cycle.

Further, the number of sets may be three or more. For example, the data processing unit 220 that can handle three types of parallel processing row numbers T (Ta<Tb<Tc) can be realized as follows. That is, as described above, the data processing unit 220 can be provided with an arithmetic unit that can process rows Ta in each divided period and an arithmetic unit that can process rows (Tb-Ta) in 1H. The data processing unit 220 can further include an arithmetic unit that can process (Tc-Tb) rows that need to be further processed in 1H. By such a recursive method, it is possible to prepare a data processing unit 220 that can handle three or more sets.

In each of the above embodiments, a case has been described in which black level correction processing is performed according to various processing parameters. However, the type of data processing is not limited to black level correction processing, and the method of the above embodiment can also be applied to, for example, spatial filter processing such as low-pass filter processing, resolution conversion processing, or compression processing. .

Note that the image sensor according to each of the above embodiments can be manufactured as follows. First, the number of pixels that are provided in the first layer 100 and that generate pixel signals is determined depending on the purpose of the image sensor. Furthermore, a plurality of vertical lines that are provided in the first layer 100 and connect a plurality of pixels in the vertical direction of the first layer 100 are determined depending on the purpose of the image sensor. The first layer has common hardware regardless of the determined number of pixels and the number of vertical lines, and is provided with a processing circuit that performs data processing based on the pixel signals acquired from the first layer 100. A second layer 110 stacked on top of the second layer 100 is prepared. Then, the first layer 100 provided with the determined number of pixels and vertical lines is stacked on the second layer 110 . By using a common second layer in this way, the development cost or manufacturing cost of the image sensor can be reduced.

On the other hand, the processing parameters of the second layer 110 are set according to the determined number of pixels and the number of vertical lines so as to be able to accommodate various numbers of pixels and vertical lines. Such processing parameters can be written in, for example, a memory included in the second layer 110. Furthermore, the operation of the selector shown in FIG. 7, for example, may be fixed according to such processing parameters.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

100: first layer, 110: second layer, 210: ADC section, 220: data processing section, 230: data output section, 250: black level correction section

Claims (16)

a plurality of pixels provided in the first layer and generating pixel signals;
a plurality of vertical lines provided in the first layer and connecting the plurality of pixels in the vertical direction of the first layer;
Data processing based on pixel signals obtained from the first layer, which is provided in a second layer stacked on the first layer, and whose processing parameters can be controlled according to the number of pixels or the number of vertical lines. A processing circuit that performs
An imaging device comprising: pixel signals from a plurality of pixel rows are transmitted in parallel from the first layer to the second layer in a first period. The processing circuit performs data processing on a pixel signal from one pixel row of the plurality of pixel rows over a first small period included in a second period following the first period,
The imaging device according to claim 1, wherein the length of the first short period can be changed depending on the number of pixels. The processing circuit repeats data processing based on pixel signals from some of the pixel row groups among the plurality of pixel rows in a second period following the first period. Performs data processing based on image signals from pixel rows,
The imaging device according to claim 1, wherein the number of times the data processing is repeated in the second period can be changed depending on the number of vertical lines. The processing circuit performs data processing on pixel signals from some pixel row groups among the plurality of pixel rows over a first small period included in a second period following the first period. done in parallel,
2. The imaging device according to claim 1, wherein the number of pixel rows in the partial pixel row group can be changed according to the number of pixels or the number of vertical lines. The operation of the imaging device is switchable between a first mode and a second mode depending on the number of pixels or the number of vertical lines,
The processing circuit has a first arithmetic unit and a second arithmetic unit,
In the first mode, pixel signals from the same pixel row among the plurality of pixel rows are input to the first arithmetic unit and the second arithmetic unit,
In the second mode, pixel signals from mutually different pixel rows among the plurality of pixel rows are input to each of the first arithmetic unit and the second arithmetic unit. , The imaging device according to claim 1. a plurality of pixels provided in the first layer and generating pixel signals;
An imaging device comprising: a plurality of vertical lines provided in the first layer and connecting the plurality of pixels in the vertical direction of the first layer;
In a first period, pixel signals from a plurality of pixel rows are transmitted in parallel from the first layer to a second layer stacked on the first layer,
The imaging device further includes a processing circuit provided in the second layer, which performs data processing based on pixel signals from a first pixel row group of the plurality of pixel rows in a first sub-period. , a processing circuit that performs data processing based on pixel signals from a second group of pixel rows among the plurality of pixel rows in a second sub-period. further comprising a memory provided in the second layer and storing pixel signals transmitted from the first layer,
The memory is configured to transmit at least some of the pixel signals from the plurality of pixel rows from the first layer to the second layer in parallel in a second period following the first period. 7. The imaging device according to claim 6, wherein pixel signals from another plurality of pixel rows are simultaneously stored. the time from the end of the first period until at least some of the pixel signals from the plurality of pixel rows are first input to the processing circuit; 8. The imaging device according to claim 7, wherein the time until at least some of the pixel signals from the plurality of pixel rows are first input to the processing circuit is different. Controlling whether at least some of the pixel signals from the plurality of pixel rows and at least some of the pixel signals from the other plurality of pixel rows are simultaneously input to the processing circuit. The imaging device according to claim 7 or 8, characterized in that it is possible. The imaging device according to claim 6, wherein the number of pixel rows included in the first pixel row group and the number of pixel rows included in the second pixel row group are different. The processing circuit has a first arithmetic unit and a second arithmetic unit,
In the first short period, pixel signals from different pixel rows included in the first pixel row group are input to each of the first arithmetic unit and the second arithmetic unit,
In the second short period, pixel signals from the same pixel row included in the second pixel row group are input to the first arithmetic unit and the second arithmetic unit. , The imaging device according to claim 10. Over a period of a predetermined length within the first sub-period, the processing circuit processes pixel signals from a first number of pixels of one pixel row included in the first pixel row group. perform data processing,
Over the period of the predetermined length within the second sub-period, the processing circuit controls a second pixel row that is greater than the first number of pixel rows included in the second pixel row group. The imaging device according to claim 10 or 11, wherein data processing is performed on pixel signals from several pixels. The processing circuit includes:
performing data processing on pixel signals from one pixel row included in the first pixel row group over the first short period;
7. The method according to claim 6, wherein data processing is performed on a pixel signal from one pixel row included in the second pixel row group over the second short period that is longer than the first short period. The imaging device described. The processing circuit has a first arithmetic unit and a second arithmetic unit,
The first arithmetic unit performs data processing on pixel signals from pixel rows included in the first pixel row group,
The imaging device according to claim 13, wherein the second arithmetic unit performs data processing on pixel signals from pixel rows included in the second pixel row group. Over a period of a predetermined length within the first sub-period, the processing circuit processes pixel signals from a third number of pixels of one pixel row included in the first pixel row group. perform data processing,
Over the period of the predetermined length within the second sub-period, the processing circuit controls a fourth pixel row that is less than the third number of pixel rows included in the second pixel row group. The imaging device according to claim 13 or 14, wherein data processing is performed on pixel signals from several pixels. A method for manufacturing an imaging device according to any one of claims 2 to 15, comprising:
a step of determining the number of pixels that generate pixel signals provided in the first layer;
determining the number of vertical lines provided in the first layer and connecting the plurality of pixels in the vertical direction of the first layer;
The first layer has common hardware regardless of the determined number of pixels and the number of vertical lines, and is provided with a processing circuit that performs data processing based on pixel signals acquired from the first layer. a step of preparing a second layer laminated on the
setting processing parameters for the second layer according to the determined number of pixels and the number of vertical lines;
laminating the first layer provided with the determined number of pixels and the vertical line on the second layer;
A manufacturing method characterized by having the following.

Images