WO2021090663A1

WO2021090663A1 - Image capture device and method for correcting pixel signal in image capture device

Info

Publication number: WO2021090663A1
Application number: PCT/JP2020/039150
Authority: WO
Inventors: 進大木
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2019-11-05
Filing date: 2020-10-16
Publication date: 2021-05-14
Also published as: TW202123683A

Abstract

The purpose of the present invention is to ensure the linearity of pixel output characteristics without reducing charge transfer efficiency. The present invention provides an image capture device comprising a plurality of pixels capable of accumulating charges generated in accordance with the amount of incident light. The plurality of pixels are arrayed so as to include a first pixel formed so as to have a first saturation charge amount, and a second pixel formed so as to have a second saturation charge amount smaller than the first saturation charge amount.

Description

Pixel signal correction processing method in imaging device and imaging device

The present technology relates to an imaging device and a pixel signal correction processing method in the imaging device.

In recent years, the density and miniaturization of pixels of imaging devices using CCD (Charge Coupled Device) and CMOS (Complementary MOS) have been increasing. Correspondingly, it is required to improve the saturated charge amount per unit area of the pixel so that the characteristics of the pixel do not deteriorate. Generally, in order to improve the saturated charge amount of a pixel, the potential well is deepened so that more charge is accumulated in the charge storage region of the pixel.

However, if the potential well is deepened, problems such as blooming in which the electric charge overflows to the surrounding pixels will occur, and the electric charge transfer efficiency will decrease. That is, since the charge transfer efficiency decreases due to the deepening of the potential well, there is a trade-off relationship between the saturated charge amount and the charge transfer efficiency. The decrease in transfer efficiency deteriorates the linearity between the magnitude of the light energy incident on the pixel and the amount of charge output by the pixel, especially in the low light region. Due to the deterioration of linearity, the white balance in the low illuminance region is lost, and so-called coloring or the like in which a specific color component (for example, a green component) floats occurs. Therefore, some techniques for preventing the deterioration of image quality due to such deterioration of linearity have been proposed.

For example, Patent Document 1 below discloses a technique for reducing deterioration of color reproducibility due to incomplete transfer, particularly deterioration of color reproducibility at high sensitivity. Specifically, Patent Document 1 describes, for each settable shooting sensitivity, for each color generated by using output signals of different colors included in the imaging signal output from the imaging means corresponding to different exposure amounts. The color ratio correction coefficient is stored in the storage means, and the correction means for correcting the image pickup signal output from the image pickup means is controlled by using the stored color ratio correction coefficient to correspond to the shooting sensitivity set in the shooting of the subject. The color ratio correction coefficient is acquired from the storage means, and the output signals of different colors included in the image pickup signal of the subject output from the image pickup means are obtained, and the color ratio correction coefficient of the corresponding color among the acquired color ratio correction coefficients is obtained. The imaging apparatus to be used and corrected is disclosed.

Further, Patent Document 2 below discloses a technique for improving the accuracy of a pixel signal output from an image pickup device included in an image pickup device. Specifically, Patent Document 2 provides incomplete transfer charge amount information according to the incomplete transfer charge amount remaining in the charge storage unit at the time of charge transfer from the charge storage unit to the charge holding unit for each of the plurality of pixels. , A storage means that stores in advance for each pixel corresponding to the charge storage unit, and a correction means that corrects the pixel signal output by the signal output unit based on the incompletely transferred charge amount information stored by the storage means. Disclose a device comprising.

Japanese Unexamined Patent Publication No. 2013-150051 Japanese Unexamined Patent Publication No. 2012-231421

A technique as disclosed in the above document acquires a correction coefficient or data for each pixel in advance and stores it in a storage means in order to correct the deterioration of the linearity of pixel characteristics due to incomplete transfer of electric charge or deterioration of transfer efficiency. I had to do it. Further, there is a problem that the amount of correction data acquired and stored in advance becomes enormous in order to cope with changes in the operating temperature and the exposure time.

Therefore, an object of the present technology is to provide an imaging device and a pixel signal correction processing method in the imaging device that can secure the linearity of the pixel characteristics without lowering the charge transfer efficiency.

The present technology for solving the above problems is configured to include the following invention-specific matters or technical features.

The present technology according to a certain viewpoint is an imaging device including a plurality of pixels capable of accumulating electric charges generated according to the amount of incident light. In the imaging device, the plurality of pixels have a first pixel formed so as to have a first saturated charge amount, and a second saturated charge amount smaller than the first saturated charge amount. Includes a second pixel formed.

Further, this technology according to another viewpoint is a pixel signal correction processing method in an imaging device. The correction processing method includes reading out the electric charge from a plurality of pixels accumulating electric charges generated according to the amount of incident light, and correcting a signal based on the amount of the electric charge read out from the plurality of pixels. ,including. Here, the plurality of pixels are formed so as to have a first pixel formed to have a first saturated charge amount and a second saturated charge amount smaller than the first saturated charge amount. Also includes a second pixel. Further, the correction is any of the plurality of pixels based on the sensitivity ratio based on the amount of electric charge transferred from the first pixel and the sensitivity ratio based on the amount of electric charge transferred from the second pixel. Corrects the signal based on the amount of charge transferred from.

Note that, in the present specification and the like, the means does not simply mean a physical means, but also includes a case where the function of the means is realized by software. Further, the function of one means may be realized by two or more physical means, or the function of two or more means may be realized by one physical means.

Further, the "system" means a logical assembly of a plurality of devices (or functional modules that realize a specific function), and whether or not each device or functional module is in a single housing. Is not particularly limited.

Other technical features, objectives, effects or advantages of the present technology will be clarified by the following embodiments described with reference to the attached drawings. Further, the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

It is a figure which shows an example of the structure of the image pickup device which concerns on one Embodiment of this technique. It is a figure explaining an example of the arrangement of the pixel in the pixel array part which concerns on one Embodiment of this technique. It is a figure explaining an example of the arrangement of the pixel in the pixel array part which concerns on one Embodiment of this technique. It is a figure explaining an example of the arrangement of the pixel in the pixel array part which concerns on one Embodiment of this technique. It is a partial vertical sectional view which shows an example of the schematic structure of the pixel in the pixel array part which concerns on one Embodiment of this technique. It is a partial vertical sectional view which shows an example of the schematic structure of the pixel in the pixel array part which concerns on one Embodiment of this technique. It is a partial vertical sectional view which shows an example of the schematic structure of the pixel in the pixel array part which concerns on one Embodiment of this technique. It is a partial cross-sectional plan view for demonstrating the electrode of a pixel in the pixel array part which concerns on one Embodiment of this technique. It is a flowchart for demonstrating an example of the pixel signal correction processing in the image pickup apparatus which concerns on one Embodiment of this technique. It is a figure explaining an example of the pixel signal correction processing in the image pickup apparatus which concerns on one Embodiment of this technique. It is a flowchart for demonstrating an example of the pixel signal correction processing in the image pickup apparatus which concerns on one Embodiment of this technique. It is a figure explaining an example of the pixel signal correction processing in the image pickup apparatus which concerns on one Embodiment of this technique. It is a block diagram which shows an example of the structure of the image pickup apparatus as an electronic device to which the technique concerning this disclosure is applied. It is a block diagram which shows the schematic structure example of the vehicle control system which is an example of the mobile body control system to which the technique which concerns on this disclosure can be applied. It is a figure which shows the example of the installation position of the imaging unit. It is a figure which shows an example of the schematic structure of the endoscopic surgery system to which the technique (the present technique) which concerns on this disclosure can be applied. It is a block diagram which shows an example of the functional structure of the camera head and CCU shown in FIG.

Hereinafter, embodiments of the present technology will be described with reference to the drawings. However, the embodiments described below are merely examples, and there is no intention of excluding the application of various modifications and techniques not specified below. The present technology can be implemented with various modifications (for example, combining each embodiment) within a range that does not deviate from the purpose. Further, in the description of the following drawings, the same or similar parts are represented by the same or similar reference numerals. The drawings are schematic and do not necessarily match the actual dimensions and ratios. Even between drawings, there may be parts where the relationship and ratio of dimensions are different from each other.

[First Embodiment]
FIG. 1 is a diagram showing an example of a configuration of an imaging device according to an embodiment of the present technology. As shown in the figure, the image pickup device 1 includes, for example, a control unit 10, a pixel array unit 20, a vertical drive unit 30, a horizontal drive unit 40, and a column processing unit 50. Further, the image pickup device 1 may include a signal processing unit 60 and an image memory 70. The imaging device 1 can be configured, for example, as a system on chip (SOC), but is not limited to this.

The control unit 10 is a circuit that comprehensively controls the image pickup device 1. The control unit 10 may include a timing generator (not shown) that generates various timing signals. The control unit 10 controls the operations of the vertical drive unit 30, the horizontal drive unit 40, and the column processing unit 50 according to various timing signals generated by the timing generator based on, for example, a clock signal supplied from the outside.

The pixel array unit 20 includes a group of photoelectric conversion elements arranged in an array that generates and stores electric charges according to the intensity of incident light. The embedded photodiode is an aspect of the photoelectric conversion element 222 (see FIG. 4A). Each or some of the plurality of photoelectric conversion elements 222 may constitute one pixel. Each pixel typically includes a color filter (see FIG. 4A) and is configured to receive light of a color component corresponding to the color filter. As the arrangement of pixels, for example, a quad arrangement and a Bayer arrangement are known, but the arrangement is not limited to this. In the figure, the vertical direction of the pixel array unit 20 is referred to as a column direction or a vertical direction, and the left-right direction is defined as a row direction or a horizontal direction. The details of the pixel configuration in the pixel array unit 20 will be described later.

The vertical drive unit 30 includes a shift register, an address decoder (not shown), and the like. Under the control of the control unit 10, the vertical drive unit 30 drives, for example, a group of pixels of the pixel array unit 20 in a row-by-row manner in the vertical direction. In the present disclosure, the vertical drive unit 30 sweeps out (reset) unnecessary charges from the read-out scanning circuit 32 and the photoelectric conversion element 222 (see FIG. 4A) that scan for reading signals, and the sweep-out scanning circuit 34. Can include.

The read-out scanning circuit 32 sequentially selects and scans the pixel groups of the pixel array unit 20 row by row in order to read out a signal based on the electric charge from each pixel.

The sweep scanning circuit 34 performs sweep scanning for the read row for which the read operation is performed by the read scan circuit 32, ahead of the read operation by the time of the operating speed of the electronic shutter. The so-called electronic shutter operation is performed by sweeping (resetting) unnecessary charges by the sweep scanning circuit 34. The electronic shutter operation refers to an operation of sweeping out the electric charge of the photoelectric conversion element 222 and newly starting exposure (accumulation of electric charge).

The signal based on the electric charge read by the read operation by the read scan circuit 32 corresponds to the magnitude of the light energy incidented after the read operation immediately before or the electronic shutter operation. Then, the period from the read timing by the immediately preceding read operation or the sweep operation timing by the electronic shutter operation to the read timing by the current read operation is the charge accumulation time in the pixel.

The horizontal drive unit 40 includes a shift register, an address decoder (not shown), and the like. Under the control of the control unit 10, the horizontal drive unit 40 drives, for example, a group of pixels of the pixel array unit 20 in a row-by-column manner in the horizontal direction. By selectively driving the pixels by the vertical drive unit 30 and the horizontal drive unit 40, a signal based on the electric charge accumulated in the selected pixels is output to the column processing unit 50.

The column processing unit 50 performs predetermined processing such as CDS (Correlated Double Sampling) processing on the signals output from each of the pixel groups in the selected row of the pixel array unit 20. Specifically, the column processing unit 50 receives the difference signals output from each of the pixel groups in the selected row, and obtains the level (potential) difference indicated by the difference signals for each row of pixels. Get the signal. Further, the column processing unit 50 can remove fixed pattern noise from the acquired signal. The column processing unit 50 converts the signal subjected to such predetermined processing into a digital signal by the A / D conversion unit (not shown), and outputs this as a pixel signal.

The signal processing unit 60 is a circuit that has at least an arithmetic processing function and performs various signal processing such as arithmetic processing on the pixel signal output from the column processing unit 50. A digital signal processor (DSP) is an aspect of the signal processing unit 60. The image memory 70 temporarily stores data necessary for signal processing by the signal processing unit 60. As will be described later, the signal processing unit 60 of the present disclosure performs processing for correcting normal pixels and / or correction pixels based on pixel signals from normal pixels and / or correction pixels. The signal processing unit 60 may be configured to perform all or part of the above-mentioned arithmetic processing in the column processing unit 50.

FIG. 2 is a diagram for explaining an example of the pixel arrangement of the pixel array unit 20 according to the embodiment of the present technology. In the pixel array unit 20, each of the plurality of pixels includes a color filter, and the pixels of each color component corresponding to the color filter are arranged according to a predetermined arrangement pattern. Color filters include, but are not limited to, for example, three types of filters, red, green and blue. In the present disclosure, some of the plurality of pixels are formed to function as correction pixels. In the figure, the pixels indicated by hatching are shown as correction pixels. As will be described later, the correction pixel is a pixel formed so as to secure a charge transfer ability with a smaller amount of saturated charge as compared with a normal pixel. By performing the correction process using the signal based on such a correction pixel, it is possible to prevent the spatial resolution from being lowered in the low illuminance region.

2 (a) and 2 (b) show an example of the pixel arrangement of each color component according to the Bayer arrangement. In the Bayer arrangement, the red pixel R, the green pixel G, and the blue pixel B are arranged in a ratio of 1: 2: 1. The green pixel G may be referred to as a green pixel Gb and Gr, respectively, depending on, for example, adjacent pixels in the horizontal direction. Typically, four pixels of red pixel R, green pixels Gb and Gr, and blue pixel B form one pixel block, and the pixel block groups are arranged in an array to form a Bayer array. To do. In the example shown in FIG. 6A, one of the green pixels G (that is, the green pixel Gr) of the four adjacent pixel groups is formed as the correction green pixel Gr'. Alternatively, although not shown, the green pixel Gb may be formed as the correction green pixel Gb'.

Further, FIG. 3B shows an example in which the correction pixels are arranged so that the distance between the correction pixels is larger than the distance between the pixels of the same color component. That is, the corrected green pixels Gr'shown in FIG. 3B are decimated and arranged as compared with FIG. In this example, every other green pixel Gr is formed as a correction green pixel Gr', but the interval thereof can be appropriately set. As another example, the correction pixels (not limited to the green pixels) may be randomly arranged in an array of a plurality of pixels of the pixel array unit 20.

3A and 3B are diagrams for explaining an example of the pixel arrangement of the pixel array unit 20 according to the embodiment of the present technology. Specifically, FIGS. 3A and 3B show an example of an array of pixels in which the pixels of each color component are arranged according to a quad array. The quad array is an array in which four adjacent pixels have the same color component. As shown in the figure, if four adjacent pixels of the same color component are regarded as one pixel block, the pixel block group of each color component has a Bayer array. In other words, the quad array forms a pixel block composed of 2 × 2 pixels for each color component, and the pixel blocks are arranged according to the Bayer array. In this sense, the pixel array as shown in the figure is sometimes referred to as a quad Bayer array.

In the example shown in FIG. A (a), one pixel (for example, the lower left pixel in the pixel block in the figure) of four adjacent pixels having the same color component is formed as a correction pixel.

Further, for example, as shown in FIG. 3A (b), two pixels out of four adjacent pixels having the same color component (for example, pixels in the figure, that is, pixels located diagonally in the pixel block) are formed as correction pixels. May be done. As a result, the number of correction pixels is increased, so that it is possible to prevent the spatial resolution from being lowered in the low illuminance region.

Further, for example, as shown in FIG. 3B (c), a part of each of the four red pixel group R and the four blue pixel group B may be formed as correction pixels. In this example, the green pixel G does not include the correction pixel. Therefore, for example, by effectively using the signal read from the green pixel G in the re-mosaic process for converting to a different pixel array, the re-mosaic process can be efficiently performed.

Further, for example, as shown in FIG. 3B (d), a part of the green pixel G adjacent to the red pixel R may be formed as a correction pixel in the horizontal direction. As a result, the number of correction pixels can be reduced, and the calculation load required for the correction process can be reduced.

FIG. 4A is a partial vertical sectional view showing an example of a schematic structure of pixels in a pixel array unit according to an embodiment of the present technology. In the figure, adjacent normal pixels 200a and correction pixels 200b are shown.

As shown in the figure, the pixel 200 in the pixel array unit 20 includes, for example, a microlens 201, a filter layer 210, a semiconductor substrate 220, and a wiring layer 230, and these are laminated in this order. That is, the imaging device 1 (see FIG. 1) of the present disclosure is a back-illuminated imaging device in which a wiring layer 230 is provided on a surface opposite to the surface of the semiconductor substrate 220 to be irradiated with light.

The microlens 201 is formed on the surface of the pixel 200 and collects the incident light on the irradiation surface of the semiconductor substrate 220 via the filter layer 210. In this example, one microlens 201 is formed for one pixel 200, but the present invention is not limited to this, and one microlens 201 is formed for a plurality of (for example, two or four) pixels 200. May be formed.

The filter layer 210 includes a color filter so that each pixel 200 receives light of a specific color component. Color filters include, but are not limited to, for example, red, green and blue filters. As mentioned above, in the present disclosure, the color filter is configured to follow a quad or Bayer sequence.

The semiconductor substrate 220 includes a photoelectric conversion element 222 that receives incident light and accumulates electric charges. The photoelectric conversion element 222 is, for example, an embedded photodiode composed of a P-type semiconductor region 224 and an N-type semiconductor region 226. The N-type semiconductor region 226 of this example is a charge storage region in which charges can be stored. In this example, the N-type semiconductor region 226 of the photoelectric conversion element 222 that functions as the correction pixel 200b is smaller than the N-type semiconductor region 226 of the photoelectric conversion element 222 that functions as the normal pixel 200a (that is, the saturated charge amount). Is formed). The normal pixel 200a and the correction pixel 200b are formed, for example, by applying a resist for forming the N-type semiconductor region 226 on the epitaxial growth layer and performing implantation (impurity injection) in the semiconductor manufacturing process. The correction pixel 200b is formed, for example, by reducing the area of the opening of the resist so that the amount of impurities injected is reduced.

The wiring layer 230 includes a transfer gate electrode 232 and a metal wiring 234. The transfer gate electrode 232 is electrically connected to the photoelectric conversion element 222. A gate voltage is applied to the transfer gate electrode 232 under the control of the control unit 10. By controlling the gate voltage to the transfer gate electrode 232 during the operation of the image pickup device 1, the charge accumulated in the charge storage region is transferred to the floating diffusion 236 (see FIG. 5).

The structure of the normal pixel 200a and the correction pixel 200b is basically the same, but as described above, the correction pixel 200b is formed so that the photoelectric conversion element 222 on the semiconductor substrate 220 is smaller than that of the normal pixel 200a. In that respect, it is different from the normal pixel 200a. As described above, the photoelectric conversion element 222 of the correction pixel 200b is formed to be smaller than the photoelectric conversion element 222 of the normal pixel 200a, so that the saturation charge amount of the correction pixel 200b is smaller than the saturation charge amount of the normal pixel 200a, while the correction is performed. The charge transfer efficiency of the pixel 200b is improved.

FIG. 5 is a partial cross-sectional plan view for explaining pixel electrodes in the pixel array portion of the present technology. The figure shows pixel blocks according to a quad array. In this example, one of four adjacent pixels 200 of the same color is formed as the correction pixel 200b.

As shown in the figure, a floating diffusion 236 is formed in the central portion of the four adjacent pixels 200. Further, the transfer gate electrode 232 of each pixel 200 is arranged so as to surround the floating diffusion 236. That is, the four adjacent pixels 200 share one floating diffusion 236. In this example, the transfer gate electrode 232 of the correction pixel 200b is formed so that its area is larger than that of the transfer gate electrode 232 of the normal pixel 200a. As a result, the charge transfer from the charge storage region of the normal pixel 200a to the floating diffusion 236 can be performed more efficiently.

For example, an amplification transistor 238, a selection transistor 239, and the like are arranged around each pixel 200. The charge-based signal transferred to the floating diffusion 236 via the transfer gate electrode 232 is amplified by the amplification transistor 238 and output as a pixel signal via the selection transistor 239.

If the transfer gate electrode 232 of the correction pixel 200b described above is sufficiently large and has charge transfer characteristics, for example, as shown in FIG. 4B, the charge storage region of the correction pixel 200b (in this example, the N-type semiconductor region). 226) may be formed so as to be the same as or slightly smaller than the charge storage region of the normal pixel 200a. Further, when the charge storage region of the correction pixel 200b is sufficiently small and the charge transfer characteristics are good, as shown in FIG. 4C, the area of the transfer gate electrode 232 of the correction pixel 200b is the transfer gate of the normal pixel 200a. It may be formed to be the same as or slightly larger than that of the electrode 232.

FIG. 6 is a flowchart for explaining an example of pixel signal correction processing in the imaging device according to the embodiment of the present technology. The correction process is executed by the signal processing unit 60. In the following, the correction process for the pixel signal output from the pixel array unit 20 of the quad array shown in FIG. 3A will be described as an example.

As shown in the figure, the signal processing unit 60 receives a pixel signal output from the column processing unit 50 in units of rows (S601). The received pixel signal is temporarily held in the image memory 70 for correction processing and the like.

Subsequently, the signal processing unit 60 calculates the sensitivity ratio of the normal pixel and the sensitivity ratio of the correction pixel for the pixel of the predetermined color component based on the received pixel signal (S602). Specifically, in this example, the signal processing unit 60 calculates the sensitivity ratio of the normal red pixel R and the sensitivity ratio of the corrected red pixel R', respectively, and the sensitivity ratio of the normal blue pixel B and the corrected blue pixel B'. Calculate the sensitivity ratio of each. The sensitivity ratio of the normal red pixel R is, for example, the ratio of the charge amount of the normal red pixel R to the charge amount of the normal green pixel Gr, and the sensitivity ratio of the correction red pixel R'is, for example, the correction green pixel Gr'. It is the ratio of the charge amount of the corrected red pixel R'to the charge amount of. Further, the sensitivity ratio of the normal blue pixel B is, for example, the ratio of the charge amount of the normal blue pixel B to the charge amount of the normal green pixel Gb, and the sensitivity ratio of the correction blue pixel B'is, for example, the correction green pixel. It is a ratio of the charge amount of the corrected blue pixel B'to the charge amount of Gb'.

Next, the signal processing unit 60 calculates the difference between the calculated sensitivity ratio of the correction pixel and the sensitivity ratio of the normal pixel for the pixel having the above-mentioned predetermined color component (S603). That is, the signal processing unit 60 calculates the absolute value | Δs_R | of the difference between the calculated sensitivity ratio of the corrected red pixel R'and the sensitivity ratio of the normal red pixel R, and also calculates the sensitivity ratio of the corrected blue pixel B'and the normal. The absolute value | Δs_B | of the difference between the blue pixel B and the sensitivity ratio is calculated.

Next, the signal processing unit 60 determines whether or not the absolute value | Δs_R | of the difference between the calculated sensitivity ratio of the corrected red pixel R'and the sensitivity ratio of the normal red pixel R is smaller than the image quality parameter reference value ref_R. (S604). The image quality parameter reference value ref_R is a numerical value (for example, 0.1) adjusted and determined in advance with respect to the image quality of the red pixel. That is, when the absolute value | Δs_R | of the difference is smaller than the image quality parameter reference value ref_R, the charge amount of the normal red pixel R is sufficiently large as compared with the charge amount of the corrected red pixel R'. It means that there is no problem in the charge transfer of the red pixel R. When the signal processing unit 60 determines that the absolute value | Δs_R | of the difference between the sensitivity ratio of the corrected red pixel R'and the sensitivity ratio of the normal red pixel R is smaller than the image quality parameter reference value ref_R (Yes in S604). Subsequently, it is determined whether or not the absolute value | Δs_B | of the difference between the calculated sensitivity ratio of the corrected blue pixel B'and the sensitivity ratio of the normal blue pixel B is smaller than the image quality parameter reference value ref_B (S605). .. The image quality parameter reference value ref_B is a numerical value (for example, 0.1) adjusted and determined in advance with respect to the image quality of the blue pixel. That is, when the absolute value | Δs_B | of the difference is smaller than the image quality parameter reference value ref_B, the charge amount of the normal blue pixel B is sufficient when compared with the charge amount of the corrected blue pixel B', so that the normal blue pixel is usually a blue pixel. It means that there is no problem in the charge transfer of B.

The signal processing unit 60 determines that the absolute value | Δs_R | of the difference between the sensitivity ratio of the corrected red pixel R'and the sensitivity ratio of the normal red pixel R is smaller than the image quality parameter reference value ref_R (Yes in S604), and When it is determined that the absolute value | Δs_B | of the difference between the sensitivity ratio of the corrected blue pixel B'and the sensitivity ratio of the normal blue pixel B is smaller than the image quality parameter reference value ref_B (Yes in S605), the signal processing unit 60 , Correct the signal of the correction pixel 200b (S606). That is, when the output amount of electric charge from the pixel is the OF (OverFlow) level (that is, the maximum), the signal processing unit 60 corrects the signal of the correction pixel 200b. Specifically, as shown in FIG. 7A, the signal processing unit 60 transmits a signal based on the electric charge from the correction pixel 200b to the electric charge from three normal pixels 200a having the same color component adjacent to each other in the same pixel block. Correct based on the signal based on. At this time, the signal processing unit 60 calculates, for example, the average of the charges from the normal pixels 200a of the same color component adjacent to the correction pixels 200b, and adjusts the charges from the correction pixels 200b according to the signal based on the average of the charges. Based on the signal may be corrected. In this example, the signal processing unit 60 outputs the signal based on the electric charge of the normal pixel 200a as it is without correction.

When the signal processing unit 60 corrects the signal of the correction pixel 200b, the signal processing unit 60 sends the signal to the subsequent stage (S611) and ends the processing for the pixel signal.

Further, the signal processing unit 60 determines that the absolute value | Δs_R | of the difference between the sensitivity ratio of the corrected red pixel R'and the sensitivity ratio of the normal red pixel R is smaller than the image quality parameter reference value ref_R (Yes in S604). In addition, when it is determined that the absolute value | Δs_B | of the difference between the sensitivity ratio of the corrected blue pixel B'and the sensitivity ratio of the normal blue pixel B is not smaller than the image quality parameter reference value ref_B (No in S605), signal processing The unit 60 corrects the signals of the normal blue pixel B and the normal green pixel Gb (S607). That is, when the amount of charge output from the blue pixel is in the middle (between a high level (high illuminance region) and a low level (low illuminance region)), the signals of the normal blue pixel B and the normal green pixel Gb are corrected. Will be done. Specifically, as shown in FIG. 7B, the signal processing unit 60 transmits signals based on the charges of the normal blue pixel B and the normal green pixel Gb to the corrected blue pixel B'and the correction blue pixel B'of the same pixel block, respectively. Correction Correction is performed based on the signal based on the charge from the green pixel Gb'. In this case, the signal processing unit 60 may be configured to normally correct only the signal of the blue pixel B.

When the signal processing unit 60 corrects the signals of the normal blue pixel B and the normal green pixel Gb, the signal processing unit 60 sends the signal to the subsequent stage (S611) and ends the processing for the pixel signal.

On the other hand, when the signal processing unit 60 determines that the difference between the sensitivity ratio of the corrected red pixel R'and the sensitivity ratio of the normal red pixel R is not smaller than the image quality parameter reference value ref_R (No in S604), the calculated correction is performed. It is determined whether or not the absolute value | Δs_B | of the difference between the sensitivity ratio of the blue pixel B'and the sensitivity ratio of the normal blue pixel B is smaller than the image quality parameter reference value ref_B (S608). The signal processing unit 60 determines that the absolute value | Δs_R | of the difference between the sensitivity ratio of the corrected red pixel R'and the sensitivity ratio of the normal red pixel R is not smaller than the image quality parameter reference value ref_R (No in S604). When it is determined that the absolute value | Δs_B | of the difference between the sensitivity ratio of the corrected blue pixel B'and the sensitivity ratio of the normal blue pixel B is smaller than the image quality parameter reference value ref_B (Yes in S608), the signal processing unit 60 Corrects the signals of the normal red pixel R and the normal green pixel Gr (S609). That is, in this example, when the amount of charge output from the red pixel is between a high level and a low level, the signals of the normal red pixel R and the normal green pixel Gr are corrected. Specifically, as shown in FIG. 7C, the signal processing unit 60 transmits signals based on the charges of the normal red pixel R and the normal green pixel Gr to the corrected red pixel R'and the corrected green of the same pixel block, respectively. The correction is based on the signal based on the charge from the pixel Gr'. In this case, the signal processing unit 60 may usually correct only the signal of the red pixel R.

When the signal processing unit 60 corrects the signals of the normal red pixel R and the normal green pixel Gr, the signal processing unit 60 sends the signal to the subsequent stage (S611) and ends the processing for the pixel signal.

Further, the signal processing unit 60 determines that the absolute value | Δs_R | of the difference between the sensitivity ratio of the corrected red pixel R'and the sensitivity ratio of the normal red pixel R is not smaller than the image quality parameter reference value ref_R (No. of S604). ), And when it is determined that the absolute value | Δs_B | of the difference between the sensitivity ratio of the corrected blue pixel B'and the sensitivity ratio of the normal blue pixel B is smaller than the image quality parameter reference value ref_B (No in S608), signal processing The unit 60 corrects the signal of the normal pixel 200a (S610). That is, when the output amount of electric charge is at an extremely low level, the normal pixel 200a is corrected. Specifically, as shown in FIG. 7D, the signal processing unit 60 converts a signal based on the charges of the three normal pixels 200a into a signal based on the charges from adjacent correction pixels 200b in the same pixel block. Correct based on.

When the signal processing unit 60 corrects the signal of the normal pixel 200a, the signal processing unit 60 sends the signal to the subsequent stage (S611) and ends the processing for the pixel signal.

As described above, in the case of the pixel arrangement of the quad arrangement, the signal processing unit 60 can correct the correction pixels 200b in the high illuminance region and can correct the normal pixels 200a in the low illuminance region. This makes it possible to prevent deterioration of the linearity of the pixel characteristics. Further, the signal processing unit 60 can correct the pixels 200 for each color component.

FIG. 8 is a flowchart for explaining an example of pixel signal correction processing in the imaging device according to the embodiment of the present technology. The correction process is executed by the signal processing unit 60. In the following, the correction process for the pixel signal output from the pixel array unit 20 of the Bayer array shown in FIG. 2 will be described as an example. Further, in this example, the green pixel Gr is used as the correction pixel.

As shown in the figure, the signal processing unit 60 receives a pixel signal output from the column processing unit 50 in units of rows (S801). The received pixel signal is temporarily held in the image memory 70 for correction processing and the like.

Subsequently, the signal processing unit 60 calculates the sensitivity ratio of the normal pixel and the sensitivity ratio of the correction pixel for the pixel of the predetermined color component based on the received pixel signal (S802). Specifically, the signal processing unit 60 calculates the sensitivity ratio of the normal green pixel Gb and the sensitivity ratio of the corrected green pixel Gr', respectively. The sensitivity ratio of the normal green pixel Gb is, for example, the ratio of the charge amount of the normal green pixel Gb to the charge amount of the normal red pixel R, and the sensitivity ratio of the corrected green pixel Gr'is, for example, the charge amount of the normal red pixel R. It is the ratio of the charge amount of the corrected green pixel Gr'to the charge amount.

Next, the signal processing unit 60 calculates the difference between the calculated sensitivity ratio of the correction pixel and the sensitivity ratio of the normal pixel for the pixel of the predetermined color component (S803). That is, the signal processing unit 60 calculates the absolute value | Δs_G | of the difference between the calculated sensitivity ratio of the normal green pixel Gb and the sensitivity ratio of the corrected green pixel Gr'.

Next, the signal processing unit 60 determines whether or not the absolute value | Δs_G | of the difference between the calculated sensitivity ratio of the normal green pixel Gb and the sensitivity ratio of the corrected green pixel Gr'is smaller than the image quality parameter reference value ref_G. (S804). The image quality parameter reference value ref_G is, for example, a predetermined numerical value (for example, 0.1). That is, when the absolute value | Δs_G | of the difference is smaller than the image quality parameter reference value ref_G, the amount of charge of the normal green pixel Gb is sufficient when compared with the amount of charge of the corrected green pixel Gr', that is, it is usually green. It means that there is no problem in the charge transfer of the pixel Gb.

When the signal processing unit 60 determines that the absolute value | Δs_G | of the difference between the sensitivity ratio of the corrected green pixel Gr'and the sensitivity ratio of the normal green pixel Gb is smaller than the image quality parameter reference value ref_G (Yes in S804), Correction The signal of the green pixel Gr'is corrected (S805). Specifically, as shown in FIG. 9A, the signal processing unit 60 bases the signal based on the charge from the corrected green pixel Gr'based on the signal based on the charge from the normal green pixel Gb having the same color component. to correct. The normal green pixel Gb having the same color component is a pixel arranged around the correction green pixel Gr'. At this time, the signal processing unit 60 may correct the signal based on the charge from the correction pixel 200b, for example, according to the output amount of the charge from the normal pixel 200a of the different color component adjacent to the correction pixel 200b. When the signal processing unit 60 corrects the signal of the corrected green pixel Gr', the signal processing unit 60 sends the signal to the subsequent stage (S806) and ends the processing for the pixel signal.

On the other hand, when the signal processing unit 60 determines that the difference between the sensitivity ratio of the corrected green pixel Gr'and the sensitivity ratio of the normal green pixel Gb is not smaller than the image quality parameter reference value ref_G (No in S804), the normal green pixel Gb (S806). Specifically, as shown in FIG. 9B, the signal processing unit 60 converts the signal based on the charge from the normal green pixel Gb into the charge from the corrected green pixel Gr'of the same color component in the same pixel block. Correct based on the signal based. When the signal processing unit 60 normally corrects the signal of the green pixel Gb, the signal processing unit 60 sends the signal to the subsequent stage (S806) and ends the processing for the pixel signal.

As described above, in the case of the Bayer array of pixels, the signal processing unit 60 can correct the correction pixels 200b in the high illuminance region and can correct the normal pixels 200a in the low illuminance region. This makes it possible to prevent deterioration of the linearity of the pixel characteristics.

<Example of application to electronic devices>
The image pickup device 1 described above can be applied to various electronic devices such as an image pickup device such as a digital still camera or a digital video camera, a mobile phone having an image pickup function, or another device having an image pickup function. ..

FIG. 10 is a block diagram showing an example of the configuration of an imaging device as an electronic device to which the present technology is applied. As shown in the figure, the image pickup device 100 may include an optical system 12, a shutter device 14, an image pickup device 1, a control unit 10, a signal processing unit 60, an image memory 70, and a monitor 80.

The optical system 12 may be composed of one or a plurality of lenses. The optical system 12 guides the light from the subject to the image pickup device 1 and causes the pixel array unit 20 of the image pickup device 1 to form an image. Further, the optical system 12 adjusts the focus of the lens and controls the drive according to the control of the control unit 10.

As described above, the shutter device 14 performs an electronic shutter operation by sweeping out (resetting) unnecessary charges by the sweeping scanning circuit 34 of the imaging device 1. The shutter device 14 controls the light irradiation period and the light receiving period of the image pickup device 1 according to the control of the control unit 10.

The monitor 80 displays the image data obtained by the signal processing unit 60 performing signal processing. The user of the image pickup apparatus (for example, the photographer) can observe the image data from the monitor 80 through the eyepiece (not shown).

Each of the above embodiments is an example for explaining the present technology, and is not intended to limit the present technology to these embodiments only. The present technology can be implemented in various forms as long as it does not deviate from its gist.

For example, in the methods disclosed herein, steps, actions or functions may be performed in parallel or in a different order, as long as the results are not inconsistent. The steps, actions and functions described are provided merely as examples, and some of the steps, actions and functions can be omitted and combined with each other to the extent that they do not deviate from the gist of the invention. It may be one, or other steps, actions or functions may be added.

In addition, although various embodiments are disclosed in the present specification, specific features (technical matters) in one embodiment are added to other embodiments, or the other embodiments, while being appropriately improved. It can be replaced with a specific feature in the embodiment, and such a form is also included in the gist of the present technology.

<Example of application to mobiles>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on a moving body of any kind such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot. You may.

FIG. 11 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technique according to the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via the communication network 12001. In the example shown in FIG. 11, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown.

The drive system control unit 12010 controls the operation of the device related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 provides a driving force generator for generating the driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism for adjusting and a braking device for generating a braking force of a vehicle.

The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, blinkers or fog lamps. In this case, the body system control unit 12020 may be input with radio waves transmitted from a portable device that substitutes for the key or signals of various switches. The body system control unit 12020 receives inputs of these radio waves or signals and controls a vehicle door lock device, a power window device, a lamp, and the like.

The vehicle outside information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the image pickup unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle outside information detection unit 12030 causes the image pickup unit 12031 to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as a person, a vehicle, an obstacle, a sign, or characters on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal according to the amount of the light received. The image pickup unit 12031 can output an electric signal as an image or can output it as distance measurement information. Further, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

The in-vehicle information detection unit 12040 detects the in-vehicle information. For example, a driver state detection unit 12041 that detects the driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether the driver is dozing.

The microcomputer 12051 calculates the control target value of the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and the drive system control unit. A control command can be output to 12010. For example, the microcomputer 12051 realizes ADAS (Advanced Driver Assistance System) functions including vehicle collision avoidance or impact mitigation, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, and the like. It is possible to perform cooperative control for the purpose of.

Further, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, so that the driver can control the driver. It is possible to perform coordinated control for the purpose of automatic driving, etc., which runs autonomously without depending on the operation.

Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the external information detection unit 12030, and performs coordinated control for the purpose of anti-glare such as switching the high beam to the low beam. It can be carried out.

The audio image output unit 12052 transmits the output signal of at least one of the audio and the image to the output device capable of visually or audibly notifying the passenger or the outside of the vehicle of the information. In the example of FIG. 11, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an onboard display and a heads-up display.

FIG. 12 is a diagram showing an example of the installation position of the imaging unit 12031.

In FIG. 12, the vehicle 12100 has

image pickup units

12101, 12102, 12103, 12104, 12105 as the image pickup unit 12031.

The

imaging units

12101, 12102, 12103, 12104, 12105 are provided at positions such as the front nose, side mirrors, rear bumpers, back doors, and the upper part of the windshield in the vehicle interior of the vehicle 12100, for example. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 12100. The

imaging units

12102 and 12103 provided in the side mirrors mainly acquire images of the side of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The images in front acquired by the

imaging units

12101 and 12105 are mainly used for detecting the preceding vehicle, pedestrians, obstacles, traffic lights, traffic signs, lanes, and the like.

Note that FIG. 12 shows an example of the photographing range of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the

imaging units

12102 and 12103 provided on the side mirrors, respectively, and the imaging range 12114 indicates the imaging range of the

imaging units

12102 and 12103. The imaging range of the imaging unit 12104 provided on the rear bumper or the back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 as viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image pickup units 12101 to 12104 may be a stereo camera composed of a plurality of image pickup elements, or may be an image pickup element having pixels for phase difference detection.

For example, the microcomputer 12051 has a distance to each three-dimensional object within the imaging range 12111 to 12114 based on the distance information obtained from the imaging units 12101 to 12104, and a temporal change of this distance (relative velocity with respect to the vehicle 12100). By obtaining, it is possible to extract as the preceding vehicle a three-dimensional object that is the closest three-dimensional object on the traveling path of the vehicle 12100 and that travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, 0 km / h or more). it can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in front of the preceding vehicle in advance, and can perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. In this way, it is possible to perform cooperative control for the purpose of automatic driving or the like in which the vehicle travels autonomously without depending on the operation of the driver.

For example, the microcomputer 12051 converts three-dimensional object data related to a three-dimensional object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, electric poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that can be seen by the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 via the audio speaker 12061 or the display unit 12062. By outputting an alarm to the driver and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be provided.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. Such pedestrian recognition includes, for example, a procedure for extracting feature points in an image captured by an imaging unit 12101 to 12104 as an infrared camera, and pattern matching processing for a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. It is done by the procedure to determine. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a square contour line for emphasizing the recognized pedestrian. The display unit 12062 is controlled so as to superimpose and display. Further, the audio image output unit 12052 may control the display unit 12062 so as to display an icon or the like indicating a pedestrian at a desired position.

The above is an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technique according to the present disclosure can be applied to the imaging unit 12301 among the configurations described above. Specifically, some of the plurality of pixels in the pixel array unit of the imaging unit 12301 may be formed so as to function as correction pixels. By applying the technique according to the present disclosure to the image pickup unit 12301, it is possible to perform correction processing of correction pixels or normal pixels and prevent deterioration of image quality due to deterioration of linearity of pixel characteristics.

<Example of application to endoscopic surgery system>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the techniques according to the present disclosure may be applied to endoscopic surgery systems.

FIG. 13 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technique according to the present disclosure (the present technique) can be applied.

FIG. 13 illustrates how the surgeon (doctor) 11131 is performing surgery on patient 11132 on patient bed 11133 using the endoscopic surgery system 11000. As shown, the endoscopic surgery system 11000 includes an endoscope 11100, other surgical tools 11110 such as an abdominal tube 11111 and an energy treatment tool 11112, and a support arm device 11120 that supports the endoscope 11100. , A cart 11200 equipped with various devices for endoscopic surgery.

The endoscope 11100 is composed of a lens barrel 11101 in which a region having a predetermined length from the tip is inserted into the body cavity of the patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 configured as a so-called rigid mirror having a rigid barrel 11101 is illustrated, but the endoscope 11100 may be configured as a so-called flexible mirror having a flexible barrel. Good.

An opening in which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and the light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101 to be an objective. It is irradiated toward the observation target in the body cavity of the patient 11132 through the lens. The endoscope 11100 may be a direct endoscope, a perspective mirror, or a side endoscope.

An optical system and an image sensor are provided inside the camera head 11102, and the reflected light (observation light) from the observation target is focused on the image sensor by the optical system. The observation light is photoelectrically converted by the image sensor, and an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image is generated. The image signal is transmitted as RAW data to the camera control unit (CCU: Camera Control Unit) 11201.

The CCU11201 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and comprehensively controls the operations of the endoscope 11100 and the display device 11202. Further, the CCU 11201 receives an image signal from the camera head 11102, and performs various image processing on the image signal for displaying an image based on the image signal, such as development processing (demosaic processing).

The display device 11202 displays an image based on the image signal processed by the CCU 11201 under the control of the CCU 11201.

The light source device 11203 is composed of, for example, a light source such as an LED (Light Emitting Diode), and supplies irradiation light to the endoscope 11100 when photographing an operating part or the like.

The input device 11204 is an input interface for the endoscopic surgery system 11000. The user can input various information and input instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) by the endoscope 11100.

The treatment tool control device 11205 controls the drive of the energy treatment tool 11112 for cauterizing, incising, sealing a blood vessel, or the like of a tissue. The pneumoperitoneum device 11206 uses a gas in the pneumoperitoneum tube 11111 to inflate the body cavity of the patient 11132 for the purpose of securing the field of view by the endoscope 11100 and securing the work space of the operator. To send. Recorder 11207 is a device capable of recording various information related to surgery. The printer 11208 is a device capable of printing various information related to surgery in various formats such as text, images, and graphs.

The light source device 11203 that supplies the irradiation light to the endoscope 11100 when photographing the surgical site can be composed of, for example, an LED, a laser light source, or a white light source composed of a combination thereof. When a white light source is configured by combining RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high accuracy. Therefore, the light source device 11203 adjusts the white balance of the captured image. It can be carried out. Further, in this case, the laser light from each of the RGB laser light sources is irradiated to the observation target in a time-division manner, and the drive of the image sensor of the camera head 11102 is controlled in synchronization with the irradiation timing to correspond to each of RGB. It is also possible to capture the image in a time-division manner. According to this method, a color image can be obtained without providing a color filter on the image sensor.

Further, the drive of the light source device 11203 may be controlled so as to change the intensity of the output light at predetermined time intervals. By controlling the drive of the image sensor of the camera head 11102 in synchronization with the timing of changing the light intensity to acquire images in a time-divided manner and synthesizing the images, so-called high dynamic without blackout and overexposure Range images can be generated.

Further, the light source device 11203 may be configured to be able to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, by utilizing the wavelength dependence of light absorption in body tissue to irradiate light in a narrow band as compared with the irradiation light (that is, white light) in normal observation, the surface layer of the mucous membrane. A so-called narrow band imaging (Narrow Band Imaging) is performed in which a predetermined tissue such as a blood vessel is photographed with high contrast. Alternatively, in the special light observation, fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiating with excitation light. In fluorescence observation, the body tissue is irradiated with excitation light to observe the fluorescence from the body tissue (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and the body tissue is injected. It is possible to obtain a fluorescence image by irradiating excitation light corresponding to the fluorescence wavelength of the reagent. The light source device 11203 may be configured to be capable of supplying narrow band light and / or excitation light corresponding to such special light observation.

FIG. 14 is a block diagram showing an example of the functional configuration of the camera head 11102 and CCU11201 shown in FIG.

The camera head 11102 includes a lens unit 11401, an imaging unit 11402, a driving unit 11403, a communication unit 11404, and a camera head control unit 11405. CCU11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and CCU11201 are communicatively connected to each other by a transmission cable 11400.

The lens unit 11401 is an optical system provided at a connection portion with the lens barrel 11101. The observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and incident on the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

The image pickup unit 11402 is composed of an image pickup element. The image sensor constituting the image pickup unit 11402 may be one (so-called single plate type) or a plurality (so-called multi-plate type). When the image pickup unit 11402 is composed of a multi-plate type, for example, each image pickup element may generate an image signal corresponding to each of RGB, and a color image may be obtained by synthesizing them. Alternatively, the image pickup unit 11402 may be configured to have a pair of image pickup elements for acquiring image signals for the right eye and the left eye corresponding to 3D (Dimensional) display, respectively. The 3D display enables the operator 11131 to more accurately grasp the depth of the biological tissue in the surgical site. When the image pickup unit 11402 is composed of a multi-plate type, a plurality of lens units 11401 may be provided corresponding to each image pickup element.

Further, the imaging unit 11402 does not necessarily have to be provided on the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101 immediately after the objective lens.

The drive unit 11403 is composed of an actuator, and the zoom lens and focus lens of the lens unit 11401 are moved by a predetermined distance along the optical axis under the control of the camera head control unit 11405. As a result, the magnification and focus of the image captured by the imaging unit 11402 can be adjusted as appropriate.

The communication unit 11404 is composed of a communication device for transmitting and receiving various information to and from the CCU11201. The communication unit 11404 transmits the image signal obtained from the image pickup unit 11402 as RAW data to the CCU 11201 via the transmission cable 11400.

Further, the communication unit 11404 receives a control signal for controlling the drive of the camera head 11102 from the CCU 11201 and supplies the control signal to the camera head control unit 11405. The control signal includes, for example, information to specify the frame rate of the captured image, information to specify the exposure value at the time of imaging, and / or information to specify the magnification and focus of the captured image, and the like. Contains information about the condition.

The above-mentioned imaging conditions such as frame rate, exposure value, magnification, and focus may be appropriately specified by the user, or may be automatically set by the control unit 11413 of CCU11201 based on the acquired image signal. Good. In the latter case, the so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function are mounted on the endoscope 11100.

The camera head control unit 11405 controls the drive of the camera head 11102 based on the control signal from the CCU 11201 received via the communication unit 11404.

The communication unit 11411 is composed of a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

Further, the communication unit 11411 transmits a control signal for controlling the drive of the camera head 11102 to the camera head 11102. Image signals and control signals can be transmitted by telecommunications, optical communication, or the like.

The image processing unit 11412 performs various image processing on the image signal which is the RAW data transmitted from the camera head 11102.

The control unit 11413 performs various controls related to the imaging of the surgical site and the like by the endoscope 11100 and the display of the captured image obtained by the imaging of the surgical site and the like. For example, the control unit 11413 generates a control signal for controlling the drive of the camera head 11102.

Further, the control unit 11413 causes the display device 11202 to display an image captured by the surgical unit or the like based on the image signal processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image by using various image recognition techniques. For example, the control unit 11413 detects the shape, color, and the like of the edge of an object included in the captured image to remove surgical tools such as forceps, a specific biological part, bleeding, and mist when using the energy treatment tool 11112. Can be recognized. When displaying the captured image on the display device 11202, the control unit 11413 may superimpose and display various surgical support information on the image of the surgical unit by using the recognition result. By superimposing and displaying the surgical support information and presenting it to the surgeon 11131, it is possible to reduce the burden on the surgeon 11131 and to allow the surgeon 11131 to proceed with the surgery reliably.

The transmission cable 11400 that connects the camera head 11102 and CCU11201 is an electric signal cable that supports electric signal communication, an optical fiber that supports optical communication, or a composite cable thereof.

Here, in the illustrated example, the communication was performed by wire using the transmission cable 11400, but the communication between the camera head 11102 and the CCU11201 may be performed wirelessly.

The above is an example of an endoscopic surgery system to which the technology according to the present disclosure can be applied. The technique according to the present disclosure can be applied to the imaging unit 11402 of the camera head 11102 among the configurations described above. Specifically, some of the plurality of pixels in the pixel array unit of the image pickup unit 11402 may be formed so as to function as correction pixels. By applying the technique according to the present disclosure to the image pickup unit 11402, it is possible to perform correction processing of correction pixels or normal pixels and prevent deterioration of image quality due to deterioration of pixel-specific linearity.

Although the endoscopic surgery system has been described here as an example, the technique according to the present disclosure may be applied to other, for example, a microscopic surgery system.

In addition, the present technology may be configured to include the following technical matters.
(1)
An imaging device including a plurality of pixels capable of accumulating electric charges generated according to the amount of incident light.
The plurality of pixels
A first pixel formed to have a first saturated charge amount and
A second pixel formed to have a second saturated charge amount smaller than the first saturated charge amount, and the like.
Imaging device.
(2)
The plurality of pixels consist of an array of pixel blocks including the plurality of the first pixels and at least one of the second pixels.
The imaging device according to (1) above.
(3)
Each of the pixel blocks is composed of four adjacent pixels having the same color component, and the array of the pixel blocks constitutes a Bayer array.
The imaging device according to (1) or (2) above.
(4)
The plurality of first pixels in the pixel block are provided so as to correspond to any of the predetermined color components.
The at least one second pixel in the pixel block is provided so as to correspond to any of the color components.
The imaging device according to (3) above.
(5)
The at least one second pixel is provided so as to correspond to the green component.
The imaging device according to (1) or (4) above.
(6)
The at least one second pixel is provided so as to correspond to a color component other than the green component.
The imaging device according to (1) or (4) above.
(7)
The second pixel is arranged diagonally to each other in the pixel block.
The imaging device according to (2) above.
(8)
The plurality of the second pixels are arranged so that the distance between the second pixels is larger than the size of the pixel block.
The imaging device according to (2) or (7) above.
(9)
The plurality of pixels are provided so as to correspond to any one of the predetermined color components according to the Bayer arrangement.
The imaging device according to (1) above.
(10)
The plurality of second pixels are provided so as to correspond to either a green component adjacent to a blue component or a green component adjacent to a red component in one direction of the Bayer array.
The imaging device according to (9) above.
(11)
The plurality of second pixels are arranged so that the distance between the second pixels is larger than the distance between any of the color components.
The imaging device according to (9) or (10) above.
(12)
The area of the transfer gate electrode for transferring the charge from the charge accumulating region in the second pixel is the area of the transfer gade electrode for transferring the charge from the charge accumulating region in the first pixel. Greater than
The imaging device according to any one of (1) to (11).
(13)
The charge storage region for accumulating charges in the second pixel is smaller than the charge storage region for accumulating charges in the first pixel.
The imaging device according to any one of (1) to (12).
(14)
A signal processing circuit for processing a signal based on the amount of charge transferred from the plurality of pixels is further provided.
The signal processing circuit is a signal based on the amount of charge transferred from any of the plurality of pixels based on the amount of charge transferred from the first pixel and the amount of charge transferred from the second pixel. To correct,
The imaging device according to any one of (1) to (13).
(15)
The signal processing circuit has a first sensitivity ratio based on the amount of electric charge transferred from each of the plurality of first pixels and a second sensitivity ratio based on the amount of electric charge transferred from the second pixel. On the basis of,
Corrects a signal based on the amount of charge transferred from any of the plurality of pixels.
The imaging device according to (14) above.
(16)
The signal processing circuit calculates the difference between the first sensitivity ratio and the second sensitivity ratio, and transfers the difference from any of the plurality of pixels based on the calculated difference and a predetermined reference value. Correct the signal based on the amount of charge
The imaging device according to (14) or (15).
(17)
In the signal processing circuit, the difference between the first sensitivity ratio with respect to the red component and the second sensitivity ratio is smaller than the predetermined reference value, and the first sensitivity ratio with respect to the blue component and the second sensitivity ratio are described. When the difference from the sensitivity ratio of is smaller than the predetermined reference value, the signal based on the amount of charge transferred from the second pixel is corrected.
The imaging device according to (16) above.
(18)
In the signal processing circuit, the difference between the first sensitivity ratio with respect to the red component and the second sensitivity ratio is smaller than the predetermined reference value, and the first sensitivity ratio with respect to the blue component and the second sensitivity ratio are described. When the difference from the sensitivity ratio of is not smaller than the predetermined reference value, a signal based on the amount of charge transferred from the first pixel of at least the blue component is converted to the amount of charge transferred from the second pixel. Based on the signal based on the correction, or the difference between the first sensitivity ratio and the second sensitivity ratio regarding the red component is not smaller than the predetermined reference value, and the first sensitivity ratio regarding the blue component is not smaller than the predetermined reference value. When the difference between the sensitivity ratio and the second sensitivity ratio is smaller than the predetermined reference value, a signal based on the amount of charge transferred from the first pixel of at least the red component is transferred from the second pixel. Correct based on the signal based on the amount of charge to be made,
The imaging device according to (16) or (17).
(19)
In the signal processing circuit, the difference between the first sensitivity ratio and the second sensitivity ratio regarding the red component is not smaller than the predetermined reference value, and the first sensitivity ratio and the first sensitivity ratio regarding the blue component are not smaller than the predetermined reference value. When the difference from the sensitivity ratio of 2 is not smaller than the predetermined reference value, the signal based on the amount of charge transferred from the first pixel in the same color component is converted to the amount of charge transferred from the second pixel. Correct based on the signal based,
The imaging device according to any one of (16) to (18).
(20)
This is a signal correction processing method in an imaging device.
Reading the charge from a plurality of pixels that have accumulated the charge generated according to the amount of incident light, and
Including correcting a signal based on the amount of charge read from the plurality of pixels.
The plurality of pixels have a first pixel formed to have a first saturated charge amount and a second pixel formed to have a second saturated charge amount smaller than the first saturated charge amount. Including the pixels of
The correction is performed from any of the plurality of pixels based on the sensitivity ratio based on the amount of charge transferred from the first pixel and the sensitivity ratio based on the amount of charge transferred from the second pixel. Correct the signal based on the amount of charge transferred,
Correction processing method.

1 ... Imaging device 10 ... Control unit 20 ... Pixel array unit 200a ... Normal pixel 200b ... Correction pixel 201 ... Microlens 210 ... Filter layer 220 ... Semiconductor substrate 222 ... Photoelectric conversion element 224 ... P-type semiconductor region 226 ... N-type semiconductor region 230 ... Wiring layer 232 ... Transfer gate electrode 234 ... Metal wiring 236 ... Floating diffusion 238 ... Amplification transistor 239 ... Selective transistor 30 ... Vertical drive unit 32 ... Read scanning circuit 34 ... Sweep scanning circuit 40 ... Horizontal drive unit 50 ... Column processing Unit 60 ... Signal processing unit 70 ... Image memory

Claims

An imaging device including a plurality of pixels capable of accumulating electric charges generated according to the amount of incident light.
The plurality of pixels
A first pixel formed to have a first saturated charge amount and
A second pixel formed to have a second saturated charge amount smaller than the first saturated charge amount, and the like.
Imaging device.
The plurality of pixels consist of an array of pixel blocks including the plurality of the first pixels and at least one of the second pixels.
The imaging device according to claim 1.
Each of the pixel blocks is composed of four adjacent pixels having the same color component, and the array of the pixel blocks constitutes a Bayer array.
The imaging device according to claim 2.
The plurality of first pixels in the pixel block are provided so as to correspond to any of the predetermined color components.
The at least one second pixel in the pixel block is provided so as to correspond to any of the color components.
The imaging device according to claim 3.
The at least one second pixel is provided so as to correspond to the green component.
The imaging device according to claim 4.
The at least one second pixel is provided so as to correspond to a color component other than the green component.
The imaging device according to claim 4.
The second pixel is arranged diagonally to each other in the pixel block.
The imaging device according to claim 2.
The plurality of the second pixels are arranged so that the distance between the second pixels is larger than the size of the pixel block.
The imaging device according to claim 2.
The plurality of pixels are provided so as to correspond to any one of the predetermined color components according to the Bayer arrangement.
The imaging device according to claim 1.
The plurality of second pixels are provided so as to correspond to either a green component adjacent to a blue component or a green component adjacent to a red component in one direction of the Bayer array.
The imaging device according to claim 9.
The plurality of second pixels are arranged so that the distance between the second pixels is larger than the distance between any of the color components.
The imaging device according to claim 9.
The area of the transfer gate electrode for transferring the charge from the charge accumulating region in the second pixel is the area of the transfer gate electrode for transferring the charge from the charge accumulating region in the first pixel. Greater than
The imaging device according to claim 1.
The charge storage region for accumulating charges in the second pixel is smaller than the charge storage region for accumulating charges in the first pixel.
The imaging device according to claim 1.
A signal processing circuit for processing a signal based on the amount of charge transferred from the plurality of pixels is further provided.
The signal processing circuit is a signal based on the amount of charge transferred from any of the plurality of pixels based on the amount of charge transferred from the first pixel and the amount of charge transferred from the second pixel. To correct,
The imaging device according to claim 1.
The signal processing circuit has a first sensitivity ratio based on the amount of electric charge transferred from each of the plurality of first pixels and a second sensitivity ratio based on the amount of electric charge transferred from the second pixel. On the basis of,
Corrects a signal based on the amount of charge transferred from any of the plurality of pixels.
The imaging device according to claim 14.
The signal processing circuit calculates the difference between the first sensitivity ratio and the second sensitivity ratio, and transfers the difference from any of the plurality of pixels based on the calculated difference and a predetermined reference value. Correct the signal based on the amount of charge
The imaging device according to claim 15.
In the signal processing circuit, the difference between the first sensitivity ratio with respect to the red component and the second sensitivity ratio is smaller than the predetermined reference value, and the first sensitivity ratio with respect to the blue component and the second sensitivity ratio are described. When the difference from the sensitivity ratio of is smaller than the predetermined reference value, the signal based on the amount of charge transferred from the second pixel is corrected.
The imaging device according to claim 16.
In the signal processing circuit, the difference between the first sensitivity ratio with respect to the red component and the second sensitivity ratio is smaller than the predetermined reference value, and the first sensitivity ratio with respect to the blue component and the second sensitivity ratio are described. When the difference from the sensitivity ratio of is not smaller than the predetermined reference value, a signal based on the amount of charge transferred from the first pixel of at least the blue component is converted to the amount of charge transferred from the second pixel. Based on the signal based on the correction, or the difference between the first sensitivity ratio and the second sensitivity ratio regarding the red component is not smaller than the predetermined reference value, and the first sensitivity ratio regarding the blue component is not smaller than the predetermined reference value. When the difference between the sensitivity ratio and the second sensitivity ratio is smaller than the predetermined reference value, a signal based on the amount of charge transferred from the first pixel of at least the red component is transferred from the second pixel. Correct based on the signal based on the amount of charge to be made,
The imaging device according to claim 16.
In the signal processing circuit, the difference between the first sensitivity ratio and the second sensitivity ratio regarding the red component is not smaller than the predetermined reference value, and the first sensitivity ratio and the first sensitivity ratio regarding the blue component are not smaller than the predetermined reference value. When the difference from the sensitivity ratio of 2 is not smaller than the predetermined reference value, the signal based on the amount of charge transferred from the first pixel in the same color component is converted to the amount of charge transferred from the second pixel. Correct based on the signal based,
The imaging device according to claim 16.
This is a pixel signal correction processing method in an imaging device.
Reading the charge from a plurality of pixels that have accumulated the charge generated according to the amount of incident light, and
Including correcting a pixel signal based on the amount of charge read from the plurality of pixels.
The plurality of pixels have a first pixel formed to have a first saturated charge amount and a second pixel formed to have a second saturated charge amount smaller than the first saturated charge amount. Including the pixels of
The correction is performed from any of the plurality of pixels based on the sensitivity ratio based on the amount of charge transferred from the first pixel and the sensitivity ratio based on the amount of charge transferred from the second pixel. Correct the pixel signal based on the amount of charge transferred,
Correction processing method.