JP2023027618A - Imaging device and electronic apparatus - Google Patents

Abstract

To suppress increase in a device size.SOLUTION: An imaging device according to one embodiment includes: a pixel array equipped with a plurality of light-receiving pixels; a readout unit for generating an image signal comprising pixel values read out respectively from the light-receiving pixels; and a signal processor for processing the image signal output from the readout unit. The pixel array includes a plurality of pixel pairs comprising at least two light-receiving pixels that share one on-chip lens. The signal processor includes: an adder that executes first addition processing to add pixel values read out from a first light-receiving pixel in each of at least two pixel pairs among the plurality of pixel pairs, and to add pixel values read out from a second light-receiving pixel different from the first light-receiving pixel, in each of the pixel pairs; and a generator that generates information regarding an image-surface phase difference between the first and second light-receiving pixels on the basis of, a plurality of pixel values included in the image signal after the first addition processing.SELECTED DRAWING: Figure 11

Description

The present disclosure relates to imaging devices and electronic devices.

For example, there are imaging devices that perform autofocus based on the image plane phase difference. Conventionally, when generating the image plane phase difference information, the sensitivity difference correction of the input image is performed, then pixel addition is performed, and then the pixel values after the addition are used to generate the image plane phase difference information. rice field.

JP 2016-52051 A

However, in recent years, pixel array patterns have become more complex as pixels have become finer, and sensitivity difference correction processing has also become more complicated. There is a problem that

Therefore, the present disclosure proposes an imaging device and an electronic device capable of suppressing an increase in device size.

An imaging device according to an embodiment of the present disclosure includes a pixel array including a plurality of light-receiving pixels, a reading unit that generates an image signal composed of pixel values read from each of the light-receiving pixels, and an image signal output from the readout unit. a signal processing unit that processes the image signal, the pixel array includes a plurality of pixel pairs each composed of at least two light-receiving pixels that share one on-chip lens, and the signal processing unit includes the plurality of adding the pixel values read from the first light-receiving pixels in each of at least two pixel pairs out of the pixel pairs, and reading from a second light-receiving pixel different from the first light-receiving pixel in each of the pixel pairs an addition unit that performs a first addition process for adding the pixel values obtained by adding the pixel values obtained from the first addition process; and a generator that generates information about the image plane phase difference.

1 is a block diagram showing a configuration example of an imaging device according to a first embodiment; FIG. 2 is an explanatory diagram showing one configuration example of a pixel array shown in FIG. 1; FIG. 3 is an explanatory diagram showing a configuration example of a light receiving pixel shown in FIG. 2; FIG. 3 is a circuit diagram showing a configuration example of a pixel block shown in FIG. 2; FIG. 3 is a circuit diagram showing a configuration example of another pixel block shown in FIG. 2; FIG. FIG. 3 is an explanatory diagram showing a connection example of a plurality of pixel blocks shown in FIG. 2; 2 is a block diagram showing a configuration example of a reading unit shown in FIG. 1; FIG. FIG. 2 is an explanatory diagram showing a configuration example of an image signal shown in FIG. 1; 2 is an explanatory diagram showing an example of the number of effective pixels in the imaging device shown in FIG. 1; FIG. 2 is an explanatory diagram showing one operation example of a plurality of imaging modes in the imaging device shown in FIG. 1; FIG. 4 is a block diagram showing a more detailed configuration example of a phase difference data generator according to the first embodiment; FIG. FIG. 3 is a diagram of one unit extracted from an arrangement example of light receiving elements in the pixel array shown in FIG. 2; FIG. 4 is a diagram for explaining same-color pixel addition processing that is performed on pixel values read from pixel blocks according to the first embodiment (No. 1); FIG. 8 is a diagram for explaining same-color pixel addition processing that is performed on pixel values read from each pixel block according to the first embodiment (part 2); FIG. 9 is a diagram for explaining same-color pixel addition processing that is performed on pixel values read from each pixel block according to the first embodiment (No. 3); FIG. 12 is a diagram for explaining same-color pixel addition processing that is performed on pixel values read from each pixel block according to the first embodiment (No. 4); FIG. 4 is a diagram showing left pixel total values and right pixel total values finally obtained based on pixel values read from unitary units according to the first embodiment; FIG. 3 is a diagram of one unit extracted from an arrangement example of light receiving elements in the pixel array shown in FIG. 2; FIG. 4 is a diagram for explaining same-color pixel addition processing executed in HDR mode on pixel values read from each pixel block according to the first embodiment (No. 1); FIG. 10 is a diagram for explaining same-color pixel addition processing executed in HDR mode on pixel values read from each pixel block according to the first embodiment (No. 2); FIG. 11 is a diagram for explaining same-color pixel addition processing executed in HDR mode on pixel values read from each pixel block according to the first embodiment (No. 3); FIG. 10 is a diagram for explaining same-color pixel addition processing performed in HDR mode on pixel values read from each pixel block according to the first embodiment (No. 4); FIG. 10 is a diagram for explaining same-color pixel addition processing performed in HDR mode on pixel values read from each pixel block according to the first embodiment (No. 5); FIG. 16 is a diagram for explaining same-color pixel addition processing executed in HDR mode on pixel values read from each pixel block according to the first embodiment (No. 6); FIG. 10 is a diagram for explaining same-color pixel addition processing performed in HDR mode on pixel values read from each pixel block according to the first embodiment (No. 7); FIG. 11 is a diagram for explaining same-color pixel addition processing executed in HDR mode on pixel values read from each pixel block according to the first embodiment (No. 8); FIG. 4 is a diagram showing left pixel total values and right pixel total values of low-luminance pixels and high-luminance pixels, respectively, finally obtained based on pixel values read out from unitary units according to the first embodiment; FIG. 10 is a diagram of one unit extracted from an arrangement example of light-receiving elements in a pixel array according to a first modification of the first embodiment; FIG. 10 is a diagram for explaining same-color pixel addition processing performed on pixel values read from each pixel block according to the first modification of the first embodiment (No. 1); FIG. 12 is a diagram for explaining same-color pixel addition processing executed on pixel values read from each pixel block according to the first modification of the first embodiment (No. 2); FIG. 12 is a diagram for explaining same-color pixel addition processing performed on pixel values read from each pixel block according to the first modification of the first embodiment (No. 3); FIG. 12 is a diagram for explaining same-color pixel addition processing executed on pixel values read from each pixel block according to the first modification of the first embodiment (No. 4); FIG. 10 is a diagram showing left pixel total values and right pixel total values finally obtained based on pixel values read from unit units according to the first modification of the first embodiment; It is the figure which extracted one unit from the example of arrangement|positioning of the light receiving element in the pixel array based on the 2nd modification of 1st Embodiment. FIG. 10 is a diagram for explaining same-color pixel addition processing executed on pixel values read out for image plane phase difference detection from the unit according to the second modification of the first embodiment; FIG. 10 is a diagram showing left pixel total values and right pixel total values finally obtained based on pixel values read from unit units according to the second modification of the first embodiment; FIG. 11 is a diagram showing left pixel total values and right pixel total values finally obtained based on pixel values read from unit units according to the third modification of the first embodiment; FIG. 14 is a diagram showing left pixel total values and right pixel total values finally obtained based on pixel values read from unit units according to the fourth modification of the first embodiment; FIG. 14 is a diagram showing left pixel total values and right pixel total values finally obtained based on pixel values read from unit units according to the fifth modification of the first embodiment; It is a block diagram showing one structural example of the imaging device which concerns on 2nd Embodiment. 41 is an explanatory diagram showing one configuration example of the pixel array shown in FIG. 40; FIG. FIG. 42 is an explanatory diagram showing a connection example of a plurality of pixel blocks shown in FIG. 41; FIG. 42 is an explanatory diagram showing an operation example of the pixel array shown in FIG. 41; FIG. 11 is an explanatory diagram showing one configuration example of a pixel array according to a modification of the second embodiment; 35 is an explanatory diagram showing a connection example of a plurality of pixel blocks shown in FIG. 34; FIG. It is explanatory drawing showing the usage example of an imaging device. 1 is a block diagram showing an example of a schematic configuration of a vehicle control system; FIG. FIG. 4 is an explanatory diagram showing an example of installation positions of an outside information detection unit and an imaging unit;

Embodiments of the present disclosure will be described in detail below with reference to the drawings. In addition, in the following embodiment, the overlapping description is abbreviate|omitted by attaching|subjecting the same code|symbol to the same site|part.

Also, the present disclosure will be described according to the order of items shown below.
1. First Embodiment 1.1 Configuration Example 1.2 General Operation Example 1.3 Generation of Image Plane Phase Difference Information (Phase Difference Data DF) 1.3.1 More Detailed Configuration Example of Phase Difference Data Generation Unit 1.1. 3.2 Concrete example of same-color pixel addition processing in imaging mode MC 1.3.3 Concrete example of same-color pixel addition processing in HDR mode 1.4 Actions and effects 1.5 Modifications 1.5.1 First Modified example 1.5.2 Second modified example 1.5.3 Third modified example 1.5.4 Fourth modified example 1.5.5 Fifth modified example 2. Second Embodiment 2.1 Configuration Example 2.2 Action and Effect 2.3 Modification 3. Usage example of imaging device 4 . Example of application to mobile objects

1. 1st embodiment

First, a solid-state imaging device and an electronic device according to the first embodiment of the present disclosure will be described in detail with reference to the drawings.

1.1 Configuration Example FIG. 1 is a block diagram showing a configuration example of an imaging apparatus according to this embodiment. The imaging device 1 includes a pixel array 11 , a driving section 12 , a reference signal generating section 13 , a reading section 20 , a signal processing section 15 and an imaging control section 18 . The signal processing unit 15 may be arranged on the same chip (including stacked chips) as the chip on which the pixel array 11 is provided, or may be arranged on a different chip.

The pixel array 11 has a plurality of light receiving pixels P arranged in a matrix. The light receiving pixel P is configured to generate a signal SIG including a pixel voltage Vpix according to the amount of light received.

FIG. 2 shows an example of the arrangement of the light receiving pixels P in the pixel array 11. As shown in FIG. FIG. 3 shows an example of a schematic cross-sectional structure of the pixel array 11. As shown in FIG. The pixel array 11 has multiple pixel blocks 100 and multiple lenses 101 .

The plurality of pixel blocks 100 includes pixel blocks 100R, 100Gr, 100Gb and 100B. In the pixel array 11, a plurality of light-receiving pixels P are arranged in units (units U) of four pixel blocks 100 (pixel blocks 100R, 100Gr, 100Gb, and 100B).

The pixel block 100R has eight light-receiving pixels P (light-receiving pixels PR) including red (R) color filters 55, and the pixel block 100Gr has ten light-receiving pixels including green (G) color filters 55. The pixel block 100Gb has 10 light-receiving pixels P (light-receiving pixels PGb) including green (G) color filters 55, and the pixel block 100B has blue (B) color filters. It has eight light-receiving pixels P (light-receiving pixels PB) including color filters 55 . In FIG. 2, the difference in color of the color filters is expressed using hatching. The arrangement pattern of the light-receiving pixels PR in the pixel block 100R and the arrangement pattern of the light-receiving pixels PB in the pixel block 100B are the same. The patterns are identical to each other. In the unit U, the pixel block 100Gr is located at the upper left, the pixel block 100R is located at the upper right, the pixel block 100B is located at the lower left, and the pixel block 100Gb is located at the lower right. In this manner, the pixel blocks 100R, 100Gr, 100Gb, and 100B are arranged in a so-called Bayer arrangement with the pixel block 100 as a unit.

As shown in FIG. 3, the pixel array 11 includes a semiconductor substrate 51, a semiconductor region 52, an insulating layer 53, a multilayer wiring layer 54, a color filter 55, and a light shielding film . The semiconductor substrate 51 is a support substrate on which the imaging device 1 is formed, and is a P-type semiconductor substrate. The semiconductor region 52 is a semiconductor region provided at a position corresponding to each of the plurality of light receiving pixels P in the substrate of the semiconductor substrate 51, and is doped with an N-type impurity to form the photodiode PD. . The insulating layer 53 is provided on the boundary of a plurality of light receiving pixels P arranged side by side on the XY plane in the substrate of the semiconductor substrate 51, and in this example, is a DTI (Deep Trench Isolation) layer formed using an oxide film or the like. is. The multilayer wiring layer 54 is provided on the semiconductor substrate 51 on the surface opposite to the light incident surface S of the pixel array 11, and includes a plurality of wiring layers and an interlayer insulating film. The wiring in the multilayer wiring layer 54 is configured to connect, for example, a transistor (not shown) provided on the surface of the semiconductor substrate 51 with the driving section 12 and the reading section 20 . The color filter 55 is provided on the semiconductor substrate 51 on the light incident surface S of the pixel array 11 . The light shielding film 116 is provided on the light incident surface S of the pixel array 11 so as to surround two light receiving pixels P (hereinafter also referred to as pixel pairs 90) arranged side by side in the X direction.

The multiple lenses 101 are so-called on-chip lenses, and are provided on the color filter 55 on the light incident surface S of the pixel array 11 . A light shielding film 56 may be arranged on the light incident surface S for optically separating adjacent pixels. The lens 101 is provided above two light receiving pixels P (pixel pairs 90) arranged side by side in the X direction. Four lenses 101 are provided above the eight light-receiving pixels P in the pixel block 100R, five lenses 101 are provided above the ten light-receiving pixels P in the pixel block 100Gr, and ten lenses 101 are provided in the pixel block 100Gb. Five lenses 101 are provided above the light receiving pixels P of the pixel block 100B, and four lenses 101 are provided above the eight light receiving pixels P of the pixel block 100B. The lenses 101 are arranged side by side in the X and Y directions. The lenses 101 arranged in the Y direction are arranged with a shift of one light receiving pixel P in the X direction. In other words, the pixel pairs 90 aligned in the Y direction are arranged with a shift of one light receiving pixel P in the X direction.

With this configuration, the images of the two light receiving pixels P in the pixel pair 90 corresponding to one lens 101 are shifted from each other. The imaging device 1 generates phase difference data DF based on so-called image plane phase differences detected by the plurality of pixel pairs 90 . For example, in a camera equipped with the imaging device 1, the defocus amount is determined based on this phase difference data DF, and the position of the photographing lens is moved based on the defocus amount. In this way, the camera can realize autofocus.

FIG. 4 shows a configuration example of the pixel block 100Gr. FIG. 5 shows a configuration example of the pixel block 100R. FIG. 6 shows a wiring example of the pixel blocks 100R, 100Gr, 100Gb and 100B. In addition, in FIG. 6, for convenience of explanation, the plurality of pixel blocks 100 are drawn apart from each other.

The pixel array 11 has multiple control lines TRGL, multiple control lines RSTL, multiple control lines SELL, and multiple signal lines VSL. The control line TRGL extends in the X direction (horizontal direction in FIGS. 4 to 6) and has one end connected to the driving section 12 . A control signal STRG is supplied from the driving section 12 to the control line TRGL. The control line RSTL extends in the X direction and has one end connected to the driving section 12 . A control signal SRST is supplied from the driving section 12 to the control line RSTL. The control line SELL extends in the X direction and has one end connected to the drive unit 12 . A control signal SSEL is supplied from the drive unit 12 to the control line SELL. The signal line VSL extends in the Y direction (vertical direction in FIGS. 4 to 6) and has one end connected to the reading section 20 . The signal line VSL transmits the signal SIG generated by the light receiving pixel P to the reading unit 20 .

The pixel block 100Gr (FIG. 4) has ten photodiodes PD, ten transistors TRG, a floating diffusion FD, and transistors RST, AMP, and SEL. Ten photodiodes PD and ten transistors TRG respectively correspond to ten light-receiving pixels PGr included in the pixel block 100Gr. The transistors TRG, RST, AMP, and SEL are N-type MOS (Metal Oxide Semiconductor) transistors in this example.

The photodiode PD is a photoelectric conversion element that generates an amount of charge corresponding to the amount of light received and accumulates the generated charge inside. The photodiode PD has an anode grounded and a cathode connected to the source of the transistor TRG.

The transistor TRG has a gate connected to the control line TRGL, a source connected to the cathode of the photodiode PD, and a drain connected to the floating diffusion FD. Gates of ten transistors TRG are connected to different control lines TRGL among ten control lines TRGL (control lines TRGL1 to TRGL6 and TRGL9 to TRGL12 in this example).

Floating diffusion FD is configured to accumulate charges transferred from photodiode PD via transistor TRG. The floating diffusion FD is configured using, for example, a diffusion layer formed on the surface of the semiconductor substrate. In FIG. 4, the floating diffusion FD is shown using a capacitive element symbol.

The transistor RST has a gate connected to the control line RSTL, a drain supplied with the power supply voltage VDD, and a source connected to the floating diffusion FD.

The transistor AMP has a gate connected to the floating diffusion FD, a drain supplied with the power supply voltage VDDH, and a source connected to the drain of the transistor SEL.

The transistor SEL has a gate connected to the control line SELL, a drain connected to the source of the transistor AMP, and a source connected to the signal line VSL.

With this configuration, in the light-receiving pixel P, the charges accumulated in the photodiode PD are discharged by turning on the transistors TRG and RST based on the control signals STRG and SRST, for example. When these transistors TRG and RST are turned off, the exposure period T is started, and an amount of charge corresponding to the amount of light received is accumulated in the photodiode PD. After the exposure period T ends, the light receiving pixel P outputs the signal SIG including the reset voltage Vreset and the pixel voltage Vpix to the signal line VSL. Specifically, first, the light-receiving pixel P is electrically connected to the signal line VSL by turning on the transistor SEL based on the control signal SSEL. Thereby, the transistor AMP is connected to a constant current source 21 (described later) of the reading section 20 and operates as a so-called source follower. As will be described later, the light-receiving pixel P has a voltage of the floating diffusion FD during a P-phase (pre-charge phase) period TP after the voltage of the floating diffusion FD is reset by turning on the transistor RST. A voltage corresponding to the voltage is output as a reset voltage Vreset. In addition, in the light receiving pixel P, during the D-phase (data phase) period TD after the charge is transferred from the photodiode PD to the floating diffusion FD by turning on the transistor TRG, The resulting voltage is output as the pixel voltage Vpix. A difference voltage between the pixel voltage Vpix and the reset voltage Vreset corresponds to the amount of light received by the light receiving pixel P during the exposure period T. FIG. In this manner, the light-receiving pixel P outputs the signal SIG including the reset voltage Vreset and the pixel voltage Vpix to the signal line VSL.

The pixel block 100R (FIG. 5) has eight photodiodes PD, eight transistors TRG, a floating diffusion FD, and transistors RST, AMP, and SEL. The eight photodiodes PD and the eight transistors TRG respectively correspond to the eight light receiving pixels PR included in the pixel block 100R. The gates of the eight transistors TRG are connected to different control lines TRGL among the eight control lines TRGL (control lines TRGL1, TRGL2, TRGL5 to TRGL10 in this example).

As shown in FIG. 6, the pixel blocks 100Gr and 100R belonging to the same row arranged in the X direction are connected to a plurality of control lines TRGL among the same 12 control lines TRGL (control lines TRGL1 to TRGL12). . In this example, the control lines TRGL1 to TRGL12 are arranged in this order from bottom to top in FIG. The pixel block 100Gr is connected to 10 control lines TRGL (control lines TRGL1 to TRGL6, TRGL9 to TRGL12) out of 12 control lines TRGL (control lines TRGL1 to TRGL12), and the pixel block 100R is connected to the 12 control lines TRGL (control lines TRGL1 to TRGL6, TRGL9 to TRGL12). It is connected to eight control lines TRGL (control lines TRGL1, TRGL2, TRGL5 to TRGL10) out of the control lines TRGL (control lines TRGL1 to TRGL12).

Although not shown, the pixel blocks 100Gr and 100R belonging to the same row and arranged in the X direction are connected to one control line RSTL and one control line SELL.

Further, as shown in FIG. 6, the pixel blocks 100Gr belonging to the same column arranged in the Y direction are connected to one signal line VSL. Similarly, pixel blocks 100R belonging to the same column arranged in the Y direction are connected to one signal line VSL.

The pixel block 100B, like the pixel block 100R (FIG. 5), has eight photodiodes PD, eight transistors TRG, a floating diffusion FD, and transistors RST, AMP, and SEL. Eight photodiodes PD and eight transistors TRG respectively correspond to eight light receiving pixels PB included in the pixel block 100B. Gates of the eight transistors TRG are connected to different control lines TRGL among the eight control lines TRGL.

The pixel block 100Gb, like the pixel block 100Gr (FIG. 4), has ten photodiodes PD, ten transistors TRG, a floating diffusion FD, and transistors RST, AMP, and SEL. Ten photodiodes PD and ten transistors TRG respectively correspond to ten light-receiving pixels PGb included in the pixel block 100Gb. Gates of the ten transistors TRG are connected to different control lines TRGL among the ten control lines TRGL.

As shown in FIG. 6, the pixel blocks 100B and 100Gb belonging to the same row arranged in the X direction are connected to a plurality of control lines TRGL out of the same 12 control lines TRGL. Although not shown, the pixel blocks 100B and 100Gb belonging to the same row and arranged in the X direction are connected to one control line RSTL and one control line SELL. Further, as shown in FIG. 6, the pixel blocks 100B belonging to the same column arranged in the Y direction are connected to one signal line VSL. Similarly, the pixel blocks 100Gb belonging to the same column arranged in the Y direction are connected to one signal line VSL.

The drive unit 12 ( FIG. 1 ) is configured to drive the plurality of light receiving pixels P in the pixel array 11 based on instructions from the imaging control unit 18 . Specifically, the driving unit 12 supplies a plurality of control signals STRG to the plurality of control lines TRGL in the pixel array 11, supplies a plurality of control signals SRST to the plurality of control lines RSTL, and supplies a plurality of control signals SRST to the plurality of control lines. A plurality of light-receiving pixels P in the pixel array 11 are driven by supplying a plurality of control signals SSEL to the SELL.

The reference signal generator 13 is configured to generate the reference signal RAMP based on the instruction from the imaging controller 18 . The reference signal RAMP has a so-called ramp waveform in which the voltage level gradually changes over time during the period (the P-phase period TP and the D-phase period TD) in which the reading unit 20 performs AD conversion. The reference signal generation unit 13 supplies such a reference signal RAMP to the reading unit 20 .

The reading unit 20 is configured to generate the image signal Spic0 by performing AD conversion based on the signal SIG supplied from the pixel array 11 via the signal line VSL based on the instruction from the imaging control unit 18. be done.

FIG. 7 shows a configuration example of the reading unit 20. As shown in FIG. In addition to the reading unit 20, FIG. 6 also shows the reference signal generating unit 13, the signal processing unit 15, and the imaging control unit 18. As shown in FIG. The reading unit 20 has a plurality of constant current sources 21 , a plurality of AD (Analog to Digital) conversion units ADC, and a transfer control unit 27 . The plurality of constant current sources 21 and the plurality of AD converters ADC are provided corresponding to the plurality of signal lines VSL, respectively. The constant current source 21 and AD converter ADC corresponding to one signal line VSL will be described below.

The constant current source 21 is configured to apply a predetermined current to the corresponding signal line VSL. One end of the constant current source 21 is connected to the corresponding signal line VSL, and the other end is grounded.

The AD conversion unit ADC is configured to perform AD conversion based on the signal SIG on the corresponding signal line VSL. The AD converter ADC has capacitive elements 22 and 23 , a comparator circuit 24 , a counter 25 and a latch 26 .

One end of the capacitive element 22 is connected to the signal line VSL and supplied with the signal SIG, and the other end is connected to the comparison circuit 24 . A reference signal RAMP supplied from the reference signal generation unit 13 is supplied to one end of the capacitive element 23 , and the other end is connected to the comparison circuit 24 .

The comparison circuit 24 performs a comparison operation based on the signal SIG supplied from the light receiving pixel P via the signal line VSL and the capacitive element 22 and the reference signal RAMP supplied from the reference signal generation section 13 via the capacitive element 23. is configured to generate the signal CP by performing The comparison circuit 24 sets the operating point by setting the voltages of the capacitive elements 22 and 23 based on the control signal AZ supplied from the imaging control section 18 . After that, the comparison circuit 24 performs a comparison operation to compare the reset voltage Vreset included in the signal SIG with the voltage of the reference signal RAMP in the P-phase period TP, and performs a comparison operation of A comparison operation is performed to compare the pixel voltage Vpix and the voltage of the reference signal RAMP.

The counter 25 is configured to perform a counting operation of counting the pulses of the clock signal CLK supplied from the imaging control section 18 based on the signal CP supplied from the comparison circuit 24 . Specifically, the counter 25 generates the count value CNTP by counting the pulses of the clock signal CLK until the signal CP transitions in the P-phase period TP, and converts the count value CNTP into a digital signal having a plurality of bits. Output as code. Further, the counter 25 generates a count value CNTD by counting the pulses of the clock signal CLK until the signal CP transitions in the D-phase period TD, and outputs the count value CNTD as a digital code having a plurality of bits. It is designed to

The latch 26 is configured to temporarily hold the digital code supplied from the counter 25 and to output the digital code to the bus wiring BUS based on the instruction from the transfer control section 27 .

The transfer control unit 27 is configured to control the latches 26 of the plurality of AD conversion units ADC to sequentially output the digital code to the bus wiring BUS based on the control signal CTL supplied from the imaging control unit 18. be. The reading unit 20 uses the bus wiring BUS to sequentially transfer the plurality of digital codes supplied from the plurality of AD conversion units ADC to the signal processing unit 15 as the image signal Spic0.

The signal processing unit 15 (FIG. 1) is configured to perform predetermined signal processing based on the image signal Spic0 and an instruction from the imaging control unit 18 to generate the image signal Spic. The signal processor 15 has an image data generator 16 and a phase difference data generator 17 . The image data generator 16 is configured to generate image data DP representing a captured image by performing predetermined image processing based on the image signal Spic0. The phase difference data generation unit 17 is configured to generate phase difference data DF indicating an image plane phase difference by performing predetermined image processing based on the image signal Spic0. The signal processing unit 15 generates an image signal Spic including the image data DP generated by the image data generation unit 16 and the phase difference data DF generated by the phase difference data generation unit 17 .

FIG. 8 shows an example of the image signal Spic. The signal processing unit 15 generates the image signal Spic by, for example, alternately arranging the image data DP related to the light-receiving pixels P in multiple rows and the phase difference data DF related to the light-receiving pixels P in multiple rows. Then, the signal processing section 15 outputs such an image signal Spic.

The imaging control unit 18 supplies control signals to the driving unit 12, the reference signal generating unit 13, the reading unit 20, and the signal processing unit 15, and controls the operation of these circuits, thereby controlling the operation of the imaging device 1. configured to A control signal Sctl is supplied to the imaging control unit 18 from the outside. This control signal Sctl includes, for example, information about the zoom magnification of so-called electronic zoom. The imaging control unit 18 controls the operation of the imaging device 1 based on the control signal Sctl.

Here, the light-receiving pixel P corresponds to a specific example of the "light-receiving pixel" in the present disclosure. Pixel pair 90 corresponds to a specific example of "pixel pair" in the present disclosure. Pixel block 100 corresponds to a specific example of "pixel block" in the present disclosure. For example, the pixel block 100Gr corresponds to a specific example of "first pixel block" in the present disclosure. For example, the pixel block 100R corresponds to a specific example of "second pixel block" in the present disclosure. The lens 101 corresponds to a specific example of "lens" in the present disclosure. The control line TRGL corresponds to a specific example of "control line" in the present disclosure. The insulating layer 53 corresponds to a specific example of "insulating layer" in the present disclosure.

1.2 Schematic Operation Example Next, a schematic operation example of the imaging apparatus 1 according to the present embodiment will be described with reference to FIGS. 1 and 7. FIG. The drive unit 12 sequentially drives the plurality of light receiving pixels P in the pixel array 11 based on instructions from the imaging control unit 18 . The reference signal generator 13 generates the reference signal RAMP based on the instruction from the imaging controller 18 . The light-receiving pixel P outputs the reset voltage Vreset as the signal SIG during the P-phase period TP, and outputs the pixel voltage Vpix corresponding to the amount of received light as the signal SIG during the D-phase period TD. The reading unit 20 generates the image signal Spic0 based on the signal SIG supplied from the pixel array 11 via the signal line VSL and the instruction from the imaging control unit 18 . In the signal processing unit 15, the image data generation unit 16 performs predetermined image processing based on the image signal Spic0 to generate image data DP representing the captured image, and the phase difference data generation unit 17 generates the image signal Predetermined image processing is performed based on Spic0 to generate phase difference data DF representing the image plane phase difference. Then, the signal processing unit 15 generates an image signal Spic including the image data DP and the phase difference data DF. The imaging control unit 18 supplies control signals to the driving unit 12, the reference signal generating unit 13, the reading unit 20, and the signal processing unit 15, and controls the operation of these circuits, thereby controlling the operation of the imaging device 1. do.

Here, the imaging control unit 18 controls the operation of the imaging device 1 based on the control signal Sctl including information about the zoom magnification of the electronic zoom. A zoom operation in the imaging device 1 will be described below.

FIG. 9 shows an example of the number of light-receiving pixels P (the number of effective pixels) of a captured image when the zoom magnification is changed from 1× to 10×. In FIG. 9, the solid line indicates the number of effective pixels of the imaging device 1. In FIG. 10A and 10B show an example of the zoom operation in the imaging apparatus 1, where (A) shows the operation when the zoom magnification is 1×, and (B) shows the operation when the zoom magnification is 2×. and (C) shows the operation when the zoom magnification is 3 times.

The imaging device 1 has three imaging modes M (imaging modes MA, MB, MC). The imaging control unit 18 selects one of the three imaging modes MA to MC based on the information about the zoom magnification included in the control signal Sctl. Specifically, as shown in FIG. 9, the imaging control unit 18 selects the imaging mode MA when the zoom magnification is less than 2, and selects the imaging mode MA when the zoom magnification is 2 or more and less than 3. MB is selected, and when the zoom magnification is 3 or more, imaging mode MC is selected.

In the imaging mode MA, as shown in FIG. 10A, the imaging device 1 obtains four pixel values V (pixel values VR, VGr, VGb, VB) in each of the plurality of unit units U. In this way, the imaging device 1 generates the image data DP by generating the pixel values V at a ratio of 4 pixels to 36 pixels P. FIG. When the number of light receiving pixels P in the pixel array 11 is 108 [Mpix], pixel values V for 12 [Mpix] are calculated. As a result, the number of effective pixels is 12 [Mpix], as shown in FIG.

As shown in FIG. 9, when the zoom magnification is increased from 1 in this imaging mode MA, the number of effective pixels decreases according to the magnification. Then, when the zoom magnification becomes 2, the imaging mode M becomes the imaging mode MB.

In the imaging mode MB, the imaging device 1 obtains 16 pixel values V in each of the plurality of unit units U, as shown in FIG. 10B. In this manner, the imaging device 1 generates the image data DP by generating the pixel values V at a ratio of 16 pixels to 36 light receiving pixels P. FIG. When the number of light receiving pixels P in the pixel array 11 is 108 [Mpix], pixel values V for 48 [Mpix] are calculated. Actually, since the zoom magnification is 2 times, the imaging range is narrowed to 1/4 as shown in FIG. ).

As shown in FIG. 9, when the zoom magnification is increased from 2 in this imaging mode MB, the number of effective pixels decreases according to the magnification. Then, when the zoom magnification becomes 3, the imaging mode M becomes the imaging mode MC.

In the imaging mode MC, the imaging device 1 obtains 36 pixel values V in each of the plurality of unit units U, as shown in FIG. 10(C). In this manner, the imaging device 1 generates the image data DP by generating the pixel values V at a ratio of 36 pixels to the 36 light receiving pixels P. FIG. When the number of light receiving pixels P in the pixel array 11 is 108 [Mpix], a captured image of 108 [Mpix] can be obtained. Actually, since the zoom magnification is 3 times, the imaging range is narrowed to 1/9 as shown in FIG. ).

As described above, since the imaging apparatus 1 is provided with three imaging modes M, it is possible to reduce the change in image quality of the captured image when the zoom magnification is changed. That is, for example, two imaging modes MA and MC are provided by omitting the imaging mode MB, and the imaging mode MA is selected when the zoom magnification is less than 2. When MC is selected, the number of effective pixels greatly changes as indicated by the dashed line in FIG. That is, in this example, when the zoom magnification is 2×, the imaging mode MC is selected and the number of effective pixels is 27 [Mpix] (=108 [Mpix]/4). Therefore, there is a large difference between the number of effective pixels when the zoom magnification is, for example, 1.9 times and the number of effective pixels when the zoom magnification is 2 times. can change significantly. On the other hand, since the image pickup apparatus 1 is provided with three image pickup modes M, it is possible to reduce the change in the number of effective pixels when the zoom magnification is changed. can.

1.3 Generation of Image Plane Phase Difference Information (Phase Difference Data DF) In the imaging apparatus 1 having the configuration and operation described above, as described above, the phase difference information indicating the image plane phase difference for realizing autofocus is generated. The data FD is generated by the phase difference data generator 17 within the signal processor 15 .

Here, as described above, pixel values after performing sensitivity difference correction are conventionally used to generate image plane phase difference information in an imaging apparatus that performs autofocus based on the image plane phase difference. was In recent years, however, pixel array patterns have become more complicated as pixels have become finer, and sensitivity difference correction processing has also become more complicated. There was a problem of getting stuck.

In addition, as illustrated in FIG. 2, the pixel blocks 100R, 100Gr, 100Gb, and 100B that constitute the unit U, which is the minimum unit of repetition, are respectively composed of eight or ten light-receiving pixels P, so-called DecaOcta array imaging. As described above, the apparatus 1 has, for example, an imaging mode MA (see FIG. 10A) in which each pixel block 100R, 100Gr, 100Gb, and 100B is read out as one pixel, and an imaging mode in which all effective pixels P are individually read out. It operates by switching between three stages of binning modes, mode MC (see FIG. 10C) and imaging mode MB (see FIG. 10B), which reads at an intermediate resolution between imaging mode MA and imaging mode MC. However, it is difficult to apply the conventional autofocus technique using the image plane phase difference as it is to the image pickup apparatus 1 having such a plurality of stages of binning modes.

Furthermore, the imaging device 1 with the DecaOcta arrangement also supports an HDR (High Dynamic Range) mode, in which the dynamic range for luminance of some or all of the light receiving pixels P is wider than in a normal operation mode (SDR mode described later). However, even in that case, it was difficult to apply the conventional autofocus technique using the image plane phase difference as it is.

Therefore, in the present embodiment, the pixel array pattern of the image signal Spic0 input to the sensitivity difference correction process in the phase difference data generation unit 17 is simplified. Thereby, it is possible to suppress complication of sensitivity difference correction processing in the phase difference data generation unit 17 . More specifically, by simplifying the pixel array pattern of the image signal Spic0, it is possible to reduce the number of parallel channels for the sensitivity difference correction processing for phase difference detection. It is possible to reduce the memory capacity for the difference correction coefficients. As a result, even if the pixel array pattern is complicated, it is possible to suppress an increase in circuit scale and memory capacity, thereby suppressing an increase in device size.

Further, in the present embodiment, even if a multi-step binning mode is provided, or if it is possible to switch between an SDR (Standard Dynamic Range) mode using a normal dynamic range and an HDR mode, in other words, the normal dynamic range Even when there are light-receiving pixels P operating with a sensitivity of 0 and light-receiving pixels P operating with a sensitivity of a dynamic range wider than the normal dynamic range, the pixel arrangement pattern of the image signal Spic0 input to the sensitivity difference correction process is can be uniformly converted into a simple array pattern, so there is also the advantage of being able to apply the lightened sensitivity difference correction processing according to the present embodiment to almost all modes. be done.

1.3.1 More Detailed Configuration Example of Phase Difference Data Generation Unit FIG. 11 is a block diagram showing a more detailed configuration example of the phase difference data generation unit according to this embodiment. As shown in FIG. 11 , the phase difference data generator 17 includes a same color pixel adder 171 , a sensitivity difference corrector 172 , a phase difference information generator 173 , a phase difference detector 174 and a memory 175 . Of these, the sensitivity difference correction unit 172 and the phase difference information generation unit 173, as will be described later, perform the left and right and the right and left based on a plurality of pixel values included in the image signal Spic1 after the same color pixel addition processing by the same color pixel addition unit 171. / Or it functions as a generation unit that generates information (for example, phase difference information or phase difference data DF to be described later) regarding the image plane phase difference between the upper and lower light receiving pixels. This generator may include the phase difference detector 174 .

(Same color pixel adder 171)
The same-color pixel adder 171 applies the digital image signal Spic0 read from the image array 11 and input to the phase difference data generator 17 to the left side of the pixel pair 90 for each of the pixel blocks 100R, 100Gr, 100Gb, and 100B. pixel addition processing for summing the pixel values of the light receiving pixels P (hereinafter also referred to as left pixels) located in the pixel addition processing for summing the pixel values of the light receiving pixels P located on the right side (hereinafter also referred to as right pixels). to run. As a result, the pixel blocks 100R, 100Gr, 100Gb, and 100B are output from the same-color pixel addition unit 171 as the left total pixel value (hereinafter also referred to as the left pixel total value) and the right total pixel value (hereinafter also referred to as the right pixel total value). ) is output as an image signal Spic1 composed of a total pixel value pair.

In this way, by converting the input image signal Spic0 into the image signal Spic1 for each of the left and right pixels for each of the pixel blocks 100R, 100Gr, 100Gb, and 100B, regardless of the resolution of the image signal Spic0 (with or without binning), In other words, it is possible to unify the resolution of the image signal Spic1 input to the sensitivity difference correction unit 172 without depending on the mode being executed by the imaging device 1. It becomes possible to apply the sensitivity difference correction processing related to

In addition, the pixel blocks 100R, 100Gr, 100Gb, and 100B have the resolution of the image signal Spic1 input to the sensitivity difference corrector 172 independently of the number of pixels of the pixel blocks 100R, 100Gr, 100Gb, and 100B in the image signal Spic0. Since it is possible to reduce the resolution to the minimum unit, it is possible to reduce the circuit scale by reducing the number of parallel channels for the sensitivity difference correction processing, and to reduce the capacity of the memory 175 that stores the sensitivity difference correction coefficient table. is possible, and as a result, it is possible to suppress an increase in device size.

Furthermore, since it is possible to output the image signal Spic1 with a uniform resolution without depending on the pixel arrangement of the input image signal Spic0, each pixel block 100R, 100Gr, 100Gb, Imaging devices with any pixel array pattern, such as non-rectangular pixel array patterns such as squares and rectangles for 100B, and pixel array patterns in which the number of pixels constituting each pixel block 100R, 100Gr, 100Gb, and 100B are not uniform. It is possible to apply this embodiment.

In imaging mode MA, a set of left pixel total value and right pixel total value is read out for each of the pixel blocks 100R, 100Gr, 100Gb, and 100B. The image signal Spic0 input to 15 is composed of the total pixel value of the right pixels P and the total pixel value of the left pixels P for each of the pixel blocks 100R, 100Gr, 100Gb, and 100B. Therefore, in the case of the imaging mode MA, the same color pixel adder 171 may be bypassed and the image signal Spic0 may be directly input to the sensitivity difference corrector 172 .

Further, when operating in the HDR mode, light-receiving pixels P operating in a low dynamic range (hereinafter also referred to as low-brightness pixels) and light-receiving pixels P operating in a high dynamic range or low dynamic range (hereinafter referred to as high In that case, the left pixel total value and right pixel total value of the low luminance pixels P and the left pixel A sum value and a right pixel sum value may be determined.

(Sensitivity difference correction unit 172)
The sensitivity difference correction unit 172 acquires a sensitivity difference correction coefficient table from the memory 175, and converts the coefficients managed by this table into the left pixel total value and the right By multiplying the pixel total value by the pixel total value, the difference in luminance value (pixel value) between the left pixel total value and the right pixel total value due to the sensitivity difference between the left pixel and the right pixel is corrected.

In the sensitivity difference correction coefficient table, for example, pairs of sensitivity difference correction coefficients corresponding to pixel blocks 100R, 100Gr, 100Gb, and 100B constituting the pixel array 11 one-to-one correspond to the pixel blocks 100R, 100R, and 100B on the pixel array 11, respectively. It may have a table structure arranged in a matrix according to the arrangement of 100Gr, 100Gb, and 100B. In addition, the sensitivity difference correction coefficient for correcting the left pixel total value (hereinafter also referred to as the left sensitivity difference correction coefficient) and the sensitivity difference correction coefficient for correcting the right pixel total value (hereinafter also referred to as the right sensitivity difference correction coefficient) A value obtained in advance by experience, simulation, experiment, or the like may be used as the value.

Furthermore, in the present embodiment, since same-color pixel addition processing is performed before sensitivity difference correction processing, in all imaging modes MA, MB, and MC, in other words, in all multi-step binning modes, It is possible to use the same sensitivity difference correction coefficient table.

Furthermore, even when it is possible to switch between the SDR mode and the HDR mode, by executing the same-color pixel addition process before the sensitivity difference correction process, the left pixel total value and the right pixel total value of the high-luminance pixels P It is possible to reduce the residual sensitivity difference when using the same sensitivity difference correction coefficient table as the sensitivity difference correction coefficient table for the low-luminance pixel P. This suggests that even if the same sensitivity difference correction coefficient table is used in SDR mode and HDR mode, operation can be performed without problems.

(Phase difference information generator 173)
The phase difference information generation unit 173 calculates the total value or average value of the left pixel total value and the right pixel total value for each unit unit U with respect to the image signal Spic1 after the sensitivity difference correction. A pair (hereinafter also referred to as a luminance pair) of luminance information of the left pixel P (hereinafter also referred to as left luminance information) and luminance information of the right pixel P (hereinafter also referred to as right luminance information) for each U is generated. Then, the phase difference information generation unit 173 generates phase difference information by obtaining a ratio, a difference, or the like between the generated left luminance information and right luminance information.

When operating in the HDR mode, the phase difference information generation unit 173 generates left luminance information (also referred to as left low luminance information) of the light receiving pixel P, which is a left pixel and is a low luminance pixel, for each unit unit U. , right luminance information (also referred to as right low luminance information) of the light receiving pixel P that is a right pixel and a low luminance pixel, and left luminance information (also referred to as left high luminance information) of the light receiving pixel P that is a left pixel and a high luminance pixel. ) and right luminance information (also referred to as right high luminance information) of the light-receiving pixel P, which is a right pixel and a high luminance pixel. Subsequently, the phase difference information generation unit 173 generates left luminance information by adding a value obtained by multiplying the left low luminance information by a predetermined coefficient to the left high luminance information. Similarly, the phase difference information generation unit 173 generates right luminance information by adding a value obtained by multiplying the right low luminance information by a predetermined coefficient to the right high luminance information. Then, the phase difference information generation unit 173 generates phase difference information by obtaining a ratio, a difference, or the like between the generated left luminance information and right luminance information.

Note that when white balance adjustment is performed on the image signal, this white balance adjustment processing may be performed within the phase difference information generation unit 173, or may be performed at a processing stage prior to the phase difference information generation unit 173. may For example, a processing unit for white balance adjustment may be added to the output stage of the sensitivity difference correction unit 172 . Alternatively, the white balance adjustment may be performed before the phase difference information generator 173 obtains the phase difference information. In addition, the phase difference information generation unit 173 is not limited to the white balance adjustment purpose, and for various adjustment purposes, the left pixel total value and the right pixel total value for each of the pixel blocks 100R, 100Gr, 100Gb, and 100B are given predetermined coefficients. The left luminance information and the right luminance information may be generated by multiplying and summing them.

(Phase difference detector 174)
The phase difference detector 174 generates and outputs phase difference data DF based on the phase difference information output from the phase difference information generator 173 . Note that the phase difference detector 174 may be omitted when the phase difference information is used as the phase difference data DF as it is.

(Memory 175)
The memory 175 stores the sensitivity difference correction coefficient table as described above. The memory 175 may be provided as a dedicated memory for the phase difference data generator 17 or may be provided as a shared memory with the image data generator 16 .

1.3.2 Concrete Example of Same-Color Pixel Addition Processing in Imaging Mode MC Next, the operation of the same-color pixel addition section 171 according to the present embodiment in each imaging mode will be described with specific examples. FIG. 12 is a diagram of one unit U extracted from the arrangement example of the light receiving elements P in the pixel array 11 shown in FIG. 13 to 16 are diagrams for explaining the same-color pixel addition process performed on pixel values read out from each pixel block. FIG. 17 is a diagram showing the left pixel total value and the right pixel total value finally obtained based on the pixel values read out from the unit U. In FIG. In addition, below, the case where the imaging mode is the imaging mode MC will be described as an example. Further, in this specific example, the case where the arrangement pattern of the light receiving pixels P is the DecaOcta arrangement is illustrated.

As shown in FIG. 12, one unit unit U includes a pixel block 100Gr made up of ten light receiving elements P, a pixel block 100R made up of eight light receiving elements P, a pixel block 100B made up of eight light receiving elements P, It is composed of a pixel block 100Gb consisting of ten light receiving elements P. As shown in FIG.

Here, in the pixel block 100Gr, GrL1 to GrL5 indicate the pixel values VGr read from the left pixel and each left pixel in the pixel pair 90, and GrR1 to GrR5 indicate the pixel values read from the right pixel and each right pixel. Let VGr denote VGr. Similarly, in the pixel block 100R, RL1 to RL4 indicate the left pixel and the pixel value VR read from each left pixel, and RR1 to RR4 indicate the right pixel and the pixel value VR read from each right pixel. In the pixel block 100B, BL1 to BL4 indicate the left pixel and the pixel value VB read from each left pixel, and BR1 to BR4 indicate the right pixel and the pixel value VB read from each right pixel. , in the pixel block 100Gb, GbL1 to GbL5 indicate the left pixel and the pixel value VGb read from each left pixel, and GbR1 to GbR5 indicate the right pixel and the pixel value VGb read from each right pixel. .

As shown in FIG. 13, the same-color pixel addition unit 171 adds pixel values GrL1, GrL2, GrL3, GrL4, and GrL5 read from the left pixel P among the pixel values VGr read from the pixel block 100Gr. Thus, the left pixel total value GrL of the pixel block 100Gr is calculated. Similarly, the same-color pixel addition unit 171 adds the pixel values GrR1, GrR2, GrR3, GrR4, and GrR5 read from the right pixel P among the pixel values VGr read from the pixel block 100Gr to obtain pixel A right pixel total value GrR of the block 100Gr is calculated.

As shown in FIGS. 14 to 16, pixel values RL1 to RL4, BL1 to BL4, or GbL1 to GbL5 read from left pixels P are similarly applied to other pixel blocks 100R, 100B, and 100Gb. By adding, the left pixel total values RL, BL, and GbL of the pixel blocks 100R, 100B, and 100Gb are calculated, and the pixel values RR1 to RR4, BR1 to BR4, or GbR1 to GbR5 read from the right pixels P are calculated. By adding, the right pixel total values RR, BR, and GbR of the pixel blocks 100R, 100B, and 100Gb are calculated.

As a result, as shown in FIG. 17, based on the pixel values VGr, VR, VB, and VGb read out from the unit unit U, the same-color pixel addition unit 171 outputs the left pixel total value GrL and the right pixel value GrL to the pixel block 100Gr. The total pixel value pair 111Gr with the total value GrR is arranged, the total pixel value pair 111R with the left pixel total value RL and the right pixel total value RR is arranged in the pixel block 100R, and the left pixel total value BL and the total pixel value RR are arranged in the pixel block 100B. An image signal Spic1 in which the total pixel value pair 111B with the right pixel total value BR is arranged and the total pixel value pair 111Gb with the left pixel total value GbL and the right pixel total value GbR is arranged in the pixel block 100Gb is output.

Note that the same-color pixel addition processing in the imaging mode MB can be easily performed from the above description by replacing the 36 pixel values V read from one unit unit U in the imaging mode MC with 16 pixel values. detailed description is omitted here. In imaging mode MA, the number of pixel values V read out from one unit unit U is four. Since the data is read out directly from the array unit 11, the same-color pixel addition processing may be bypassed.

1.3.3 Specific Example of Same-Color Pixel Addition Processing in HDR Mode Next, the operation in HDR mode of the same-color pixel adder 171 according to this embodiment will be described with a specific example. FIG. 18 is a diagram of one unit U extracted from the arrangement example of the light receiving elements P in the pixel array 11 shown in FIG. 19 to 26 are diagrams for explaining the same-color pixel addition process performed on the pixel values read out from each pixel block. FIG. 27 is a diagram showing left pixel total values and right pixel total values of low-luminance pixels and high-luminance pixels, respectively, which are finally obtained based on the pixel values read out from the unit U. FIG. In this specific example, the arrangement pattern of the light-receiving pixels P is the DecaOcta arrangement.

In the example shown in FIG. 18, in the pixel block 100Gr, GrL1L to GrL3L indicate the left pixel and the pixel value VGr read from each left pixel in the pixel pair 90 of the low-luminance pixel PL, and GrR1L to GrR3L indicate the low-luminance pixel PL. and GrL1H to GrL2H represent the pixel values VGr read from the left pixel and each left pixel in the pixel pair 90 of the high luminance pixel PH. and GrR1H to GrR2H denote the right pixel and the pixel value VGr read from each right pixel in the pixel pair 90 of the high luminance pixel PH.

Similarly, in the pixel block 100R, RL1L to RL2L indicate the left pixel in the pixel pair 90 of the low-luminance pixel PL and the pixel value VR read from each left pixel, and RR1L to RR2L indicate the pixel pair 90 of the low-luminance pixel PL. , and RL1H to RL2H indicate the pixel values VR read from the left pixel and each left pixel in the pixel pair 90 of the high-brightness pixel PH, RR1H to RR2H denote the right pixel in the pixel pair 90 of the high luminance pixel PH and the pixel value VR read from each right pixel.

Similarly, in the pixel block 100B, BL1L to BL2L indicate the left pixel in the pixel pair 90 of the low-luminance pixel PL and the pixel value VB read from each left pixel, and BR1L to BR2L indicate the pixel pair 90 of the low-luminance pixel PL. shows the pixel value VB read from the right pixel and each right pixel in , and BL1H to BL2H show the pixel value VB read from the left pixel and each left pixel in the pixel pair 90 of the high luminance pixel PH, BR1H to BR2H denote the right pixel in the pixel pair 90 of the high luminance pixel PH and the pixel value VB read from each right pixel.

Similarly, in the pixel block 100Gb, GbL1L to GbL3L indicate the left pixel in the pixel pair 90 of the low-luminance pixel PL and the pixel value VGb read from each left pixel, and GbR1L to GbR3L indicate the pixel pair 90 of the low-luminance pixel PL. GbL1H to GbL2H indicate the pixel values VGb read from the right pixel and each right pixel in the pixel pair 90 of the high luminance pixel PH, and GbR1H to GbR2H indicates the pixel value VGb read from the right pixel and each right pixel in the pixel pair 90 of the high luminance pixel PH.

As shown in FIG. 19, the same-color pixel addition unit 171 adds the pixel values GrL1H and GrL2H read from the left pixel P among the pixel values VGr read from the high-brightness pixels PH in the pixel block 100Gr. , the left pixel total value GrLH of the high luminance pixels PH in the pixel block 100Gr is calculated. Similarly, the same-color pixel addition unit 171 adds the pixel values GrR1H and GrR2H read from the right pixel P, among the pixel values VGr read from the high-brightness pixels PH in the pixel block 100Gr, to obtain the pixel block 100Gr. The right pixel total value GrRH of the high luminance pixels PH in 100Gr is calculated.

Further, as shown in FIG. 20, the same-color pixel addition unit 171 adds the pixel values GrL1L, GrL2L, and GrL3L read from the left pixel P out of the pixel values VGr read from the low-luminance pixels PL in the pixel block 100Gr. is added to calculate the left pixel total value GrLL of the low luminance pixels PL in the pixel block 100Gr. Similarly, the same-color pixel addition unit 171 adds the pixel values GrR1L, GrR2L, and GrR3L read from the right pixel P among the pixel values VGr read from the low-luminance pixels PL in the pixel block 100Gr. A right pixel total value GrRL of the low luminance pixels PL in the pixel block 100Gr is calculated.

As shown in FIGS. 21 to 26, pixel values RL1H to RL2H and BL1H to BL2H read from the high luminance pixels PH of the left pixels P are similarly applied to the other pixel blocks 100R, 100B, and 100Gb. Alternatively, by adding GbL1H to GbL2H, the left pixel total values RLH, BLH, and GbLH of the high-luminance pixels PH in the pixel blocks 100R, 100B, and 100Gb are calculated, and read from the high-luminance pixel PH of the right pixels P. By adding the output pixel values RR1H to RR2H, BR1H to BR2H, or GbR1H to GbR3H, the right pixel total values RRH, BRH, and GbRH of the low-luminance pixels PL in the pixel blocks 100R, 100B, and 100Gb are calculated ( 21, 23 and 25). Similarly, by adding the pixel values RL1L to RL2L, BL1L to BL2L, or GbL1L to GbR3L read from the low-luminance pixels PL of the left pixels P, the low-luminance pixels in the pixel blocks 100R, 100B, and 100Gb By calculating the left pixel total values RLH, BLH, and GbLH of PL and adding the pixel values RR1L to RR2L, BR1L to BR2L, or GbR1L to GbR3L read from the low-luminance pixels PL of the right pixels P , right pixel total values RRL, BRL, and GbRL of low-luminance pixels PL in pixel blocks 100R, 100B, and 100Gb (see FIGS. 22, 24, and 26).

As a result, as shown in FIG. 27, based on the pixel values VGr, VR, VB, and VGb read out from the unit unit U, the pixel block 100Gr from the same-color pixel adder 171 outputs the left pixel total of the high-brightness pixel PH. A total pixel value pair 111GrH of the value GrLH and the right pixel total value GrRH, and a total pixel value pair 111GrL of the left pixel total value GrLL and the right pixel total value GrRL of the low-luminance pixel PL are arranged, and the high-luminance pixels are arranged in the pixel block 100R. A total pixel value pair 111RH of left pixel total value RLH and right pixel total value RRH of PH and a total pixel value pair 111RL of left pixel total value RLL and right pixel total value RRL of low luminance pixel PL are arranged in the pixel block 100B. A total pixel value pair 111BH of left pixel total value BLH and right pixel total value BRH of high luminance pixel PH and left pixel total value BLL and right pixel total value BRL of low luminance pixel PL are arranged in pixel block 100Gb. A total pixel value pair 111GbH of the left pixel total value GbLH and the right pixel total value GbRH of the high-luminance pixel PH and a total pixel value pair 111GbL of the left pixel total value GbLL and the right pixel total value GbRL of the low-luminance pixel PL are Arranged image signal Spic1 is output.

In the HDR mode, as described above, the phase difference information generating section 173 converts the left low-luminance information of the low-luminance pixel PL among the left pixels into the left pixel total values GrLL, RLL, The right low-luminance information of the low-luminance pixel PL of the right pixels is calculated from the right pixel total values GrRL, RRL, BRL and GbRL, and the left high-luminance information of the high-luminance pixel PH of the left pixels is calculated from BLL and GbLL. Information is calculated from the left pixel total values GrLH, RLH, BLH and GbLH, and the right high luminance information of the high luminance pixel PH among the right pixels is calculated from the right pixel total values GrRH, RRH, BRH and GbRH. Then, the phase difference information generation unit 173 generates left luminance information and right luminance information by adding a value obtained by multiplying the calculated luminance information by a coefficient as necessary, and generates left luminance information and right luminance information. Phase difference information is generated by obtaining a ratio, a difference, or the like with luminance information.

1.4 Actions and Effects As described above, in the present embodiment, the image signal Spic0 input to the phase difference data generation unit 17 is generated by one pixel on the left and right (total value of left pixels) for each of the pixel blocks 100R, 100Gr, 100Gb, and 100B and right pixel total value) is converted into an image signal Spic1. Thereby, the resolution of the image signal Spic1 input to the sensitivity difference correction unit 172 is unified regardless of the resolution of the image signal Spic0 (with or without binning), in other words, regardless of the mode in which the imaging apparatus 1 is executing. Therefore, the sensitivity difference correction processing according to this embodiment can be applied to substantially all modes.

In addition, the resolution of the image signal Spic1 input to the sensitivity difference correction unit 172 is set to the minimum pixel blocks 100R, 100Gr, 100Gb, and 100B, regardless of the number of pixels in each of the pixel blocks 100R, 100Gr, 100Gb, and 100B in the image signal Spic0. Since it is possible to reduce the resolution to a unit, it is possible to reduce the circuit scale by reducing the number of parallel channels for sensitivity difference correction processing and reduce the capacity of the memory 175 for storing the sensitivity difference correction coefficient table. As a result, it becomes possible to suppress an increase in device size.

Furthermore, since it is possible to output the image signal Spic1 with a uniform resolution without depending on the pixel arrangement of the input image signal Spic0, each pixel block 100R, 100Gr, 100Gb, Imaging devices with any pixel array pattern, such as non-rectangular pixel array patterns such as squares and rectangles for 100B, and pixel array patterns in which the number of pixels constituting each pixel block 100R, 100Gr, 100Gb, and 100B are not uniform. It is possible to apply this embodiment.

Furthermore, in this embodiment, not only when the binning mode is provided in a plurality of stages, but also when it is possible to switch between an SDR (Standard Dynamic Range) mode using a normal dynamic range and an HDR mode, sensitivity difference correction processing Since it is possible to uniformly convert the pixel arrangement pattern of the image signal Spic0 to be input to the Spic0 into a simple arrangement pattern, the lightened sensitivity difference correction processing according to this embodiment is applied to almost all modes. There is also the advantage of being able to

In this embodiment, the pixel pairs 90 in the pixel array 11 are all composed of two light-receiving pixels P adjacent to each other in the X direction. Part or all of 90 may be composed of two light receiving elements P adjacent to each other in the Y direction. When all the pixel pairs 90 are composed of the light receiving pixels P adjacent in the X direction, autofocusing based on the image plane phase difference in the horizontal direction becomes possible, and all the pixel pairs 90 are composed of the light receiving pixels P adjacent in the Y direction. When configured, autofocusing based on the vertical image plane phase difference is possible, and a part of the pixel pair 90 is composed of the light receiving pixels P adjacent in the X direction, and the rest is composed of the light receiving pixels P adjacent in the Y direction. When configured, autofocus based on horizontal and vertical image plane phase differences is possible.

1.5 Modifications In the above-described embodiment, a specific example of same-color pixel addition processing when the array pattern of the light-receiving pixels P is the DecaOcta array has been described. is not limited to this. In the following, therefore, several examples of same-color pixel addition processing in array patterns other than the DecaOcta array will be described.

1.5.1 First Modification In the first modification, each of the pixel blocks 100R, 100Gr, 100Gb, and 100B constituting the unit U is composed of a total of 16 light-receiving pixels P of 4×4 pixels. A case of the QQBC arrangement is illustrated.

FIG. 28 is a diagram of one unit U extracted from the arrangement example of the light receiving elements P in the pixel array 11 shown in FIG. 29 to 32 are diagrams for explaining the same-color pixel addition process performed on the pixel values read out from each pixel block. FIG. 33 is a diagram showing the left pixel total value and the right pixel total value finally obtained based on the pixel values read out from the unit U. In FIG. In addition, although the case where the image pickup mode is the image pickup mode MC will be exemplified below, the case where the image pickup mode is the image pickup mode MB is similarly applicable.

As shown in FIG. 28, in the unit unit U according to this modified example, each of the pixel blocks 100R, 100Gr, 100Gb, and 100B has a total of 16 light receiving pixels P arranged in 4×4 pixels. In such an array pattern, one on-chip lens 101 is arranged across a total of four light receiving pixels P arranged in a 2×2 matrix. Accordingly, each of the pixel blocks 100R, 100Gr, 100Gb, and 100B is provided with a total of four on-chip lenses 101 arranged in a 2×2 matrix.

In this way, of the four light receiving pixels P that share one on-chip lens 101, two light receiving pixels P that are adjacent in the row direction (X direction) can be used to acquire the image plane phase difference in the horizontal direction. Form a pixel pair 90X. On the other hand, two light-receiving pixels P that are adjacent in the row direction (Y direction) form a pixel pair 90Y that can be used to acquire the image plane phase difference in the vertical direction.

Here, as shown in FIGS. 29 to 32, in the pixel block 100Gr, GrL1u, GrL1d, GrL2u, GrL2d, GrL3u, GrL3d, GrL4u and GrL4d are the left pixels in the pixel pair 90X and the pixels read from the respective left pixels. Let GrR1u, GrR1d, GrR2u, GrR2d, GrR3u, GrR3d, GrR4u and GrR4d denote the right pixel in pixel pair 90X and the pixel value VGr read from each right pixel. Among them, GrL1u, GrR1u, GrL2u, GrR2u, GrL3u, GrR3u, GrL4u, and GrR4u are the pixel values read from the upper light-receiving pixel P (hereinafter also referred to as the upper pixel) and each upper pixel in the pixel pair 90Y. GrL1d, GrR1d, GrL2d, GrR2d, GrR2d, GrL3d, GrR3d, GrL4d, and GrR4d are the pixel values VGr read from the lower light-receiving pixel P (hereinafter also referred to as the lower pixel) and each lower pixel in the pixel pair 90Y. shall indicate

Similarly, in pixel block 100R, RL1u, RL1d, RL2u, RL2d, RL3u, RL3d, RL4u and RL4d denote left pixels and pixel values VGr read from each left pixel, and RR1u, RR1d, RR2u, RR2d, RR3u. , RR3d, RR4u and RR4d indicate pixel values VGr read from the right pixel and each right pixel, and RL1u, RR1u, RL2u, RR2u, RL3u, RR3u, RL4u and RR4u are read from the upper pixel and each upper pixel. RL1d, RR1d, RL2d, RR2d, RL3d, RR3d, RL4d, and RR4d denote the lower pixel and the pixel value VGr read from each lower pixel, and in the pixel block 100B, BL1u, BL1d , BL2u, BL2d, BL3u, BL3d, BL4u and BL4d indicate pixel values VGr read from the left pixel and each left pixel, and BR1u, BR1d, BR2u, BR2d, BR3u, BR3d, BR4u and BR4d indicate right pixel and each pixel value VGr. BL1u, BR1u, BL2u, BR2u, BL3u, BR3u, BL4u and BR4u indicate the pixel values VGr read from the upper pixels and the respective upper pixels, BL1d, BR1d, BL2d, BR2d, BL3d, BR3d, BL4d, and BR4d indicate the lower pixel and the pixel value VGr read from each lower pixel. GbL4d indicates the left pixel and the pixel value VGr read from each left pixel, and GbR1u, GbR1d, GbR2u, GbR2d, GbR3u, GbR3d, GbR4u and GbR4d indicate the right pixel and the pixel value VGr read from each right pixel. GbL1u, GbR1u, GbL2u, GbR2u, GbL3u, GbR3u, Gb and GbR4u indicate the pixel value VGr read from the upper pixel and each upper pixel, and GbL1d, GbR1d, GbL2d, GbR2d, GbL3d, GbR3d, GbL4d and GbR4d indicates the lower pixel and the pixel value VGr read from each lower pixel.

As shown in FIG. 29, the same-color pixel adding unit 171 adds the pixel values GrL1u, GrL1d, GrL2u, GrL2d, GrL3u, GrL3d, By adding GrL4u and GrL4d, the left pixel total value GrL of the pixel block 100Gr is calculated. Similarly, the same color pixel addition unit 171 adds the pixel values GrR1u, GrR1d, GrR2u, GrR2d, GrR3u, GrR3d, GrR4u and GrR4d read from the right pixel P among the pixel values VGr read from the pixel block 100Gr. By adding, the right pixel total value GrR of the pixel block 100Gr is calculated.

Further, the same-color pixel addition unit 171 adds the pixel values GrL1u, GrR1u, GrL2u, GrR2u, GrL3u, GrR3u, GrL4u, and GrR4u read from the upper pixel P among the pixel values VGr read from the pixel block 100Gr. By doing so, the upper pixel total value Gru of the pixel block 100Gr is calculated. Similarly, the same-color pixel addition unit 171 adds the pixel values GrL1d, GrR1d, GrL2d, GrR2d, GrL3d, GrR3d, GrL4d, and GrR4d read from the lower pixel P among the pixel values VGr read from the pixel block 100Gr. By adding, the right pixel total value GrR of the pixel block 100Gr is calculated.

As shown in FIGS. 30 to 32, pixel values RL1u to RL4u and RL1d to RL4d, BL1u to BL4u and BL1d to RL1u to RL4u and RL1d to RL4d read from the left pixel P are similarly applied to the other pixel blocks 100R, 100B, and 100Gb. By adding BL4d or GbL1u to GbL4u and GbL1d to GbL4d, the left pixel total values RL, BL, and GbL of the pixel blocks 100R, 100B, and 100Gb are calculated, and the pixel values RR1u to By adding RR4u and RR1d to RR4d, BR1u to BR4u and BR1d to BR4d, or GbR1u to GbR4u and GbR1d to GbR4d, right pixel total values RR, BR and GbR of pixel blocks 100R, 100B and 100Gb are calculated. By adding the pixel values RL1u to RL4u and RR1u to RR4u, BL1u to BL4u and BR1u to BR4u read from the upper pixels P, or GbL1u to GbL4u and GbR1u to GbR4u, the pixel blocks 100R, 100B and 100Gb are obtained. Calculate the upper pixel total values Ru, Bu, Gbu, and the pixel values RL1d to RL4d and RR1d to RR4d, BL1d to BL4d and BR1d to BR4d read from the lower pixels P, or GbL1d to GbL4d and GbR1d to GbR4d By adding, the lower pixel total values Rd, Bd, and Gbd of the pixel blocks 100R, 100B, and 100Gb are calculated.

As a result, as shown in FIG. 33, based on the pixel values VGr, VR, VB, and VGb read out from the unit unit U, the same-color pixel adder 171 outputs the left pixel total value GrL and the right pixel total value GrL to the pixel block 100Gr. The total pixel value pair 111Grh with the total value GrR is arranged, the total pixel value pair 111Rh with the left pixel total value RL and the right pixel total value RR is arranged in the pixel block 100R, and the left pixel total value BL and the total pixel value RR are arranged in the pixel block 100B. An image signal Spic1h in which a total pixel value pair 111Bh of the right pixel total value BR is arranged and a total pixel value pair 111Gbh of the left pixel total value GbL and the right pixel total value GbR is arranged in the pixel block 100Gb is output. A total pixel value pair 111Grv of the upper pixel total value Gru and the lower pixel total value Grd is arranged in the pixel block 100Gr, and a total pixel value pair 111Rv of the upper pixel total value Ru and the lower pixel total value Rd is arranged in the pixel block 100R. A total pixel value pair 111Bv of the upper pixel total value Bu and the lower pixel total value Bd is arranged in the pixel block 100B, and a total pixel value pair 111Bv of the upper pixel total value Gbu and the lower pixel total value Gbd is arranged in the pixel block 100Gb. An image signal Spic1v in which 111Gbv is arranged is output.

1.5.2 Second Modification In the second modification, each of the pixel blocks 100R, 100Gr, 100Gb, and 100B that make up the unit U is a pixel array composed of a total of nine light-receiving pixels P of 3×3 pixels. A case is exemplified. In this modified example, the pixel values added for detecting the image plane phase difference are not the pixel values read from the light receiving pixels P of the same color. shall be read as "addition".

FIG. 34 is a diagram of one unit U extracted from the arrangement example of the light receiving elements P in the pixel array 11 shown in FIG. 35A and 35B are diagrams for explaining the same-color pixel addition process performed on the pixel values read from the unit U for image plane phase difference detection. FIG. 36 is a diagram showing the left pixel total value and the right pixel total value finally obtained based on the pixel values read out from the unit U. In FIG.

As shown in FIG. 34, in the unit unit U according to this modified example, each of the pixel blocks 100R, 100Gr, 100Gb, and 100B has a total of nine light-receiving pixels P arranged in 3×3 pixels. Further, in this modification, in such an arrangement pattern, a total of four light-receiving pixels P of 2×2 pixels located at the center lower side of the unit unit U are light-receiving pixels for image plane phase difference detection (hereinafter referred to as image plane phase difference detection pixels). (also called surface detection pixels). Specifically, two light-receiving pixels PB1 and PB2 positioned at the lower right of the pixel block 100B and two light-receiving pixels PGb1 and PGb2 positioned at the lower left of the pixel block 100Gb and adjacent to the light-receiving pixels PB1 and PB2, respectively. , are used as image plane detection pixels. Therefore, in this modification, the light-receiving pixels PB1 and PGb1 form a pixel pair 90-1 sharing one on-chip lens 101, and the light-receiving pixels PB2 and PGb2 form a pixel pair 90-1 sharing one on-chip lens 101. 2.

As shown in FIG. 35, the same-color pixel addition unit 171 calculates the left pixel total value BL by adding the pixel values BL1 and BL2 read from the light-receiving pixels PB1 and PB2 that are the left pixels. The right pixel total value GrR is calculated by adding the pixel values GbR1 and GbR2 read from certain light receiving pixels PGb1 and PGb2.

As a result, as shown in FIG. 36, the same color pixel adder 171 outputs an image signal Spic1 in which each unit U is composed of a total pixel value pair 310 of the left pixel total value GrL and the right pixel total value GrR. be done.

In this modification, the memory 175 may store a sensitivity difference correction coefficient table that considers the difference in light sensitivity between the light-receiving pixels PB1 and PB2 and the light-receiving pixels Gb1 and Gb2. As a result, even if there is a difference in light sensitivity between the light receiving pixels PB1 and PB2 and the light receiving pixels Gb1 and Gb2, phase difference information can be generated based on the pixel values read out from these pixels.

1.5.3 Third Modification In the third modification, the pixel arrangement pattern of the pixel array 11 is the DecaOcta arrangement exemplified in the first embodiment, and the A case of detecting an image plane phase difference using a part of pixel values will be exemplified.

FIG. 37 is a diagram showing the left pixel total value and right pixel total value finally obtained based on the pixel values read out from the unit U. In FIG.

As shown in FIG. 37, in this modification, the same-color pixel adder 171 reads from one of the pixel blocks 100R, 100Gr, 100Gb, and 100B included in the unit unit U (the pixel block 100Gb in this example). Using the output pixel values (pixel values GbL1 to GbL5 and GbR1 to GbR5 in this example), a left pixel total value GbL and a right pixel total value GbR are calculated, and a total pixel value pair 410 based on these is calculated for each unit unit U. , and outputs an image signal Spic1 having a total pixel value pair of .

In this modified example, the pixel block 100Gb is used as the pixel block used for detecting the image plane phase difference. good too. Further, in this modification, the case where all the light-receiving pixels P constitute the pixel pair 90 with the adjacent pixels is illustrated, but the present invention is not limited to this, and the light-receiving pixels P not used for detecting the image plane phase difference are A separate on-chip lens 101 that does not span adjacent pixels may be provided.

1.5.4 Fourth Modification In a fourth modification, a case of detecting an image plane phase difference using pixel values further narrowed down in the third modification will be described.

FIG. 38 is a diagram showing the left pixel total value and the right pixel total value finally obtained based on the pixel values read out from the unit U. In FIG.

As shown in FIG. 38, in this modification, from one pixel pair 90 included in one of pixel blocks 100R, 100Gr, 100Gb, and 100B included in the unit unit U (pixel block 100Gb in this example), The read pixel values (pixel values GbL1 and GbR1 in this example) are used as they are as the left pixel total value GbL1 and the right pixel total value GbR1. In that case, the same-color pixel addition processing in the same-color pixel addition unit 171 may be bypassed. That is, the same-color pixel adder 171 extracts the pixel values (GbL1 and GbR1) of the target pixel pair 90 from the input image signal Spic0, and sums the total pixel value pair 510 for each unit U. An image signal Spic1 may be output as pixel value pairs.

In this modified example, the pixel pair 90 used for detecting the image plane phase difference is exemplified as one pixel pair 90 in the pixel block 100Gb. , 100B may be the pixel pairs 90 used for detecting the image plane phase difference. Further, in this modification, the case where all the light-receiving pixels P constitute the pixel pair 90 with the adjacent pixels is illustrated, but the present invention is not limited to this, and the light-receiving pixels P not used for detecting the image plane phase difference are A separate on-chip lens 101 that does not span adjacent pixels may be provided.

1.5.5 Fifth Modification In the fifth modification, as in the third modification, the pixel arrangement pattern of the pixel array 11 is the DecaOcta arrangement illustrated in the first embodiment. . However, in this modified example, a case where the imaging device 1 operates in the HDR mode is illustrated.

FIG. 39 is a diagram showing the left pixel total value and the right pixel total value finally obtained based on the pixel values read out from the unit unit U. FIG.

As shown in FIG. 39, in this modification, one of the pixel blocks 100R, 100Gr, 100Gb, and 100B included in the unit unit U (in this example, the pixel block 100Gb) is a pixel for image plane phase difference detection. Used as a block.

Here, in the example shown in FIG. 39, in the pixel block 100Gb, GbL1L to GbL3L indicate the left pixel and the pixel value VGb read from each left pixel in the pixel pair 90 of the low luminance pixel PL, and GbR1L to GbR3L indicate the low luminance pixel value VGb. The pixel values VGb read from the right pixel and each right pixel in the pixel pair 90 of the luminance pixel PL are shown, and GbL1H to GbL2H are the left pixel and pixels read from each left pixel in the pixel pair 90 of the high luminance pixel PH. GbR1H to GbR2H indicate the right pixel and the pixel value VGb read from each right pixel in the pixel pair 90 of the high luminance pixel PH.

Therefore, in this modification, the same-color pixel addition unit 171 adds the pixel values GbL1H and GrL2H read from the left pixel P among the pixel values VGb read from the high-brightness pixels PH in the pixel block 100Gb. , the left pixel total value GbLH of the high luminance pixels PH in the pixel block 100Gb is calculated. Similarly, the same-color pixel addition unit 171 adds the pixel values GbR1H and GbR2H read from the right pixel P among the pixel values VGb read from the high-brightness pixels PH in the pixel block 100Gb, thereby A right pixel total value GbRH of the high luminance pixel PH at 100 Gb is calculated.

Further, the same-color pixel addition unit 171 adds the pixel values GbL1L, GbL2L, and GbL3L read from the left pixel P among the pixel values VGb read from the low-luminance pixels PL in the pixel block 100Gb. A left pixel total value GbLL of the low-luminance pixels PL in the block 100Gb is calculated. Similarly, the same-color pixel addition unit 171 adds the pixel values GbR1L, GbR2L, and GbR3L read from the right pixel P among the pixel values VGb read from the low-luminance pixels PL in the pixel block 100Gb. A right pixel total value GbRL of the low luminance pixels PL in the pixel block 100Gb is calculated.

Then, the same-color pixel addition unit 171 adds a total pixel value pair 111H including the left pixel total value GbLH and the right pixel total value GbRH of the high-luminance pixel PH, and the left pixel total value GbLL and the right pixel total value GbRL of the low-luminance pixel PL. An image signal Spic1 having a total pixel value pair 111L and a total pixel value pair 610 for each unit unit U is output.

In this modified example, the pixel block 100Gb is used as the pixel block used for detecting the image plane phase difference. good too. Further, in this modification, the case where all the light-receiving pixels P constitute the pixel pair 90 with the adjacent pixels is illustrated, but the present invention is not limited to this, and the light-receiving pixels P not used for detecting the image plane phase difference are A separate on-chip lens 101 that does not span adjacent pixels may be provided.

2. Second Embodiment Next, a solid-state imaging device and an electronic device according to a second embodiment of the present disclosure will be described in detail with reference to the drawings. Note that, in the following description, the configurations, operations, and effects that are the same as those of the above-described embodiment will be referred to to omit overlapping descriptions.

In the first embodiment described above, as shown in FIG. 2, one unit unit U includes pixel blocks 100Gr and 100Gb each including ten light-receiving pixels P and a pixel block 100R including eight light-receiving pixels P. , 100B, but it is not limited to this.

2.1 Configuration Example FIG. 40 shows a configuration example of the imaging device 2 according to this embodiment. The imaging device 2 includes a pixel array 31 , a driving section 32 and a signal processing section 35 .

FIG. 41 shows an example of arrangement of the light receiving pixels P in the pixel array 31. As shown in FIG. The pixel array 31 has multiple pixel blocks 300 and multiple lenses 101 .

The plurality of pixel blocks 300 includes pixel blocks 300R, 300Gr, 300Gb and 300B. In the pixel array 31, a plurality of light-receiving pixels P are arranged in units (units U) of four pixel blocks 300 (pixel blocks 300R, 300Gr, 300Gb, and 300B).

The pixel block 300R has eight light-receiving pixels P (light-receiving pixels PR) including red (R) color filters 55, and the pixel block 300Gr has eight light-receiving pixels including green (G) color filters 55. The pixel block 300Gb has 10 light-receiving pixels P (light-receiving pixels PGb) including green (G) color filters 55, and the pixel block 300B has blue (B) color filters. It has eight light-receiving pixels P (light-receiving pixels PB) including color filters 55 . The arrangement pattern of the light-receiving pixels PR in the pixel block 300R, the arrangement pattern of the light-receiving pixels PGr in the pixel block 300Gr, the arrangement pattern of the light-receiving pixels PGb in the pixel block 300Gb, and the arrangement pattern of the light-receiving pixels PB in the pixel block 300B are the same. . In the unit U, the pixel block 300Gr is located at the upper left, the pixel block 300R is located at the upper right, the pixel block 300B is located at the lower left, and the pixel block 300Gb is located at the lower right. In this manner, the pixel blocks 300R, 300Gr, 300Gb, and 300B are arranged in a so-called Bayer arrangement with the pixel block 300 as a unit.

In the pixel array 31, a plurality of light-receiving pixels P are arranged side by side in an oblique direction. That is, in the pixel array 11 (FIG. 2) according to the first embodiment described above, the plurality of light receiving pixels P are arranged side by side in the X direction and the Y direction, but in the pixel array 31 (FIG. 41) , a plurality of light-receiving pixels P are arranged in parallel in an oblique direction. As a result, the two light receiving pixels P in the pixel pair 90 are also arranged side by side in an oblique direction. A lens 101 is provided above the pixel pair 90 . In this description, an oblique direction upward to the left is called an oblique direction, and a direction perpendicular to the oblique direction is called a reverse oblique direction. Further, in this description, the light-receiving pixel P positioned diagonally to the upper left is also referred to as a left pixel, and the light-receiving pixel P positioned diagonally to the right is also referred to as a right pixel.

The configuration of the pixel block 300 (pixel blocks 300R, 300Gr, 300Gb, 300B) is the same as the pixel block 100R (FIG. 5) according to the above-described first embodiment, and includes eight photodiodes PD and eight photodiodes PD. It has a transistor TRG, a floating diffusion FD, and transistors RST, AMP, and SEL. Eight photodiodes PD and eight transistors TRG correspond to eight light-receiving pixels P included in the pixel block 300, respectively.

FIG. 42 shows a wiring example of the pixel blocks 300R, 300Gr, 300Gb and 300B. In addition, in FIG. 42, the plurality of pixel blocks 300 are drawn apart from each other for convenience of explanation.

The pixel array 31 has a plurality of control lines TRGL, a plurality of control lines RSTL, a plurality of control lines SELL, and a plurality of signal lines VSL, like the pixel array 11 according to the first embodiment described above. are doing. The control line TRGL extends in the X direction and has one end connected to the driving section 32 . The control line RSTL extends in the X direction and has one end connected to the driving section 32 . The control line SELL extends in the X direction and has one end connected to the driving section 32 . The signal line VSL extends in the Y direction and has one end connected to the readout section 20 .

The pixel block 300Gr and the pixel block 300R belonging to the same row and arranged in the X direction are connected to the same eight control lines TRGL. Although not shown, the pixel blocks 300Gr and 300R belonging to the same row and arranged in the X direction are connected to one control line RSTL and one control line SELL. Pixel blocks 300Gr belonging to the same column and arranged in the Y direction are connected to one signal line VSL. Similarly, pixel blocks 300R belonging to the same column arranged in the Y direction are connected to one signal line VSL.

Similarly, the pixel blocks 300B and 300Gb belonging to the same row and arranged in the X direction are connected to the same eight control lines TRGL. Although not shown, the pixel blocks 300B and 300Gb belonging to the same row and arranged in the X direction are connected to one control line RSTL and one control line SELL. Pixel blocks 300B belonging to the same column and arranged in the Y direction are connected to one signal line VSL. Similarly, the pixel blocks 300Gb belonging to the same column arranged in the Y direction are connected to one signal line VSL.

Thereby, in the pixel array 31, for example, the line density of the control lines TRGL can be reduced more than the pixel array 11 according to the first embodiment described above. Specifically, in the pixel array 11 (FIGS. 2 and 6) according to the first embodiment described above, the line density of the control lines TRGL is four per side of the light receiving pixel P. As shown in FIG. On the other hand, in the pixel array 31 (FIGS. 41 and 42) according to the present embodiment, the line density of the control lines TRGL is 2.83 per side of the light receiving pixel P (=2√2). As a result, in the pixel array 31, for example, wiring of the control lines TRGL can be facilitated, or resolution can be enhanced by reducing the pitch of the light receiving pixels P. FIG.

The drive unit 32 ( FIG. 40 ) is configured to drive the plurality of light receiving pixels P in the pixel array 31 based on instructions from the imaging control unit 18 . The signal processing unit 35 is configured to perform predetermined signal processing based on the image signal Spic0 and an instruction from the imaging control unit 18 to generate the image signal Spic. The signal processor 35 has an image data generator 36 and a phase difference data generator 37 . The phase difference data generation section 37 may have, for example, the same configuration as the phase difference data generation section 17 described with reference to FIG. 11 in the first embodiment.

For example, the imaging device 2 selects the first imaging mode M when the zoom magnification is less than 2, and selects the second imaging mode M when the zoom magnification is 2 or more and less than 2√2. If the zoom magnification is 2√2 or more, the third imaging mode M can be selected.

In this way, by arranging the light receiving pixels P as shown in FIG. It is possible to reduce the possibility that there will be a difference between the amount and the amount.

That is, as shown in FIG. 43, even in this case, the light-receiving pixel PGr associated with green detects not only green light but also red light LR, and the light-receiving pixel PGb associated with green detects not only green light but also red light LR. Blue light LB may also be detected. However, the arrangement position of the pixel block 300Gr and the arrangement position of the pixel block 300Gb are symmetrical with respect to the pixel block 300R and similarly with respect to the pixel block 300B. Therefore, for example, the amount of red light LR leaking from the pixel block 300R to the pixel block 300Gr is substantially the same as the amount of red light LR leaking from the pixel block 300R to the pixel block 300Gb. Similarly, for example, the amount of blue light LB leaking from the pixel block 300B to the pixel block 300Gr is substantially the same as the amount of blue light LB leaking from the pixel block 300B to the pixel block 300Gb. Therefore, for example, depending on the balance between the light intensity of the red light LR and the light intensity of the blue light LB, it is possible to reduce the possibility of a difference occurring between the amount of light received by the light-receiving pixel PGr and the amount of light received by the light-receiving pixel PGb. can.

2.2 Functions and Effects In the imaging device 2 including the pixel array 31 configured as described above, as in the first embodiment, the image signal Spic0 input to the phase difference data generation unit 37 is generated in the pixel blocks 300R, 300R, and 300R. Each of 300Gr, 300Gb, and 300B is converted into an image signal Spic1 of one left and right pixel (corresponding to the left pixel total value and right pixel total value). As a result, the resolution of the image signal Spic1 input to the sensitivity difference correction unit 172 is unified regardless of the resolution of the image signal Spic0 (whether or not there is binning), in other words, regardless of the mode being executed by the imaging device 2. Therefore, the sensitivity difference correction processing according to this embodiment can be applied to substantially all modes.

In addition, the resolution of the image signal Spic1 input to the sensitivity difference correction unit 172 is set to the minimum pixel blocks 300R, 300Gr, 300Gb, and 300B, regardless of the number of pixels in each of the pixel blocks 300R, 300Gr, 300Gb, and 300B in the image signal Spic0. Since it is possible to reduce the resolution to a unit, it is possible to reduce the circuit scale by reducing the number of parallel channels for sensitivity difference correction processing and reduce the capacity of the memory 175 for storing the sensitivity difference correction coefficient table. As a result, it becomes possible to suppress an increase in device size.

Furthermore, since it is possible to output the image signal Spic1 with a uniform resolution without depending on the pixel arrangement of the input image signal Spic0, each pixel block 300R, 300Gr, 300Gb, It is also applicable to imaging devices with any pixel array pattern, such as a non-rectangular pixel array pattern such as a square or rectangular pixel block 300B, or a pixel array pattern in which the number of pixels constituting each pixel block 300R, 300Gr, 300Gb, and 300B is not uniform. It is possible to apply this embodiment.

Furthermore, in this embodiment, not only when the binning mode is provided in a plurality of stages, but also when it is possible to switch between an SDR (Standard Dynamic Range) mode using a normal dynamic range and an HDR mode, sensitivity difference correction processing Since it is possible to uniformly convert the pixel arrangement pattern of the image signal Spic0 to be input to the Spic0 into a simple arrangement pattern, the lightened sensitivity difference correction processing according to this embodiment is applied to almost all modes. There is also the advantage of being able to

In the present embodiment, the pixel pairs 90 in the pixel array 31 are all composed of two light-receiving pixels P that are adjacent in the diagonal direction. A part or the whole of 90 may be composed of two light receiving elements P adjacent to each other in the opposite oblique direction. When all the pixel pairs 90 are composed of the light receiving pixels P adjacent in the diagonal direction, autofocus based on the image plane phase difference in the diagonal direction is possible, and all the pixel pairs 90 are composed of the light receiving pixels P adjacent in the opposite diagonal direction. , autofocus based on the image plane phase difference in the opposite oblique direction is possible, and a part of the pixel pair 90 is composed of the light receiving pixels P adjacent in the oblique direction, and the rest is composed of the light receiving pixels P adjacent in the opposite oblique direction. When configured with pixels P, autofocus based on image plane phase differences in oblique and reverse oblique directions is possible.

Other configurations, operations, and effects may be the same as those of the above-described embodiment or modifications thereof, and detailed descriptions thereof are omitted here.

2.3 Modifications In the above-described second embodiment, four pixel blocks 300 (pixel blocks 300R, 300Gr, 300Gb, and 300B) are arranged as a unit (unit U), but the present invention is not limited to this. A detailed description will be given below with an example of arranging eight pixel blocks as a unit. An imaging device 2A according to the present embodiment includes a pixel array 31A, a driving section 32A, and a signal processing section 35A, like the imaging device 2 (FIG. 40).

FIG. 44 shows an example of arrangement of the light receiving pixels P in the pixel array 31A. The pixel array 31A has multiple pixel blocks 300 and multiple lenses 101 .

The pixel array 31A has a plurality of pixel blocks 300 and a plurality of lenses 101, like the pixel array 31 (FIG. 41).

The plurality of pixel blocks 300 includes pixel blocks 300R1, 300R2, 300Gr1, 300Gr2, 300Gb1, 300Gb2, 300B1 and 300B2. The pixel block 300 (pixel blocks 300R1, 300R2, 300Gr1, 300Gr2, 300Gb1, 300Gb2, 300B1, 300B2) has four light receiving pixels P. As shown in FIG. Specifically, each of the pixel blocks 300R1 and 300R2 has four light-receiving pixels PR, each of the pixel blocks 300Gr1 and 300Gr2 has four light-receiving pixels PGr, and each of the pixel blocks 300Gb1 and 300Gb2 has: It has four light receiving pixels PGb, and each of the pixel blocks 300B1 and 300B2 has four light receiving pixels PB. The pixel block 300 has four photodiodes PD, four transistors TRG, a floating diffusion FD, and transistors RST, AMP, and SEL. Four photodiodes PD and four transistors TRG correspond to four light receiving pixels P included in the pixel block 300, respectively.

FIG. 45 shows a wiring example of the pixel blocks 300R1, 300R2, 300Gr1, 300Gr2, 300Gb1, 300Gb2, 300B1 and 300B2. In addition, in FIG. 44, the plurality of pixel blocks 300 are drawn apart from each other for convenience of explanation.

The pixel blocks 300Gr1, 300Gr2, 300R1, 300R2 belonging to the same row arranged in the X direction are connected to the same four control lines TRGL. Although not shown, the pixel blocks 300Gr1, 300Gr2, 300R1, 300R2 belonging to the same row and arranged in the X direction are connected to one control line RSTL and one control line SELL. Pixel blocks 300Gr1 belonging to the same column aligned in the Y direction are connected to one signal line VSL, and pixel blocks 300Gr2 belonging to the same column aligned in the Y direction are connected to one signal line VSL. Similarly, the pixel blocks 300R1 belonging to the same column arranged in the Y direction are connected to one signal line VSL, and the pixel blocks 300R2 belonging to the same column arranged in the Y direction are connected to one signal line VSL.

Similarly, pixel blocks 300B1, 300B2, 300Gb1, and 300Gb2 belonging to the same row and arranged in the X direction are connected to the same four control lines TRGL. Although not shown, the pixel blocks 300B1, 300B2, 300Gb1 and 300Gb2 belonging to the same row and arranged in the X direction are connected to one control line RSTL and one control line SELL. Pixel blocks 300B1 belonging to the same column aligned in the Y direction are connected to one signal line VSL, and pixel blocks 300B2 belonging to the same column aligned in the Y direction are connected to one signal line VSL. Similarly, the pixel blocks 300Gb1 belonging to the same column arranged in the Y direction are connected to one signal line VSL, and the pixel blocks 300Gb2 belonging to the same column arranged in the Y direction are connected to one signal line VSL.

Thereby, in the pixel array 31A, for example, the line density of the control lines TRGL can be reduced more than in the pixel array 31 according to the modification. Specifically, in the pixel array 31A, the line density of the control lines TRGL is 1.41 per side of the light receiving pixel P (=√2). Accordingly, in the pixel array 31A, for example, wiring of the control lines TRGL can be facilitated, or the pitch of the light receiving pixels P can be reduced.

The driving section 32A is configured to drive the plurality of light receiving pixels P in the pixel array 31 based on instructions from the imaging control section 18 . The signal processing unit 35A is configured to generate the image signal Spic by performing predetermined signal processing based on the image signal Spic0 and an instruction from the imaging control unit 18 . The signal processor 35A has an image data generator 36A and a phase difference data generator 37A. The phase difference data generator 37A may have, for example, the same configuration as the phase difference data generator 17 described with reference to FIG. 11 in the first embodiment.

In the above example, the two light-receiving pixels P in the pixel pair 90 are arranged side by side in an oblique direction, but the present invention is not limited to this. You may arrange side by side in a direction.

3. Usage Example of Imaging Apparatus FIG. 46 shows a usage example (an example of an electronic device) of the imaging apparatuses 1 and 2 according to the above-described embodiments. The imaging devices 1 and 2 described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

・Devices that capture images for viewing purposes, such as digital cameras and mobile devices with camera functions Devices used for transportation, such as in-vehicle sensors that capture images behind, around, and inside the vehicle, surveillance cameras that monitor running vehicles and roads, and ranging sensors that measure the distance between vehicles. Devices used in home appliances such as televisions, refrigerators, air conditioners, etc., endoscopes, and devices that perform angiography by receiving infrared light to capture images and operate devices according to gestures. Devices used for medical and health care, such as equipment used for security purposes such as monitoring cameras for crime prevention and cameras used for personal authentication, skin measuring instruments for photographing the skin, scalp Equipment used for beauty, such as a microscope for photographing Equipment used for sports, such as action cameras and wearable cameras for sports, etc. Cameras for monitoring the condition of fields and crops, etc. of agricultural equipment

4. Application Examples to Mobile Objects The technology according to the present disclosure (this technology) can be applied to various products. For example, the technology according to the present disclosure can be realized as a device mounted on any type of moving body such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility, airplanes, drones, ships, and robots. may

FIG. 47 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technology according to the present disclosure can be applied.

Vehicle control system 12000 comprises a plurality of electronic control units connected via communication network 12001 . In the example shown in FIG. 47 , vehicle control system 12000 includes drive system control unit 12010 , body system control unit 12020 , vehicle exterior information detection unit 12030 , vehicle interior information detection unit 12040 , and integrated control unit 12050 . Also, as the functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio/image output unit 12052, and an in-vehicle network I/F (interface) 12053 are illustrated.

Drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the driving system control unit 12010 includes a driving force generator for generating driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism to adjust and a brake device to generate braking force of the vehicle.

Body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, winkers or fog lamps. In this case, the body system control unit 12020 can receive radio waves transmitted from a portable device that substitutes for a key or signals from various switches. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, etc. of the vehicle.

External information detection unit 12030 detects information external to the vehicle in which vehicle control system 12000 is mounted. For example, the vehicle exterior information detection unit 12030 is connected with an imaging section 12031 . The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image of the exterior of the vehicle, and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as people, vehicles, obstacles, signs, or characters on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal according to the amount of received light. The imaging unit 12031 can output the electric signal as an image, and can also output it as distance measurement information. Also, the light received by the imaging unit 12031 may be visible light or non-visible light such as infrared rays.

The vehicle interior information detection unit 12040 detects vehicle interior information. The in-vehicle information detection unit 12040 is connected to, for example, a driver state detection section 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 detects the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether the driver is dozing off.

The microcomputer 12051 calculates control target values for the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and controls the drive system control unit. A control command can be output to 12010 . For example, the microcomputer 12051 realizes functions of ADAS (Advanced Driver Assistance System) including collision avoidance or shock mitigation of vehicles, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, etc. Cooperative control can be performed for the purpose of

In addition, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, etc. based on the information about the vehicle surroundings acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, so that the driver's Cooperative control can be performed for the purpose of autonomous driving, etc., in which vehicles autonomously travel without depending on operation.

Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the information detection unit 12030 outside the vehicle. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the vehicle exterior information detection unit 12030, and performs cooperative control aimed at anti-glare such as switching from high beam to low beam. It can be carried out.

The audio/image output unit 12052 transmits at least one of audio and/or image output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle. In the example of FIG. 47, an audio speaker 12061, a display unit 12062 and an instrument panel 12063 are illustrated as output devices. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

FIG. 48 is a diagram showing an example of the installation position of the imaging unit 12031. As shown in FIG.

In FIG. 48 , vehicle 12100 has imaging units 12101 , 12102 , 12103 , 12104 , and 12105 as imaging unit 12031 .

The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and windshield of the vehicle 12100, for example. An image pickup unit 12101 provided in the front nose and an image pickup unit 12105 provided above the windshield in the passenger compartment mainly acquire images in front of the vehicle 12100 . Imaging units 12102 and 12103 provided in the side mirrors mainly acquire side images of the vehicle 12100 . An imaging unit 12104 provided in the rear bumper or back door mainly acquires an image behind the vehicle 12100 . Forward images acquired by the imaging units 12101 and 12105 are mainly used for detecting preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, and the like.

Note that FIG. 48 shows an example of the imaging range of the imaging units 12101 to 12104 . The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided in the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided in the side mirrors, respectively, and the imaging range 12114 The imaging range of an imaging unit 12104 provided on the rear bumper or back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 determines the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and changes in this distance over time (relative velocity with respect to the vehicle 12100). , it is possible to extract, as the preceding vehicle, the closest three-dimensional object on the course of the vehicle 12100, which runs at a predetermined speed (for example, 0 km/h or more) in substantially the same direction as the vehicle 12100. can. Furthermore, the microcomputer 12051 can set the inter-vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including following stop control) and automatic acceleration control (including following start control). In this way, cooperative control can be performed for the purpose of automatic driving in which the vehicle runs autonomously without relying on the operation of the driver.

For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 converts three-dimensional object data related to three-dimensional objects to other three-dimensional objects such as motorcycles, ordinary vehicles, large vehicles, pedestrians, and utility poles. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into those that are visible to the driver of the vehicle 12100 and those that are difficult to see. Then, the microcomputer 12051 judges the collision risk indicating the degree of danger of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, an audio speaker 12061 and a display unit 12062 are displayed. By outputting an alarm to the driver via the drive system control unit 12010 and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be performed.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not the pedestrian exists in the captured images of the imaging units 12101 to 12104 . Such recognition of a pedestrian is performed by, for example, a procedure for extracting feature points in images captured by the imaging units 12101 to 12104 as infrared cameras, and performing pattern matching processing on a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. This is done by a procedure that determines When the microcomputer 12051 determines that a pedestrian exists in the images captured by the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a rectangular outline for emphasis to the recognized pedestrian. is superimposed on the display unit 12062 . Also, the audio/image output unit 12052 may control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 among the configurations described above. An imaging device mounted on a vehicle can improve the image quality of a captured image. As a result, the vehicle control system 12000 realizes a vehicle collision avoidance or collision mitigation function, a follow-up driving function based on the distance between vehicles, a vehicle speed maintenance driving function, a vehicle collision warning function, a vehicle lane deviation warning function, etc. with high accuracy. can.

Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the gist of the present disclosure. Moreover, you may combine the component over different embodiment and modifications suitably.

Further, the effects of each embodiment described in this specification are merely examples and are not limited, and other effects may be provided.

Note that the present technology can also take the following configuration.
(1)
a pixel array comprising a plurality of light receiving pixels;
a reading unit that generates an image signal composed of pixel values read from each of the light-receiving pixels;
a signal processing unit that processes the image signal output from the reading unit;
with
The pixel array includes a plurality of pixel pairs consisting of at least two light receiving pixels sharing one on-chip lens,
The signal processing unit is
adding pixel values read from first light-receiving pixels in each of at least two pixel pairs among the plurality of pixel pairs, and reading from second light-receiving pixels different from the first light-receiving pixels in each of the pixel pairs; an addition unit that performs a first addition process for adding the output pixel values;
a generation unit that generates information about an image plane phase difference between the first light receiving pixel and the second light receiving pixel based on a plurality of pixel values included in the image signal after the first addition processing;
An imaging device comprising:
(2)
The plurality of light-receiving pixels include light-receiving pixels that receive light of a first wavelength component and light-receiving pixels that receive light of a second wavelength component different from the light of the first wavelength component,
The imaging device according to (1), wherein the adding section adds the pixel values read from the light receiving pixels that receive light of the same wavelength component.
(3)
The generating unit includes a correcting unit that corrects a plurality of pixel values included in the image signal after the first addition processing based on a sensitivity difference between the plurality of light receiving pixels. imaging device.
(4)
The plurality of light receiving pixels are arranged in a matrix,
The imaging device according to any one of (1) to (3), wherein at least one of the at least two pixel pairs includes the first light-receiving pixel and the second light-receiving pixel that are adjacent in the row direction. .
(5)
The plurality of light receiving pixels are arranged in a matrix,
The imaging device according to any one of (1) to (4), wherein at least one of the at least two pixel pairs includes the first light-receiving pixel and the second light-receiving pixel that are adjacent in the column direction. .
(6)
The plurality of light receiving pixels are arranged in a matrix,
Each of the on-chip lenses is shared by the four light receiving elements arranged in 2×2,
The at least two pixel pairs are, of the four light receiving elements sharing the one on-chip lens, a first pixel pair composed of the first light receiving pixel and the second light receiving pixel adjacent in the row direction, and a column a second pixel pair consisting of a third light-receiving pixel and a fourth light-receiving pixel that are adjacent in a direction;
The adder adds the pixel values read from the first light receiving pixels in each of the first pixel pairs, and adds the pixel values read from the second light receiving pixels in each of the first pixel pairs. In addition to the first addition processing of adding the pixel values read from the third light receiving pixels in each of the second pixel pairs, adding the pixel values read from the fourth light receiving pixels in each of the second pixel pairs further executing a second addition process for adding the pixel values obtained,
The generation unit generates information about the image plane phase difference between the first light receiving pixel and the second light receiving pixel based on the plurality of pixel values included in the image signal after the first addition processing. and generating information about an image plane phase difference between the third light receiving pixel and the fourth light receiving pixel based on the plurality of pixel values included in the image signal after the second addition processing. ).
(7)
The plurality of light-receiving pixels include light-receiving pixels operating with a sensitivity of a first dynamic range and light-receiving pixels operating with a sensitivity of a second dynamic range wider than the first dynamic range,
The at least two pixel pairs are a third pixel pair consisting of the first light-receiving pixel and the second light-receiving pixel that operate with the sensitivity of the first dynamic range, and the pixel pair that operates with the sensitivity of the second dynamic range. and a fourth pixel pair consisting of the first light receiving pixel and the second light receiving pixel,
The first addition process adds the pixel values read from the first light receiving pixels in each of the third pixel pairs, and adds the pixel values read from the second light receiving pixels in each of the third pixel pairs. a third addition process for adding pixel values; adding pixel values read from the first light receiving pixels in each of the fourth pixel pairs; and reading from the second light receiving pixels in each of the fourth pixel pairs. A fourth addition process that adds the pixel values obtained,
The generating unit combines the first total pixel value calculated by the third addition process and the second total pixel value calculated by the fourth addition process, and the The imaging device according to any one of (1) to (6), wherein information about the image plane phase difference between the first light receiving pixel and the second light receiving pixel is generated based on a plurality of pixel values.
(8)
the plurality of light-receiving pixels are grouped into a plurality of pixel blocks each including two or more light-receiving pixels;
The imaging according to any one of (1) to (7), wherein the addition unit adds the pixel value of the first light receiving pixel and the pixel value of the second light receiving pixel in the same pixel block. Device.
(9)
at least one of the plurality of pixel blocks includes two or more of the pixel pairs;
The reading unit reads one or more pixel values from the first light receiving pixels in the two or more pixel pairs included in the at least one pixel block, and reads the second light receiving pixels in the two or more pixel pairs. read out one or more pixel values from
The adder adds the one or more pixel values read from the first light-receiving pixels in the two or more pixel pairs for each pixel block, and adds the pixel values in the two or more pixel pairs. The imaging device according to (8), wherein the first addition process is performed to add the one or more pixel values read from two light receiving pixels.
(10)
The pixel array comprises a plurality of unit units, each of which is a repeating unit, each of which is composed of two or more of the pixel blocks;
each of the pixel blocks is composed of the two or more light-receiving pixels that receive light in the same wavelength band;
The imaging device according to (8) or (9), wherein at least two pixel blocks in each of the unit units are composed of the two or more light-receiving pixels that receive light in different wavelength bands between the pixel blocks.
(11)
The imaging device according to (10), wherein each of the unit units is composed of two pixel blocks each composed of ten light-receiving pixels and two pixel blocks each composed of eight light-receiving pixels.
(12)
The imaging device according to (10), wherein each of the unit units is composed of four pixel blocks each composed of 16 light receiving pixels arranged in 4×4.
(13)
The imaging device according to (10), wherein each of the unit units is composed of four pixel blocks each composed of nine light receiving pixels arranged in 3×3.
(14)
The adder adds the pixel values read from the first light-receiving pixels in at least one pixel block among the plurality of pixel blocks, and adds the pixel values read from the second light-receiving pixels in the at least one pixel block. The imaging apparatus according to any one of (8) to (13) above, wherein a first addition process is performed to add the read pixel values.
(15)
further comprising a plurality of control lines extending in a row direction with respect to the pixel array and connected to the plurality of light receiving pixels;
the plurality of light-receiving pixels are arranged in a matrix in an oblique direction with respect to the extending direction of each of the control lines;
The imaging device according to any one of (1) to (14), wherein the pixel pair is composed of the at least two light-receiving pixels adjacent in the oblique direction.
(16)
An electronic device comprising the imaging device according to any one of (1) to (15).

Reference Signs List 1, 2 imaging device 11, 31 pixel array 12, 32 driving section 13 reference signal generating section 15, 35 signal processing section 16, 36 image data generating section 17, 37 phase difference data generating section 18 imaging control section 20 reading section 21 constant current source 22, 23 capacitive element 24 comparison circuit 25 counter 26 latch 51 semiconductor substrate 52 semiconductor region 53 insulating layer 54 multilayer wiring layer 55 color filter 56 light shielding film 90, 90-1, 90-2, 90X, 90Y pixel pair 100B, 100Gb, 100Gr, 100R, 300B, 300B1, 300B2, 300Gb, 300Gb1, 300Gb2, 300Gr, 300Gr1, 300Gr2, 300R, 300R1, 300R2 Pixel block 101 On-chip lens 111B, 111Gb, 111Gr, 1111, 511, 0111, 031 Total pixel value pair 171 Same color pixel adder 172 Sensitivity difference corrector 173 Phase difference information generator 174 Phase difference detector 175 Memory DF Phase difference data DP Image data FD Floating diffusion LR, LB Light P, PB, PB1, PB2, PGb , PGr, PR Light-receiving pixel PD Photodiode S Light incident surface Spic0, Spic1, Spic1h, Spic1v Image signal TRG, AMP, RST, SEL Transistor TRGL, RSTL, SELL Control line U Unit unit VSL Signal line

Claims (16)

a pixel array comprising a plurality of light receiving pixels;
a reading unit that generates an image signal composed of pixel values read from each of the light-receiving pixels;
a signal processing unit that processes the image signal output from the reading unit;
with
The pixel array includes a plurality of pixel pairs consisting of at least two light receiving pixels sharing one on-chip lens,
The signal processing unit is
adding pixel values read from first light-receiving pixels in each of at least two pixel pairs among the plurality of pixel pairs, and reading from second light-receiving pixels different from the first light-receiving pixels in each of the pixel pairs; an addition unit that performs a first addition process for adding the output pixel values;
a generation unit that generates information about an image plane phase difference between the first light receiving pixel and the second light receiving pixel based on a plurality of pixel values included in the image signal after the first addition processing;
An imaging device comprising: The plurality of light-receiving pixels include light-receiving pixels that receive light of a first wavelength component and light-receiving pixels that receive light of a second wavelength component different from the light of the first wavelength component,
The imaging device according to claim 1, wherein the adder adds the pixel values read from the light receiving pixels that receive light of the same wavelength component. The imaging device according to claim 1, wherein the generation unit includes a correction unit that corrects the plurality of pixel values included in the image signal after the first addition processing based on the sensitivity difference between the plurality of light receiving pixels. The plurality of light receiving pixels are arranged in a matrix,
The imaging device according to claim 1, wherein at least one of the at least two pixel pairs is composed of the first light receiving pixel and the second light receiving pixel that are adjacent in the row direction. The plurality of light receiving pixels are arranged in a matrix,
The imaging device according to claim 1, wherein at least one of the at least two pixel pairs is composed of the first light receiving pixel and the second light receiving pixel that are adjacent in the column direction. The plurality of light receiving pixels are arranged in a matrix,
Each of the on-chip lenses is shared by the four light receiving elements arranged in 2×2,
The at least two pixel pairs are, of the four light receiving elements sharing the one on-chip lens, a first pixel pair composed of the first light receiving pixel and the second light receiving pixel adjacent in the row direction, and a column a second pixel pair consisting of a third light-receiving pixel and a fourth light-receiving pixel that are adjacent in a direction;
The adder adds the pixel values read from the first light receiving pixels in each of the first pixel pairs, and adds the pixel values read from the second light receiving pixels in each of the first pixel pairs. In addition to the first addition processing of adding the pixel values read from the third light receiving pixels in each of the second pixel pairs, adding the pixel values read from the fourth light receiving pixels in each of the second pixel pairs further executing a second addition process for adding the pixel values obtained,
The generation unit generates information about the image plane phase difference between the first light receiving pixel and the second light receiving pixel based on the plurality of pixel values included in the image signal after the first addition processing. , generating information about an image plane phase difference between the third light-receiving pixel and the fourth light-receiving pixel based on the plurality of pixel values included in the image signal after the second addition processing. Device. The plurality of light-receiving pixels include light-receiving pixels operating with a sensitivity of a first dynamic range and light-receiving pixels operating with a sensitivity of a second dynamic range wider than the first dynamic range,
The at least two pixel pairs are a third pixel pair consisting of the first light-receiving pixel and the second light-receiving pixel that operate with the sensitivity of the first dynamic range, and the pixel pair that operates with the sensitivity of the second dynamic range. and a fourth pixel pair consisting of the first light receiving pixel and the second light receiving pixel,
The first addition process adds the pixel values read from the first light receiving pixels in each of the third pixel pairs, and adds the pixel values read from the second light receiving pixels in each of the third pixel pairs. a third addition process for adding pixel values; adding pixel values read from the first light receiving pixels in each of the fourth pixel pairs; and reading from the second light receiving pixels in each of the fourth pixel pairs. A fourth addition process that adds the pixel values obtained,
The generating unit combines the first total pixel value calculated by the third addition process and the second total pixel value calculated by the fourth addition process, and the The imaging device according to claim 1, wherein the information about the image plane phase difference between the first light receiving pixel and the second light receiving pixel is generated based on a plurality of pixel values. the plurality of light-receiving pixels are grouped into a plurality of pixel blocks each including two or more light-receiving pixels;
The imaging device according to claim 1, wherein the adder adds the pixel value of the first light receiving pixel and the pixel value of the second light receiving pixel in the same pixel block. at least one of the plurality of pixel blocks includes two or more of the pixel pairs;
The reading unit reads one or more pixel values from the first light receiving pixels in the two or more pixel pairs included in the at least one pixel block, and reads the second light receiving pixels in the two or more pixel pairs. read out one or more pixel values from
The adder adds the one or more pixel values read from the first light-receiving pixels in the two or more pixel pairs for each pixel block, and adds the pixel values in the two or more pixel pairs. The imaging device according to Claim 8, wherein the first addition process is performed to add the one or more pixel values read from two light receiving pixels. The pixel array comprises a plurality of unit units, each of which is a repeating unit, each of which is composed of two or more of the pixel blocks;
each of the pixel blocks is composed of the two or more light-receiving pixels that receive light in the same wavelength band;
The imaging device according to claim 8, wherein at least two pixel blocks in each of the unit units are composed of the two or more light-receiving pixels that receive light in different wavelength bands between the pixel blocks. 11. The imaging device according to claim 10, wherein each of said unit units is composed of two said pixel blocks consisting of ten said light receiving pixels and two said pixel blocks consisting of eight said light receiving pixels. 11. The imaging device according to claim 10, wherein each of said unit units is composed of said four pixel blocks each made up of said 16 said light receiving pixels arranged in 4x4. 11. The imaging device according to claim 10, wherein each of said unit units is composed of said four pixel blocks each made up of said nine said light receiving pixels arranged in 3x3. The adder adds the pixel values read from the first light-receiving pixels in at least one pixel block among the plurality of pixel blocks, and adds the pixel values read from the second light-receiving pixels in the at least one pixel block. The imaging apparatus according to Claim 8, wherein a first addition process is executed to add the read pixel values. further comprising a plurality of control lines extending in a row direction with respect to the pixel array and connected to the plurality of light receiving pixels;
the plurality of light-receiving pixels are arranged in a matrix in an oblique direction with respect to the extending direction of each of the control lines;
The imaging device according to claim 1, wherein the pixel pair is composed of the at least two light-receiving pixels that are adjacent in the oblique direction. An electronic device comprising the imaging device according to claim 1 .

Images