JP2021061618A - Imaging device and imaging apparatus - Google Patents

Abstract

To solve the problem in which, since PD signals that are two or more times larger are output from pupil division type focus detection pixels provided on the entire surface of an imaging area, the amount of data transferred from an imaging device to a control unit side of an imaging apparatus increases, thus a storage memory area on the imaging device side increases.SOLUTION: The imaging device includes: a pixel portion in which a plurality of pixels each including a plurality of photoelectric conversion elements is arranged; and an output control unit that adds signals from the photoelectric conversion elements in the pixel of a first region of the pixel portion and outputs them and outputs a signal from the photoelectric conversion element at each pixel in a second region of the pixel portion.SELECTED DRAWING: Figure 6

Description

The present invention relates to an image pickup device and an image pickup apparatus.

There is known an imaging device in which pupil-splitting pixels are provided on the entire surface of an imaging region in an imaging device (see, for example, Patent Document 1). It is known that an image pickup device having an image pickup device using pupil-splitting pixels on the entire surface has a problem that a large amount of data is transferred from the image pickup device.

Japanese Unexamined Patent Publication No. 2010-239337

In the first aspect of the present invention, in the pixel portion in which a plurality of pixels composed of a plurality of photoelectric conversion elements are arranged and the pixels in the first region of the pixel portion, the signals from the photoelectric conversion elements are added in the pixels. An image pickup device is provided that includes an output control unit that outputs signals from the photoelectric conversion element in the pixels in the second region of the pixel unit.

In the second aspect of the present invention, an image pickup device including the image pickup device of the first aspect and a focus control unit that performs autofocus control based on a signal from a photoelectric conversion element included in the second region. Is provided.

The outline of the above invention does not list all the features of the present invention. Sub-combinations of these feature groups can also be inventions.

It is sectional drawing of the back-illuminated type image sensor which concerns on this embodiment. It is a figure which showed typically the light flux incident on PD104. It is a figure explaining the pixel arrangement and the unit group of an image pickup chip. It is a circuit diagram corresponding to the unit group of the image pickup chip. It is a block diagram which shows the structure of the image pickup apparatus which concerns on this embodiment. It is a figure explaining the ROI which concerns on this embodiment. It is a figure explaining the PD signal sequence of ROI which calculates the defocus amount. It is a block diagram which shows the specific structure as an example of a signal processing chip. It is a flowchart explaining the addition process of a PD signal. It is a block diagram which shows the structure of another image pickup apparatus. It is a figure explaining ROI in other image pickup apparatus. It is a figure explaining the addition processing of the PD signal included in ROI. It is a flowchart explaining the addition processing of PD signal of another image pickup chip. It is a figure explaining another ROI.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the inventions that fall within the scope of the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention.

FIG. 1 is a cross-sectional view of the back-illuminated image sensor 100 according to the present embodiment. The image sensor 100 includes an image pickup chip 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are laminated and electrically connected to each other by a conductive bump 109 such as Cu.

As shown in the figure, the incident light is mainly incident in the Z-axis plus direction indicated by the white arrow. In the present embodiment, in the image pickup chip 113, the surface on the side where the incident light is incident is referred to as the back surface. Further, as shown in the coordinate axes, the left direction of the paper surface orthogonal to the Z axis is defined as the X-axis plus direction, and the Z-axis and the front direction of the paper surface orthogonal to the X-axis are defined as the Y-axis plus direction. In some subsequent figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

An example of the image pickup chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back surface side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally, and a transistor 105 provided corresponding to each PD 104. The PD layer 106 is an example of a pixel portion.

A color filter 102 is provided on the incident side of the incident light in the PD layer 106 via the passivation film 103. The color filter 102 has a plurality of types that transmit different wavelength regions from each other, and has a specific arrangement. The arrangement of the color filters 102 will be described later. In one color filter 102, two PD 104s and two transistors 105 are formed side by side in the X-axis direction. Then, one pixel is formed by one color filter 102, two PD 104s, and two transistors. The PD 104 is an example of a photoelectric conversion element. Instead of PD104, a photoelectric conversion element such as an organic photoelectric conversion film or a photogate may be used.

A microlens 101 is provided on the incident side of the incident light in the color filter 102, corresponding to the color filter 102. The microlens 101 collects incident light toward the two corresponding PD 104s. The two PD 104s each receive a luminous flux from a partial region of the incident light that does not coincide with each other in the cross-sectional region. The two PD 104s provided for one microlens 101 form one pixel 126.

Here, the state of the incident luminous flux reaching the pixel 126 will be described. FIG. 2 is a diagram schematically showing a subject luminous flux incident on the PD 104. FIG. 2A is a schematic view seen from the incident direction of the exit pupil 114, and FIG. 2B is a schematic cross-sectional view taken along the line XZ for explaining the subject luminous flux incident on the PD 104. In FIGS. 2A and 2B, the ejection pupil 114 represents the ejection pupil of an interchangeable lens in an interchangeable lens type digital camera, and in a digital camera including a photographing lens fixedly provided to an image sensor. , Represents the ejection pupil of the fixed lens.

Here, a case where the subject luminous flux is cut at the exit pupil surface as one of the cross sections orthogonal to the optical axis of the photographing lens will be described. The region of the subject luminous flux as a whole on the exit pupil surface includes the partial region 116 and the partial region 118. The subject luminous flux passing through the partial region 116 is a part of the subject luminous flux, and this partial luminous flux is received by the PD 124. Further, the subject luminous flux passing through the partial region 118 is a part of the subject luminous flux different from the subject luminous flux passing through the partial region 116, and this partial luminous flux is received by the PD 122. In other words, the partial region 116 represents a region in which the region of the PD 124 is back-projected onto the exit pupil surface, and the partial region 118 represents a region in which the PD 122 is back-projected onto the exit pupil plane. Then, PD122 and PD124, which are a set that receives each partial luminous flux divided into pupils, function as one pixel 126.

Although the partial area 116 and the partial area 118 are separated to the left and right in FIG. 2, the partial area 116 and the partial area 118 may have a partially coincident area. .. Further, the PD 122 and PD 124 are not limited to the case where the outer edge of the photoelectric conversion unit itself is formed so as to match the shape shown in the figure, and even if the outer edge is adjusted by the transmission region of the reflective film formed on the light receiving surface. Alternatively, it may be adjusted by a blocking mask provided with an opening.

Again, with reference to FIG. 1, a plurality of bumps 109 are arranged on the surface of the wiring layer 108 including the wiring 107. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the facing surfaces of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are aligned by pressurization or the like. The bumps 109 are joined together and electrically connected.

Similarly, a plurality of bumps 109 are arranged on the surfaces of the signal processing chip 111 and the memory chip 112 facing each other. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressurized, so that the aligned bumps 109 are joined to each other and electrically connected.

The bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, and micro bump bonding by solder melting may be adopted. Further, for example, about one bump 109 may be provided for one pixel group described later. Therefore, the size of the bumps 109 may be larger than the pitch of the PD 104. Further, in the peripheral region other than the pixel region in which the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel region may be provided together.

The signal processing chip 111 has TSVs (Through Silicon Vias) 110 that connect circuits provided on the front and back surfaces to each other. The TSV 110 is preferably provided in the peripheral region. Further, the TSV 110 may also be provided in the peripheral area of the image pickup chip 113 and the memory chip 112.

FIG. 3 is a diagram illustrating a pixel arrangement of the imaging chip 113 and a unit group 120. FIG. 3 shows a state in which the image pickup chip 113 is observed from the back surface side. More than 20 million pixels are arranged in a matrix in the pixel area. As shown in FIG. 3, the pixel composed of the two PD 104s is provided by dividing one color filter 102 into two in the X-axis direction. In the present embodiment, four adjacent pixels form one unit group 120. The grid lines shown in FIG. 3 show the concept that four adjacent pixels are grouped to form a unit group 120.

As shown in the partially enlarged view of the pixel region, the unit group 120 includes a set of so-called Bayer arrays including four pixels of green pixels Gb, Gr, blue pixels B, and red pixels R. The green pixel is a pixel having a green filter as the color filter 102, and receives light in the green wavelength band among the incident light. Similarly, a blue pixel is a pixel having a blue filter as a color filter 102 and receives light in a blue wavelength band, and a red pixel is a pixel having a red filter as a color filter 102 and receives light in a red wavelength band. Receive light.

FIG. 4 is a circuit diagram corresponding to the unit group 120 of the imaging chip 113. In the figure, a rectangle surrounded by a dotted line represents a circuit corresponding to one pixel. At least a part of each transistor described below corresponds to the transistor 105 of FIG.

As described above, the unit group 120 is formed of 4 pixels. Since each pixel has two PD104s, the unit group 120 has eight PD104s. The eight PD 104s of the unit group 120 are each connected to the transfer transistor 302, and each gate of each transfer transistor 302 is connected to the TX wiring 307 to which the transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to the eight transfer transistors 302.

The drain of each transfer transistor 302 is connected to the source of each corresponding reset transistor 303, and the so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to the Vdd wiring 310 to which the power supply voltage is supplied, and its gate is connected to the reset wiring 306 to which the reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to the eight reset transistors 303.

The drain of each amplification transistor 304 is connected to the Vdd wiring 310 to which the power supply voltage is supplied. Also, the source of each amplification transistor 304 is connected to the drain of each corresponding selection transistor 305. Each gate of the selection transistor 305 is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the eight selection transistors 305. Then, the source of each selection transistor 305 is connected to the common output wiring 309. The load current source 311 supplies current to the output wiring 309. That is, the output wiring 309 for the selection transistor 305 is formed by the source follower. The load current source 311 may be provided on the imaging chip 113 side or the signal processing chip 111 side.

Here, the flow from the start of charge accumulation to the pixel output after the end of accumulation will be described. When a reset pulse is applied to the reset transistor 303 through the reset wiring 306 and at the same time a transfer pulse is applied to the transfer transistor 302 through the TX wiring 307, the potentials of the PD 104 and the floating diffusion FD are reset.

When the application of the transfer pulse is released, the PD 104 converts the received incident light into an electric charge and accumulates it. After that, when the transfer pulse is applied again without the reset pulse being applied, the accumulated charge is transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation. .. Then, when the selection pulse is applied to the selection transistor 305 through the decoder wiring 308, the fluctuation of the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, the pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

As shown in the figure, in the present embodiment, the reset wiring 306 and the TX wiring 307 are common to the four pixels forming the unit group 120. That is, the reset pulse and the transfer pulse are applied to all four pixels at the same time. Therefore, all the pixels forming the unit group 120 start the charge accumulation at the same timing and end the charge accumulation at the same timing. However, the pixel signals corresponding to the accumulated charges are sequentially applied by the selection pulses of the respective selection transistors 305, and are selectively output to the output wiring 309. In the following description, the output signal output from each PD is referred to as a PD signal, and the signal to which the PD signal is added in pixel units is referred to as a pixel signal.

FIG. 5 is a block diagram showing the configuration of the image pickup apparatus 500 according to the present embodiment. The imaging device 500 includes a photographing lens 510 as a photographing optical system, and the photographing lens 510 guides a subject light beam incident along the optical axis OA to the image pickup element 100. The photographing lens 510 may be an interchangeable lens that can be attached to and detached from the image pickup apparatus 500. The image pickup device 500 mainly includes an image pickup element 100, a system control unit 501, a work memory 502, a recording unit 503, a display unit 504, and an operation unit 505.

The photographing lens 510 is composed of a plurality of optical lens groups, and forms a subject light flux from the scene in the vicinity of the focal plane thereof. In FIG. 5, the photographing lens 510 is represented by a virtual single lens arranged in the vicinity of the pupil.

The image sensor 100 delivers the pixel signal for image data generation to the image processing unit 508 of the system control unit 501. The image processing unit 508 performs various image processing using the work memory 502 as a workspace to generate image data. For example, when generating image data in a JPEG file format, a compression process is executed after performing a white balance process, a gamma process, or the like. The generated image data is recorded in the recording unit 503, converted into a display signal, and displayed on the display unit 504 for a preset time.

Further, the image sensor 100 delivers the PD signal for focus detection to the calculation unit 509 of the system control unit 501. The calculation unit 509 generates a pair of signal waveforms from the acquired plurality of PD signal sequences, and calculates a defocus amount which is an amount of deviation from the in-focus state from the pair of signal waveforms. The calculation unit 509 calculates the amount of movement to move the photographing lens 510 to the in-focus position from the calculated defocus amount. The calculation unit 509 generates a drive signal corresponding to the amount of movement of the photographing lens 510. That is, the calculation unit 509 functions as a focus control unit that performs autofocus control based on the PD signal.

The calculation unit 509 outputs the generated drive signal to the lens drive control unit 506. The lens drive control unit 506 drives the lens drive unit 507 according to the drive signal to move the photographing lens 510 forward and backward in the optical axis direction. In this way, the position of the photographing lens 510 is adjusted, and the autofocus control of the photographing lens 510 is executed so as to form a sharp image on the image sensor 100. The calculation unit 509 also executes various calculations for operating the image pickup apparatus 500.

FIG. 6 is a diagram for explaining the ROI according to the present embodiment. ROI is a coined word consisting of the first three letters of Region of interest, and indicates an "interesting area" in the pixel area on the image pickup chip 113, and focus detection is performed in that area. The ROI is set in the pixel area of the image pickup chip 113 in units of the unit group 120.

The ROI is set by the image processing unit 508. The image processing unit 508 uses a predetermined algorithm to image the subject and analyze the generated image data. The image processing unit 508 sets the ROI in the pixel region on the image pickup chip 113 in an arbitrary shape and at an arbitrary position in accordance with the analysis result. The ROI may be a square ROI 10, a rectangular ROI 12, a vertically long ROI 14 or a horizontally long ROI 16, as shown in FIG. The image processing unit 508 outputs the address information indicating the set ROI to the image sensor 100.

In the present embodiment, the ROI is set in units of the unit group 120, but the present invention is not limited to this example, and the ROI may be set so as to include at least one pixel provided in the imaging chip 113. .. Further, as an example of the address information, the start position coordinates and the end position coordinates in the X direction in the XY coordinate system divided by the unit group 120 units in the pixel area of the imaging chip 113, and the start position coordinates and the end position coordinates in the Y direction are used. This is the information to be shown. The ROI is an example of a designated area.

The predetermined algorithm in the image processing unit 508 refers to, for example, a face detection algorithm of a person when the subject is a person. In this case, the image processing unit 508 detects the face of the subject from the image data and identifies the region including the face of the subject in the pixel region. The image processing unit 508 sets the ROI in the area corresponding to the specified pixel area.

The ROI may be set by the operation of the user who uses the image pickup apparatus 500. In this case, the user may be able to input the ROI by operating the operation unit 505, or may be able to select the subject for which the ROI is to be set. When the user selects a subject, the image processing unit 508 specifies the pixel area including the subject by referring to the contrast value of each pixel signal with respect to the position of the selected subject, and the specified pixel. ROI may be set in the area corresponding to the area.

When the ROI is set by the image processing unit 508, the PD signal output from the unit group 120 not included in the ROI is added in pixel units and passed to the image processing unit 508 of the system control unit 501 as a pixel signal. .. As a result, the amount of data transferred from the image sensor 100 to the image processing unit 508 can be reduced.

On the other hand, the PD signal of the unit group 120 included in the ROI is delivered to the calculation unit 509 of the system control unit 501 for the purpose of using it for focus detection. The calculation unit 509 calculates the defocus amount from the PD signal included in the ROI, and adjusts the focus of the photographing lens 510. As described above, the PD signal included in the ROI is different from the PD signal not included in the ROI, and the PD signal is not added in pixel units, and individual PD signals are output.

The calculation unit 509 delivers the PD signals of the two PD 104s for focus detection to the image processing unit 508. The image processing unit 508 adds and processes the PD signals of the two PD 104s, and matches the format with the pixel signals directly output to the image processing unit 508. The image processing unit 508 generates image data from the pixel signal directly output to the image processing unit 508 and the pixel signal generated by the addition processing of the PD signal. The addition processing of the image processing unit 508 may have different characteristics from the addition circuit of the signal processing chip 111. In such a case, it is necessary to correct the addition result in the addition process of the image processing unit 508 so that the addition result is the same as the addition result when the PD signals of the two PD 104s are added by the addition circuit. Further, instead of performing the correction by the addition processing of the image processing unit 508, such correction may be performed by the addition circuit of the signal processing chip 111.

FIG. 7 is a diagram illustrating a PD signal sequence of the ROI for calculating the defocus amount. When calculating the defocus amount, the calculation unit 509 calculates the defocus amount from the PD signal waveforms created from the PD signal sequences of each of the red, green, and blue colors. A method of calculating the defocus amount from the green PD signal will be described as an example of calculating the defocus amount from the green PD signal string. The calculation unit 509 calculates the defocus amount from the PD signal waveforms created from PD134, PD130, and PD138 and the PD signal waveforms created from PD136, PD132, and PD140. In order to actually calculate the defocus amount, for example, a pair of PD signal waveforms created from about 100 PD signal sequences is used, but in the present embodiment, for convenience of explanation, three PD signals are used. It has been described using PD signal waveforms created from columns.

Further, in the example shown in FIG. 7, the calculation unit 509 shows an example of calculating the defocus amount using a pair of horizontal PD signal waveforms in the center of the ROI 10. However, the present invention is not limited to this, and for example, when the defocus amount cannot be calculated using a pair of PD signal waveforms, the calculation unit 509 calculates the defocus amount within the region included in the ROI 10. The position of the waveform may be arbitrarily changed.

FIG. 8 is a block diagram showing a specific configuration as an example of a signal processing chip. The signal processing chip 111 includes a sensor control unit 24, a block control unit 26, a signal control unit 28, and a drive control unit 20 that collectively controls each of these control units as shared control functions. The drive control unit 20 converts the instruction from the system control unit 501 into a control signal that can be executed by each control unit and delivers it to each of them.

The sensor control unit 24 is responsible for controlling the transmission of control pulses related to charge accumulation and charge reading of each pixel to be transmitted to the image pickup chip 113. Specifically, the sensor control unit 24 controls the start and end of charge accumulation by sending a reset pulse and a transfer pulse to the target pixel, and sends a selection pulse to the read pixel. The PD signal is output to the output wiring 309.

The block control unit 26 executes transmission of a specific pulse for specifying the unit group 120 to be controlled, which is transmitted to the image pickup chip 113. As described with reference to FIG. 2 and the like, the four pixels of the green pixels Gb, Gr, the blue pixel B, and the red pixel R form the unit group 120. The eight PD 104s constituting the four pixels included in the unit group 120 form one block. Pixels included in the same block start charge accumulation at the same timing and end charge accumulation at the same timing. Therefore, the block control unit 26 plays a role of blocking the unit group 120 by sending a specific pulse to the target unit group 120 based on the designation from the drive control unit 20. The transfer pulse and reset pulse received by each pixel via the TX wiring 307 and the reset wiring 306 are the logical product of each pulse transmitted by the sensor control unit 24 and a specific pulse transmitted by the block control unit 26.

The signal control unit 28 is mainly responsible for timing control for the A / D converter 34. The PD signals output for each unit group 120 in a predetermined order via the output wiring 309 are input to the A / D converter 34 via the CDS circuit 30 and the multiplexer 32. The A / D converter 34 is controlled by the signal control unit 28 to convert the input PD signal into a digital signal. The PD signal converted into a digital signal is passed to the demultiplexer 36. A pixel memory 38 is provided in the memory chip 112, and the PD signal is stored in the pixel memory 38 from the demultiplexer 36 as a pixel value of digital data in the order of predetermined addresses.

The signal processing chip 111 has a control memory 22. The control memory 22 stores the address information indicating the ROI output from the image processing unit 508. The control memory 22 is composed of, for example, a flash RAM.

As described above, the address information indicating the ROI is set by the image processing unit 508 based on the analysis result obtained by analyzing the image data captured in advance. Then, the drive control unit 20 receives the address information indicating the ROI from the image processing unit 508 and stores it in the control memory 22. Therefore, the drive control unit 20 also functions as a reception unit that receives address information or the like indicating the ROI from the image processing unit 508.

The drive control unit 20 sequentially receives the address information generated by the setting of the ROI executed in synchronization with the generation of the image data of the image processing unit 508 from the image processing unit 508, and stores the address information in the control memory 22. Update the address information. Thereby, for example, in moving image shooting, when the subject is moving, the ROI can be set by following the movement of the subject.

The output control unit 40 reads the address information indicating the ROI from the control memory 22. Further, the output control unit 40 reads the PD signals for each unit group 120 from the pixel memory 38 in the order of the addresses. The output control unit 40 calculates the address of the unit group from the reading order. When the address of the unit group 120 is not included in the ROI, the output control unit 40 outputs the PD signal included in the read unit group to the addition circuit 42. On the other hand, when the unit group 120 is included in the ROI, the output control unit 40 transmits the PD signal included in the read unit group 120 to the calculation unit 509 through the data transfer interface 48 and the data transfer line 512 connected to the calculation unit 509. Output to. The data transfer interface 48 is an example of the second output port.

The addition circuit 42 generates a pixel signal by executing addition processing of PD signals output from two PD 104s provided in the same color filter included in the unit group 120. Further, a data transfer interface 46 for transmitting a pixel signal to the image processing unit 508 is connected to the addition circuit 42. Further, the data transfer interface 46 is connected to a data transfer line 511 connected to the image processing unit 508. The pixel signal generated by the addition circuit 42 is output to the image processing unit 508 using the data transfer interface 46. The data transfer interface 46 provided in the adder circuit 42 is an example of the first output port.

FIG. 9 is a flowchart illustrating the PD signal addition process. The addition process of the pixel signal using ROI will be described with reference to FIG. The PD signal addition process shown in FIG. 9 is started when the image processing unit 508 outputs the address information indicating the ROI to the drive control unit 20.

The drive control unit 20 acquires the address information indicating the ROI from the image processing unit 508 (step S100) and stores it in the control memory 22. The drive control unit 20 sends a reset pulse and a transfer pulse to the image pickup chip 113 to accumulate electric charges, and sends a selection pulse for each unit group in a predetermined order to send a PD signal for each unit group 120. Acquire (step S102). Then, the drive control unit 20 stores the PD signals for each unit group 120 in the pixel memory 38 in a predetermined order.

The output control unit 40 acquires address information indicating the ROI stored in the control memory 22. The output control unit 40 reads the PD signals of the unit group 120 from the pixel memory 38 in a predetermined order (step S104). The output control unit 40 calculates the address of the unit group 120 from the order in which the unit group 120 is read, and determines whether or not the read unit group 120 is included in the ROI (step S106).

When the read unit group 120 is not included in the ROI (step S106: No), the output control unit 40 outputs the PD signal of the unit group 120 to the addition circuit 42. The addition circuit 42 adds the PD signals output from the two PD 104s provided for each color filter in the unit group 120 (step S108). Then, the output control unit 40 outputs the pixel signal generated by the addition processing in the addition circuit to the image processing unit 508 through the addition circuit 42 (step S110).

On the other hand, when the read unit group 120 is included in the ROI (step S106: Yes), the output control unit 40 outputs the PD signal that has not been added to the calculation unit 509 (step S112). ). The calculation unit 509 calculates the defocus amount using the output PD signal.

The output control unit 40 determines whether or not the PD signals of all the unit groups have been read from the pixel memory 38 and output (step S114). When the output control unit 40 has not output the PD signals of all the unit groups 120 (step S114: No), the output control unit 40 returns the process to step S104. Then, the output control unit 40 reads the PD signal of the unit group 120 stored next to the read unit group 120 from the pixel memory 38, and executes the processes of steps S106 to S120. On the other hand, when the output control unit 40 outputs the PD signals of all the unit groups 120 (step S114: Yes), the addition process ends.

The PD signal addition process shown in FIG. 9 is executed when image data is generated while performing autofocus as in live view display and moving image shooting. When capturing a still image of a subject and executing autofocus, the output control unit 40 outputs the PD signal of the unit group 120 included in the ROI to the calculation unit 509 and automatically performs the autofocus to the calculation unit 509. Perform focus. Then, when the main image pickup is executed, the output control unit 40 outputs the PD signals of all the unit groups 120 to the addition circuit 42, reduces the number of pixel signal data by executing the addition process, and then reduces the number of pixel signal data. The image processing unit 508 is made to generate image data.

As described above, even when the pupil division type focus detection pixel obtained by dividing the color filter 102 into two PD 104s is used, by setting the ROI, image processing is performed on the pixels not included in the ROI. The amount of data transferred to unit 508 can be significantly reduced. As described above, even in the image sensor having the pupil-divided focus detection pixels, it is possible to suppress an increase in the amount of data transferred from the image sensor 100 to the image sensor 500 side. As a result, it is possible to suppress an increase in the storage memory area on the image pickup apparatus 500 side.

Further, in the image pickup apparatus 500, high-speed data transfer is possible between the chips in the image pickup device 100 via the bump 109. On the other hand, since the image sensor 100 and the system control unit 501 are arranged apart from each other, the data transfer rate between the image sensor 100 and the system control unit 501 is slow. Therefore, by executing the addition process in the image sensor 100 capable of high-speed data transfer to reduce the number of pixel signal data, the image sensor 100 does not increase the transfer time and the processing time in the image sensor 100. The data transfer time between 100 and the system control unit 501 can be significantly reduced.

Further, even when the PD signal is captured in the pixel memory 38 a plurality of times, the PD signal obtained by capturing the PD signal stored in the pixel memory 38 a plurality of times is added to the PD signal by using an adder circuit. Only by sequentially adding, it is possible to deal with the situation without changing the configurations of the signal processing chip 111, the memory chip 112, and the imaging chip 113. Therefore, even when the image pickup is performed a plurality of times, the configuration of the image sensor 100 can be handled without being complicated and large-scale.

Further, even when a plurality of ROIs are set, it can be handled without significantly changing the configurations of the signal processing chip 111, the memory chip 112, and the imaging chip 113. Therefore, even when a plurality of ROIs are set, the configuration of the image sensor 100 can be dealt with without being complicated and large-scale.

Further, in the present embodiment, the calculation unit 509 shows an example of selecting a PD signal suitable for calculating the defocus amount from the region included in the ROI from the characteristics of the subject, but the output control unit 40 describes the PD signal. The ROI may be changed by analyzing the PD signal output from the ROI. An example of PD signal analysis is the measurement of the frequency component of the PD signal, and the region where the PD signal analyzed by the output control unit 40 is output is an example of the analysis region included in the ROI.

For example, when ROI 14 is set, the output control unit 40 performs a subtraction process using PD signals output from ROI 14 that are horizontally adjacent to each other in the same color and the same aperture. Here, the same aperture means that the position of the PD 104 with respect to the micro lens 101 is the same. In the example shown in FIG. 7, the subtraction process is performed using the PD 134, the PD 130, and the PD signal output from the PD 138, which are the same green pixels and are located on the left side of the microlens 101. Then, when the absolute value of the value calculated by the subtraction process is smaller than the predetermined value, the output control unit 40 determines that the frequency component of the PD signal included in the ROI is small. When it is determined that the frequency component is small, the output control unit 40 may increase the ROI or change the aspect ratio while maintaining the center of the ROI. For example, when it is determined that the frequency component of the PD signal in the ROI 14 is small, the output control unit 40 may change the ROI 14 to the ROI 16. Then, when the ROI is changed, the output control unit 40 stores the information indicating the address of the ROI 16 in the control memory 22. In this case, the output control unit 40 functions as a change unit for changing the ROI.

In the present embodiment, the output control unit 40 reads the PD signal of the unit group 120 before addition from the pixel memory 38, outputs the PD signal not included in the ROI to the addition circuit 42, and PD included in the ROI. An example of outputting a signal to the calculation unit 509 is shown. However, the output control unit 40 determines whether or not the PD signal of the unit group 120 output from the demultiplexer 36 is included in the ROI, and executes an addition process for the PD signal not included in the ROI to pixel. The PD signal included in the ROI may be stored in the pixel memory 38 as a signal without executing the addition process. Then, the output control unit 40 may read the pixel signal and the PD signal from the pixel memory 38 and output them to the image processing unit 508 and the calculation unit 509 in response to the delivery request of the image processing unit 508 and the calculation unit 509.

In this embodiment, an example of calculating the defocus amount using the red, green, and blue PD signals contained in the ROI is shown. However, a color that is not subjected to addition processing may be specified, and the defocus amount may be calculated using the PD signal of the specified specific color. As a result, even if the PD signal is included in the ROI, the PD signal of a color different from the designated color can be added and processed, so that the amount of data transferred to the image processing unit 508 can be further reduced.

The method of specifying the color to execute the addition process will be described. The image processing unit 508 specifies a specific color for calculating the defocus amount from the signal intensities of the pixels of each color included in the ROI when the subject is a single color of the specific color. For example, when the subject is a solid red image, the PD signal of the green pixel does not appear, so the defocus amount cannot be calculated from the PD signal of the green pixel. Therefore, in such a case, the image processing unit 508 detects that the signal strength of the green pixel is weak and the signal of the red pixel is strong, and designates the color of the PD signal for calculating the next defocus amount as red. Then, the information indicating the color is output to the image sensor 100.

In the image sensor 100, the drive control unit 20 receives information indicating color from the image processing unit 508. The drive control unit 20 stores the information indicating the color as well as the address information indicating the ROI in the control memory 22.

The output control unit 40 acquires information indicating the color stored in the control memory 22. The output control unit 40 reads the PD signals of the unit group 120 from the pixel memory 38 in a predetermined order. When the color of the read PD signal matches the information indicating the color stored in the control memory 22, the output control unit 40 adds the color information indicating the color to the header area of the PD signal and does not add the individual individual. The PD signal is output to the calculation unit 509. When the color of the read PD signal does not match the color indicating information stored in the control memory 22, the output control unit 40 outputs the read PD signal to the addition circuit 42 to execute the addition process, and performs image processing. Output to unit 508. The color information indicating the color is referred to when the image data is generated by the addition processing by the image processing unit 508 after the defocus calculation.

The image processing unit 508 outputs information for canceling the color designation to the image sensor 100 when the signal strength of the green pixel is restored. In the image sensor 100, the drive control unit 20 receives information for canceling the color designation from the image processing unit 508. The drive control unit 20 erases the information indicating the color from the control memory 22.

FIG. 10 is a block diagram showing the configuration of another imaging device 600. In FIG. 10, the same elements as those in FIG. 5 are assigned the same reference numbers, and duplicate description will be omitted. The image pickup device 600 shown in FIG. 10 includes an image pickup device 602 having a configuration different from that of the image pickup device 100.

FIG. 11 is a diagram illustrating ROI in another imaging device 600. The configuration of the image pickup device 602 and the arrangement of the ROI in the image pickup apparatus 600 will be described with reference to FIG. In the image pickup chip 606, each pixel includes four PD 104s. As shown in FIG. 11, the pixel group composed of four PD 104s divides one color filter 102 into two in the X-axis direction and two in the Y-axis direction orthogonal to the X-axis direction. It is provided. Further, also in the image pickup chip 606, the unit group 612 is composed of four pixels of green pixels Gb, Gr, blue pixels B, and red pixels R. Since each pixel comprises 4 PD104s, the unit group 612 has 16 PD104s.

In the imaging chip 606 shown in FIG. 11, similarly to FIG. 6, the image processing unit 508 can arbitrarily obtain an ROI of any shape such as a square ROI 50, a rectangular ROI 52, a ROI 54 long in the vertical direction, or a ROI 56 long in the horizontal direction. Set to the position of. In the present embodiment, the PD signals of the unit group 612, which are not included in the ROI, are added with the PD signals output from the four PD 104s provided in one color filter 102 and output to the image processing unit 508.

The PD signal of the unit group 612 included in the ROI is output to the calculation unit 509 without being added. The calculation unit 509 calculates the defocus amount using the output PD signal. The calculation unit 509 can calculate the amount of defocus in the horizontal direction by using a pair of PD signal waveforms composed of PD signal sequences having different opening positions in the horizontal direction. Similarly, the calculation unit 509 can calculate the amount of defocus in the vertical direction using a pair of PD signal waveforms composed of PD signal sequences having different opening positions in the vertical direction, and PD signal sequences having different opening positions in the diagonal direction. The amount of defocus in the diagonal direction can be calculated using a pair of PD signal waveforms composed of. The image pickup apparatus 600 includes pixels divided into four for one color filter 102. As a result, the calculation unit 509 can calculate the defocus amount in the horizontal direction, the vertical direction, and the diagonal direction.

Next, a configuration will be described in which the PD signals of the unit group 612 included in the ROI are also added and processed to further reduce the amount of data transfer. The image pickup apparatus 600 may add the PD signals of the unit group 612 included in the ROI by determining the direction for calculating the defocus amount, and further reduce the data transfer amount.

FIG. 12 is a diagram illustrating an addition process of PD signals included in the ROI. FIG. 12A shows the concept of addition processing of the output PD signal when calculating the defocus amount in the vertical direction. FIG. 12B shows the concept of addition processing of the output PD signal when calculating the defocus amount in the horizontal direction.

In the case shown in FIG. 12A, the PD signals output from the four PD 104s provided in one color filter 102 are added in the horizontal direction intersecting the vertical direction for calculating the defocus amount. .. For example, in the red pixel R, the PD 160 and the PD signal output from the PD 162 are added to generate the horizontal addition PD signal 170, and the PD signals output from the PD 164 and PD 166 are added to form the horizontal addition PD signal 172. Generated and output. Then, the calculation unit 509 calculates the defocus amount from the pair of signal waveforms composed of the PD signal sequence including the horizontal addition PD signal 170 in the vertical direction and the PD signal sequence including the horizontal addition PD signal 172.

In the case shown in FIG. 12B, the PD signals output from the four PD 104s provided in one color filter 102 are added in the vertical direction intersecting the horizontal direction for calculating the defocus amount. .. For example, in the red pixel R, the PD 160 and the PD signal output from the PD 164 are added to generate the vertically added PD signal 174, and the PD signals output from the PD 162 and the PD 166 are added to form the vertically added PD signal 176. Generated and output. Then, the calculation unit 509 calculates the defocus amount from the pair of signal waveforms composed of the PD signal sequence including the vertically added PD signal 174 in the horizontal direction and the PD signal sequence including the vertically added PD signal 176.

When the direction for calculating the defocus amount is an oblique direction, the PD signal in the same direction as the relevant direction is added without adding the PD signal in the direction intersecting the relevant direction. Then, the calculation unit 509 calculates the defocus amount from the pair of signal waveforms created from the PD signal sequence output without addition.

Next, a method of determining the direction for calculating the defocus amount will be described. The image processing unit 508 takes in the added pixel signal in the designated ROI region, performs Fourier transform on the pixel signal of the subject, and identifies the direction in which the frequency component in the subject space is large. The image processing unit 508 determines the direction in which the frequency component in the subject space is large in the direction in which the defocus amount is calculated.

Further, the image processing unit 508 calculates the defocus amount by performing subtraction processing using the PD signals output from the ROI, which are adjacent PD signals at the same color and the same aperture, instead of the Fourier transform. You may decide the direction to do it. The image processing unit 508 performs subtraction processing using PD signals adjacent to each other in the horizontal direction, the vertical direction, and the diagonal direction, respectively, and the direction in which the absolute value of the calculated value is large is the direction in which the frequency component in the subject space is large. Identify as. The image processing unit 508 may determine the direction in which the frequency component in the subject space is large in the direction in which the defocus amount is calculated.

The image processing unit 508 specifies the addition direction of the PD signal included in the ROI based on the direction in which the defocus amount is calculated. For example, when the direction in which the defocus amount is calculated is determined to be the vertical direction, the image processing unit 508 specifies the addition direction of the PD signal in the horizontal direction. Further, for example, when the direction for calculating the defocus amount is determined to be the horizontal direction, the image processing unit 508 specifies the addition direction of the PD signal in the vertical direction. As a result, the amount of data transfer can be reduced without deteriorating the detection accuracy of the amount of defocus. The image processing unit 508 outputs information indicating the addition direction of the PD signal to the image sensor 602. The information indicating the addition direction of the PD signal designated based on the direction in which the frequency component in the subject space is large is an example of the subject information.

FIG. 13 is a flowchart illustrating an addition process of PD signals of another imaging device 600. Of the addition processes shown in FIG. 13, the processes of step S200, step S204, step S206 and step S208 are the same as the processes of step S100, step S102, step S104 and step S106 in the addition process shown in FIG. The explanation is omitted.

The drive control unit 20 acquires information indicating the addition direction from the image processing unit 508 (step S202). The drive control unit 20 stores the acquired information indicating the addition direction in the control memory 22.

In step S208, when the read unit group 612 is not included in the ROI (step S208: No), the output control unit 40 outputs the PD signal of the unit group 120 to the addition circuit 42. .. In the addition circuit 42, the PD signals output from the four PD 104s provided for each color filter 102 in the unit group 612 are added (step S210). Then, the output control unit 40 outputs the addition-processed PD signal to the image processing unit 508 through the addition circuit 42 (step S212).

On the other hand, in step S208, when the read unit group 612 is included in the ROI (step S208: Yes), the output control unit 40 indicates information indicating the addition direction stored in the control memory 22. It is determined whether or not the addition direction of the PD signal is the vertical direction with reference to (step S214). When the information indicating the addition direction stored in the control memory 22 is the information indicating the vertical direction (step S214: Yes), the output control unit 40 has the output control unit 40 in the direction perpendicular to the header area of the PD signal. The PD signal of the unit group 612 is output to the addition circuit 42 by adding the information indicating the addition to. The addition circuit 42 adds the PD signals output from the four PD 104s in the vertical direction with reference to the header area (step S216). The PD signal added in the addition circuit 42 is output to the calculation unit 509 using the data transfer interface 48 (step S220).

On the other hand, when the information indicating the addition direction stored in the control memory 22 is the information indicating the horizontal direction (step S214: No), the output control unit 40 puts the output control unit 40 in the header area of the PD signal. Add information indicating that the addition is done in the horizontal direction. The output control unit 40 outputs the PD signal of the unit group 612 to the addition circuit 42. The addition circuit 42 adds the PD signals output from the four PD 104s in the horizontal direction with reference to the header area (step S218). The PD signal added in the addition circuit 42 is output to the calculation unit 509 using the data transfer interface 48 (step S220).

In this way, the output control unit 40 determines whether to add in the horizontal direction or in the vertical direction based on the information indicating the addition direction corresponding to the direction in which the defocus amount is calculated. Then, the output control unit 40 adds the PD signals output from the four PD 104s in the direction determined by the addition circuit 42, and outputs the PD signals to the calculation unit 509.

Following step S220, the output control unit 40 determines whether or not the PD signals of all the unit groups 612 have been output from the pixel memory 38 (step S222). When the output control unit 40 has not output the PD signals of all the unit groups 612 (step S222: No), the output control unit 40 returns the process to step S206. Then, the output control unit 40 reads the PD signal of the unit group 612 stored next to the unit group 612 read last time, and repeatedly executes the processes of steps S206 to S222. On the other hand, when the output control unit 40 outputs the PD signals of all the unit groups 120 (step S222: Yes), the addition process ends.

In this way, the image processing unit 508 determines the direction in which the defocus amount is calculated in the direction in which the frequency component in the pixel signal of the subject acquired in advance is large, and the detection accuracy of the defocus amount with respect to the direction in which the defocus amount is calculated. PD signals are added in a direction that does not decrease. In this way, by determining the addition direction of the PD signal corresponding to the direction in which the defocus amount is calculated, the data transfer amount of the PD signal included in the ROI can be further increased without deteriorating the calculation accuracy of the defocus amount. Can be reduced. In this case, the PD signal included in the ROI is added so as to be different from the PD signal not included in the ROI.

As described above, even in the pupil division type focus detection pixel in which the color filter 102 is divided into four PD 104s, by setting the ROI, the PD signal not included in the ROI can be sent to the image processing unit 508. The amount of data to be transferred can be reduced to one-fourth. Further, even in the PD signal included in the ROI, the amount of data transferred to the image processing unit 508 can be halved. As a result, it is possible to suppress an increase in the amount of data transferred from the image sensor 602 to the image sensor 600 side.

In the flowchart shown in FIG. 13, as an example of the PD signal addition process, an example in which the addition direction is an alternative process of the vertical direction and the horizontal direction is shown. However, the PD signal addition process is not limited to the alternative process, and may be a process that can perform the addition process in various directions including an oblique direction.

FIG. 14 is a diagram illustrating another ROI. The ROI 60 to be set will be described with reference to FIG. 14 in a state where the drive control unit 20 does not receive the address information indicating the ROI from the image processing unit 508.

For example, in a state immediately after the power of the image pickup apparatus 500 is turned on by the user, the drive control unit 20 does not receive the address information indicating the ROI from the image processing unit 508. In this case, the address information of the ROI 60 shown in FIG. 14 is stored in the control memory 22 in advance, and the output control unit 40 reads the address information.

The output control unit 40 executes the above-mentioned addition process based on the address information of the ROI 60 shown in FIG. The calculation unit 509 can select the PD signal of the unit group 120 in the ROI 60 corresponding to the defocus region determined after the fact, and calculate the defocus amount from the pair of signal waveforms created from the PD signal sequence.

With this configuration, even in the state before the ROI is set by the image processing unit 508, the discretely arranged ROIs 60 as shown in FIG. 14 are set in advance, so that the image sensor 100 It is possible to suppress an increase in the amount of data transferred from the image sensor to the image sensor 500.

In the image pickup apparatus 600, an example is shown in which one color filter 102 is divided into two in the X-axis direction and two in the Y-axis direction to provide the PD 104, but the present invention is not limited to this configuration. Considering that the purpose is to calculate the defocus amount, one color filter 102 may be divided into at least two in the X-axis direction and at least two in the Y-axis direction to provide the PD 104.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The order of execution of operations, procedures, steps, steps, etc. in the devices, systems, programs, and methods shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

10, 12, 14, 16, 50, 52, 54, 56, 60 ROI, 20 Drive control unit, 22 control memory, 24 sensor control unit, 26 block control unit, 28 signal control unit, 30 CDS circuit, 32 multiplexer, 34 A / D converter, 36 demultiplexer, 38 pixel memory, 40 output control unit, 42 adder circuit, 46, 48 data transfer interface, 100 imaging element, 101 microlens, 102 color filter, 103 passion film, 104, 122 , 124, 130, 132, 134, 136, 138, 140, 160, 162, 164, 166 PD, 105 Transistor, 106 PD Layer, 107 Wiring, 108 Wiring Layer, 109 Bump, 110 TSV, 111 Signal Processing Chip, 112 Memory chip, 113, 606 imaging chip, 114 ejection pupil, 116, 118 partial area, 120, 612 unit group 126 pixels, 170, 172 horizontal addition PD signal, 174, 176 vertical addition PD signal, 302 transfer transistor, 303 reset transistor , 304 amplification transistor, 305 selection transistor, 306 reset wiring, 307 TX wiring, 308 decoder wiring, 309 output wiring, 310 Vdd wiring, 311 load current source, 500, 600 image pickup device, 501 system control unit, 502 work memory, 503 Recording unit, 504 display unit, 505 operation unit, 506 lens drive control unit, 507 lens drive unit, 508 image processing unit, 509 arithmetic unit, 510 photographing lens, 511, 512 data transfer line, 602 imaging element

Claims (1)

A pixel unit in which a plurality of pixels composed of a plurality of photoelectric conversion elements are arranged and
Outputs in which signals from the photoelectric conversion element are added and output in the pixels in the first region of the pixel portion, and signals from the photoelectric conversion element are output in the pixels in the second region of the pixel portion. An image sensor including a control unit.

Images