JP2021097413A - Imaging element and imaging device - Google Patents

Abstract

To properly process photoelectric conversion data.SOLUTION: A signal processing apparatus 112 comprises: a data input unit 413 which inputs data from a plurality of pixels for performing photoelectric conversion; a memory 414 which stores the data inputted to the data input unit 413 by pixels; an addition unit 416 which adds the data stored in the memory 414 and the data input to the data input unit 413 by pixels; and a data output unit 415 which outputs the data input to the data input unit 413 or the data added by the addition unit 416.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image pickup device and an image pickup apparatus.

An electronic device including an image pickup device in which a back-illuminated image pickup chip and a signal processing chip are laminated (hereinafter, referred to as a stacked image pickup device) has been proposed (see Patent Document 1). In the stacked image sensor, the back-illuminated image sensor and the signal processing chip are laminated so as to be connected to each predetermined region via micro bumps.

Japanese Unexamined Patent Publication No. 2006-49361

In a conventional electronic device equipped with a stacked image sensor, there are not many proposals to divide an image into blocks having one or more of the above regions and acquire an image captured for each block, and the electronic device provided with a stacked image sensor is provided. The usability of the electronic device was not sufficient. Further, the conventional technique of synthesizing a plurality of images separately captured under different exposure conditions is not suitable for photographing a moving subject.

The image pickup device according to the first aspect of the present invention comprises a plurality of pixels arranged side by side in the row direction and the column direction, and a first signal output from the first pixel to the first output wiring among the plurality of pixels. The first conversion unit that converts a digital signal and the second signal output from the second pixel arranged at a position on the column direction side from the first pixel among the plurality of pixels to the second output wiring are digitally output. A second conversion unit that converts a signal, a first adder that performs addition processing on the first signal converted into a digital signal by the first conversion unit, and the digital signal converted by the second conversion unit. A second adder that performs addition processing on the second signal is provided.
The image pickup device according to the second aspect of the invention includes an image pickup device according to the first aspect.

According to the present invention, photoelectric conversion data can be appropriately processed.

It is sectional drawing of the laminated type image sensor. It is a figure explaining the pixel arrangement and the unit area of an image pickup chip. It is a circuit diagram corresponding to the unit area of an image pickup chip. It is a block diagram which shows the functional structure of an image sensor. It is a figure explaining the flow of the pixel signal per pixel. It is a block diagram which illustrates the structure of the image pickup apparatus which has an image pickup element. It is a figure which illustrates the region of interest and the peripheral region in an image sensor. It is a figure explaining the reading timing, the storage signal, and the pixel signal read from an image sensor via an arithmetic circuit. It is a flowchart explaining the flow of the photographing operation executed by the control unit of 1st Embodiment. It is a flowchart explaining the flow of the photographing operation executed by the control unit of the 2nd Embodiment.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
(First Embodiment)
<Explanation of stacked image sensor>
First, the stacked image sensor 100 mounted on an electronic device (for example, an image pickup device 1) according to the first embodiment of the present invention will be described. The stacked image sensor 100 is described in Japanese Patent Application No. 2012-139026 previously filed by the applicant of the present application. FIG. 1 is a cross-sectional view of the stacked image sensor 100. The image sensor 100 includes a back-illuminated image pickup chip 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are laminated and electrically connected to each other by a conductive bump 109 such as Cu.

As shown in the figure, the incident light is mainly incident in the Z-axis plus direction indicated by the white arrow. In the present embodiment, in the image pickup chip 113, the surface on the side where the incident light is incident is referred to as the back surface. Further, as shown in the coordinate axes, the left direction of the paper surface orthogonal to the Z axis is defined as the X-axis plus direction, and the Z-axis and the front direction of the paper surface orthogonal to the X-axis are defined as the Y-axis plus direction. In some subsequent figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

An example of the image pickup chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back surface side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and accumulating charges according to incident light, and a transistor 105 provided corresponding to the PD 104.

A color filter 102 is provided on the incident side of the incident light in the PD layer 106 via the passivation film 103. The color filter 102 has a plurality of types that transmit different wavelength regions from each other, and has a specific arrangement corresponding to each of the PD 104. The arrangement of the color filters 102 will be described later. A set of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

A microlens 101 is provided on the incident side of the incident light in the color filter 102 corresponding to each pixel. The microlens 101 collects incident light toward the corresponding PD 104.

The wiring layer 108 has a wiring 107 that transmits a pixel signal from the PD layer 106 to the signal processing chip 111. The wiring 107 may have multiple layers, and may be provided with passive elements and active elements.

A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the facing surfaces of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are aligned by pressurization or the like. The bumps 109 are joined together and electrically connected.

Similarly, a plurality of bumps 109 are arranged on the surfaces of the signal processing chip 111 and the memory chip 112 facing each other. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressurized, so that the aligned bumps 109 are joined to each other and electrically connected.

The bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, and micro bump bonding by solder melting may be adopted. Further, for example, about one bump 109 may be provided for one unit area described later. Therefore, the size of the bumps 109 may be larger than the pitch of the PD 104. Further, in the peripheral region other than the pixel region in which the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel region may be provided together.

The signal processing chip 111 has TSVs (Through Silicon Vias) 110 that connect circuits provided on the front and back surfaces to each other. The TSV 110 is preferably provided in the peripheral region. Further, the TSV 110 may also be provided in the peripheral area of the image pickup chip 113 and the memory chip 112.

FIG. 2 is a diagram illustrating a pixel arrangement and a unit area 131 of the image pickup chip 113. In particular, the state in which the image pickup chip 113 is observed from the back surface side is shown. For example, 20 million or more pixels are arranged in a matrix in the pixel area. In the present embodiment, for example, 16 pixels of adjacent 4 pixels × 4 pixels form one unit region 131. The grid lines in the figure show the concept that adjacent pixels are grouped to form a unit region 131. The number of pixels forming the unit region 131 is not limited to this, and may be about 1000, for example, 32 pixels × 64 pixels, more or less.

As shown in the partially enlarged view of the pixel region, the unit region 131 includes four so-called Bayer arrays including four pixels of green pixels Gb, Gr, blue pixels B, and red pixels R in the vertical and horizontal directions. The green pixel is a pixel having a green filter as the color filter 102, and receives light in the green wavelength band among the incident light. Similarly, a blue pixel is a pixel having a blue filter as a color filter 102 and receives light in a blue wavelength band, and a red pixel is a pixel having a red filter as a color filter 102 and receives light in a red wavelength band. Receive light.

In the present embodiment, a plurality of blocks are defined so that each block includes at least one unit area 131, and each block can control the pixels included in each block with different control parameters. That is, it is possible to acquire an imaging signal having different imaging conditions between the pixel group included in a certain block and the pixel group included in another block. Examples of control parameters are frame rate, gain, thinning rate, number of rows or columns to be added to add pixel signals, charge accumulation time or number of times, digitization bits, and the like. Further, the control parameter may be a parameter in image processing after acquiring an image signal from a pixel.

FIG. 3 is a circuit diagram corresponding to the unit region 131 of the image pickup chip 113. In FIG. 3, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. At least a part of each transistor described below corresponds to the transistor 105 of FIG.

As described above, the unit region 131 is formed from 16 pixels. The 16 PD 104s corresponding to the respective pixels are connected to the transfer transistor 302, and each gate of each transfer transistor 302 is connected to the TX wiring 307 to which the transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

The drain of each transfer transistor 302 is connected to the source of each corresponding reset transistor 303, and the so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to the Vdd wiring 310 to which the power supply voltage is supplied, and its gate is connected to the reset wiring 306 to which the reset pulse is supplied. In the present embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

The drain of each amplification transistor 304 is connected to the Vdd wiring 310 to which the power supply voltage is supplied. Also, the source of each amplification transistor 304 is connected to the drain of each corresponding selection transistor 305. Each gate of the selection transistor 305 is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. Then, the source of each selection transistor 305 is connected to the common output wiring 309. The load current source 311 supplies current to the output wiring 309. That is, the output wiring 309 for the selection transistor 305 is formed by the source follower. The load current source 311 may be provided on the imaging chip 113 side or the signal processing chip 111 side.

Here, the flow from the start of charge accumulation to the pixel output after the end of accumulation will be described. When a reset pulse is applied to the reset transistor 303 through the reset wiring 306 and at the same time a transfer pulse is applied to the transfer transistor 302 through the TX wiring 307, the potentials of the PD 104 and the floating diffusion FD are reset.

When the application of the transfer pulse is released, the PD 104 converts the received incident light into an electric charge and accumulates it. After that, when the transfer pulse is applied again without the reset pulse being applied, the accumulated charge is transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation. .. Then, when the selection pulse is applied to the selection transistor 305 through the decoder wiring 308, the fluctuation of the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, the pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

As shown in FIG. 3, in the present embodiment, the reset wiring 306 and the TX wiring 307 are common to the 16 pixels forming the unit region 131. That is, the reset pulse and the transfer pulse are applied to all 16 pixels at the same time. Therefore, all the pixels forming the unit region 131 start the charge accumulation at the same timing and end the charge accumulation at the same timing. However, the pixel signal corresponding to the accumulated charge is selectively output from the output wiring 309 by sequentially applying the selection pulse to each selection transistor 305. Further, the reset wiring 306, the TX wiring 307, and the output wiring 309 are separately provided for each unit area 131.

By configuring the circuit with the unit region 131 as a reference in this way, the charge accumulation time can be controlled for each unit region 131. In other words, pixel signals with different frame rates can be output between the unit regions 131. Furthermore, while one unit region 131 is made to perform one charge accumulation, the other unit region 131 is made to repeat charge accumulation many times to output a pixel signal each time. It is also possible to output each frame for moving images at different frame rates between the unit areas 131.

FIG. 4 is a block diagram showing a functional configuration of the image sensor 100. The analog multiplexer 411 sequentially selects 16 PD 104s forming the unit area 131, and outputs each pixel signal to the output wiring 309 provided corresponding to the unit area 131. The multiplexer 411 is formed on the imaging chip 113 together with the PD 104.

The pixel signal output via the multiplexer 411 is CDS and A / by a signal processing circuit 412 formed on the signal processing chip 111 that performs correlated double sampling (CDS) analog / digital (A / D) conversion. D conversion is performed. The A / D converted pixel signal is passed to the demultiplexer 413. The pixel signal output from the demultiplexer 413 is input to the adder 416 corresponding to each pixel. The adder 416 adds the pixel signal output from the demultiplexer 413 and the pixel signal read from the pixel memory 414 in correspondence with each pixel, and outputs the added pixel signal to the pixel memory 414 again. To do.

The pixel memory 414 stores the pixel signal from the adder 416. Each of the pixel memories 414 has a capacity capable of storing the pixel signal after addition. The demultiplexer 413, the adder 416, and the pixel memory 414 are formed on the memory chip 112.

FIG. 5 is a diagram illustrating a flow of pixel signals per pixel. In FIG. 5, the pixel signal S output from the demultiplexer 413 is input to the corresponding adder n in the adder 416. At this time, the pixel signal P stored in the corresponding memory n of the pixel memory 414 is read from the memory n and input to the adder n.

The adder n adds the input pixel signal S and the pixel signal P, and outputs the added pixel signal S + P to the pixel memory n. The pixel memory n stores the input pixel signal S + P and waits for it to be read out to the arithmetic circuit 415. Here, only the pixel signal S input to the adder n is obtained by controlling the pixel memory 414 so that the pixel signal P stored in the pixel memory n is not read when the addition is performed by the adder n. Can be output from the adder n to the pixel memory n as it is. That is, the pixel signal S from the image pickup chip 113 can be directly read from the memory n to the arithmetic circuit 415 without being added by the adder n.

The arithmetic circuit 415 processes the pixel signal stored in the pixel memory 414 and hands it over to the image processing unit in the subsequent stage. The arithmetic circuit 415 may be provided on the signal processing chip 111 or the memory chip 112.

The drive control unit 417 determines the timing at which the pixel signal is sent from the image pickup chip 113 to the signal processing chip 111 and the memory chip 112, the read / store timing of the pixel signal in the pixel memory 414, and the addition timing of the pixel signal in the adder 416. , A timing control signal is generated in order to synchronize with the transfer timing of the pixel signal to the arithmetic circuit 415.

In addition, although FIG. 4 shows the connection for one unit area 131, in reality, these exist for each unit area 131 and operate in parallel. However, the arithmetic circuit 415 does not have to exist for each unit area 131. For example, even if one arithmetic circuit 415 processes sequentially while referring to the values of the pixel memory 414 corresponding to each unit area 131 in order. Good.

As described above, the output wiring 309 is provided corresponding to each of the unit regions 131. Since the image sensor 100 has an image pickup chip 113, a signal processing chip 111, and a memory chip 112 stacked on top of each other, by using an electrical connection between the chips using bumps 109 for these output wirings 309, each chip is placed in the plane direction. Wiring can be routed without making it large.

<Explanation of imaging device>
FIG. 6 is a block diagram illustrating the configuration of the image pickup apparatus 1 having the image pickup device 100 described above. In FIG. 6, the image pickup apparatus 1 includes an image pickup optical system 10, an image pickup unit 20, an image processing unit 30, a work memory 40, a display unit 50, a recording unit 60, and a control unit 70.

The imaging optical system 10 is composed of a plurality of lenses, and guides a light flux from the field of view to the imaging unit 20. The image pickup optical system 10 may be integrally configured with the image pickup device 1 or may be interchangeably configured with respect to the image pickup device 1. Further, the imaging optical system 10 may have a built-in focus lens or a built-in zoom lens.

The image pickup unit 20 includes the above-mentioned image pickup element 100 and a drive unit 21 that drives the image pickup element 100. By driving and controlling the image sensor 100 by the control signal output from the driving unit 21, the above-mentioned block unit can be independently accumulated and controlled. The control unit 70 gives instructions such as the position, shape, and range of the block to the drive unit 21.

The image processing unit 30 cooperates with the work memory 40 to perform image processing on the image data captured by the image capturing unit 20. In the present embodiment, the image processing unit 30 also performs detection processing of a main subject included in the image in addition to normal image processing (color signal processing, gamma correction, etc.). The main subject can be detected by the image processing unit 30 by using a known face detection function. Further, in addition to face detection, for example, as described in Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940), the human body included in the image may be detected as the main subject.

The work memory 40 temporarily stores image data before and after JPEG compression and before and after MPEG compression. The display unit 50 is composed of, for example, a liquid crystal display panel 51, and displays images (still images and moving images) captured by the image pickup unit 20 and various information, and displays an operation input screen. The display unit 50 has a configuration in which the touch panel 52 is laminated on the display surface of the liquid crystal display panel 51. The touch panel 52 outputs a signal indicating a position where the user touches the liquid crystal display panel 51.

The recording unit 60 stores various data such as image data in a storage medium such as a memory card. The control unit 70 has a CPU and controls the entire operation by the image pickup device 1. In the present embodiment, the control unit 70 divides the imaging surface of the image sensor 100 (imaging chip 113) into a plurality of blocks, and acquires images at different frame rates (accumulation time) and gains between the blocks. For this purpose, the control unit 70 instructs the drive unit 21 of the position, shape, range, and control parameters for each block.

Further, the control unit 70 calculates the focus adjustment state by the imaging optical system 10 by the AF calculation unit 71. The control unit 70 further performs an exposure calculation by the AE and AWB calculation units 72 so that an appropriate exposure can be obtained.

<Area of interest and surrounding area>
In the present embodiment, the concept of the area of interest and the peripheral area is introduced in the screen to correspond to the above-mentioned plurality of blocks. FIG. 7 is a diagram illustrating a region of interest 80 and a peripheral region 90 in the image sensor 100 (imaging chip 113). The control unit 70 determines the positions of the region of interest 80 and the peripheral region 90 on the image sensor 100 (imaging chip 113) through scene recognition based on the live view image.

Here, the live view image is also called a preview image before the main imaging is performed, and refers to a monitor image acquired by the image sensor 100 at a predetermined frame rate (for example, 30 fps). In FIG. 7, a person is included on the right side of the screen of the live view image, and a tree is included on the left side of the screen of the live view image. The control unit 70 sends an instruction to the image processing unit 30 to perform a known scene recognition process on the live view image data. The image processing unit 30 analyzes the live view image and extracts the main subject area by performing the scene recognition process.

The image processing unit 30 sets the range including the detected human body as the main subject area as described above. In addition, not only a person but also an animal such as a pet may be detected, and a range including this animal may be set as a main subject area. Then, the control unit 70 sets the main subject area as the attention area 80 and the area other than the attention area 80 as the peripheral area 90.

In addition, in the state where the live view image is displayed on the liquid crystal display panel 51 of the display unit 50, the area of the main subject corresponding to the position where the user who visually recognizes the live view image touches the touch panel 52 is (displayed). It may be the area of interest 80.

For example, when a live view image is acquired, the control unit 70 determines the automatic exposure calculation and the white balance adjustment value based on the pixel signals output from the attention area 80 and the peripheral area 90 after the first accumulation time is accumulated. It is performed by the AWB calculation unit 72. The AE and AWB calculation units 72 calculate the exposure (exposure time, gain, etc.) so that, for example, the average level of the pixel signal approaches a predetermined level. Further, the AE and AWB calculation units 72 determine a white balance adjustment value for expressing a white color in white.

Furthermore, the control unit 70 generates a monitor image based on the pixel signals output from the attention area 80 and the peripheral area 90 after the accumulation of the first storage time, and displays the live view image on the display unit 50. Let me.

The control unit 70 further determines the focus adjustment state by the imaging optical system 10 based on the pixel signal output from the area of interest 80 after the accumulation of the second storage time at the time of acquiring a live view image, for example, by an AF (autofocus) calculation unit. It is calculated by 71. In the present embodiment, the second storage time is controlled to be longer than the first storage time.

The AF calculation unit 71 detects the focus adjustment state by, for example, a contrast detection method. Specifically, while moving the position of the focus lens of the imaging optical system 10, the position of the focus lens of the imaging optical system 10 is adjusted so as to increase the contrast of the image composed of the pixel signals output from the region of interest 80. To do.

The focus detection process may be performed by the phase difference detection method. In this case, the image sensor 100 (imaging chip 113) is provided with pixels for focus detection in advance. Then, the focus adjustment state (specifically, the defocus amount) by the imaging optical system 10 is detected by performing the phase difference detection calculation using the output signal from the focus detection pixel included in the focus region 80. Since the pixel and phase difference detection calculation for focus detection are known as described in, for example, Japanese Patent Application Laid-Open No. 2009-94881, detailed description thereof will be omitted.

The control unit 70 sends an instruction to the drive unit 21 when acquiring the live view image, and from the attention region 80 of the image sensor 100 (imaging chip 113), as described above, the pixel signals having different storage times (that is, the first storage). The pixel signal after the lapse of time and the pixel signal after the lapse of the second storage time) are read out in a plurality of times. Here, the accumulation control is performed separately for the region of interest 80 and the peripheral region 90 before the instruction for main imaging (still image recording or moving image recording) is performed by operating the release switch (not shown).

That is, until this imaging instruction is given, exposure calculation, white balance adjustment value determination, and live view image display are performed based on the images acquired in the attention region 80 and the peripheral region 90, and the attention region 80 is used. Focus detection processing is performed based on the acquired image.

<Reading pixel signals with different storage times>
The reading of pixel signals having different storage times will be described with reference to FIG. 8 for explaining the read timing of the pixel signal, the stored signal in the image pickup chip 113, and the pixel signal read from the image sensor 100 via the arithmetic circuit 415. To do.

The drive unit 21 controls the image sensor 100 as follows. That is, in each frame of the live view image, the accumulation start time t0 to the time t1 is defined as the first accumulation time, and the time t0 to the time t2 is defined as the second accumulation time. At time t0, the drive unit 21 starts charge accumulation for the pixels included in the attention region 80 and the peripheral region 90. Then, at time t1, the pixel signal is output from the attention area 80 and the peripheral area 90 while controlling the pixel memory 414 so as not to read the pixel signal stored in the pixel memory n illustrated in FIG. As a result, the pixel signal a accumulated during the first storage time (time t0 to time t1) is output from the demultiplexer 413 and is output as it is as the signal A via the arithmetic circuit 415. This pixel signal A (= a) is also stored in the pixel memory n.

Further, when the drive unit 21 reads out the pixel signal at time t1, the drive unit 21 immediately starts charge accumulation for the pixels included in the attention region 80. Then, at time t2, the pixel signal is output from the area of interest 80 while controlling the pixel memory 414 so as to read the pixel signal a stored in the pixel memory n illustrated in FIG. As a result, the pixel signal b accumulated between the time t1 and the time t2 is output from the demultiplexer 413, and the pixel signal b and the pixel signal a read from the pixel memory n are added by the adder n. Will be done. The added pixel signals a + b are output as signal B via the arithmetic circuit 415. Since the pixel signal B (= a + b) is the sum of the pixel signals accumulated from time t0 to time t1 and from time t1 to time t2, it is accumulated during the second accumulation time (time t0 to time t2). Corresponds to the pixel signal to be generated.

As described above, in the first embodiment, since the pixel signal B having a high signal level accumulated in the second storage time longer than the first storage time is used for the focus detection process, the pixel signal A having a low signal level is focused. Higher focus detection accuracy can be obtained as compared with the case of using it for the detection process. Further, since the pixel signal A accumulated in the first accumulation time shorter than the second accumulation time is used for the exposure calculation, the determination of the white balance adjustment value, and the display of the live view image, the calculation can be performed accurately while avoiding the influence of noise. , You can get a live view image that is easy to see without being too bright.

<Explanation of flowchart>
FIG. 9 is a flowchart illustrating a flow of a shooting operation executed by the control unit 70 of the imaging device 1 in the first embodiment. The control unit 70 repeatedly activates the process according to FIG. 9 when the power-on operation of the ON-OFF switch (not shown) is performed and each unit of the image pickup apparatus 1 is energized. In step S101 of FIG. 9, the control unit 70 determines control parameters such as the frame rate and gain for the region of interest 80 and the peripheral region 90, and proceeds to step S102. For example, the values to be applied in steps S102 and S110, which will be described later, are read from the program data and prepared.

In step S102, the control unit 70 sends an instruction to the drive unit 21 to start live view imaging by the image pickup unit 20. The acquisition of the live view image started in step S102 is performed by setting the control parameters for the peripheral region 90, for example, for substantially the entire area of the imaging surface of the image sensor 100.

In step S103, the control unit 70 sends an instruction to the drive unit 21 to transfer the first data from the image pickup unit 20. The first data corresponds to the pixel signal A (= a) read out at the time t1. In step S104, the control unit 70 causes the image processing unit 30 to process the live view image data based on the pixel signal A output from the imaging unit 20, and then displays the live view image data on the display unit 50.

In step S105, the control unit 70 sends an instruction to the drive unit 21 to transfer the second data from the image pickup unit 20. The second data corresponds to the pixel signal B (= a + b) read out at the time t2. In step S106, the control unit 70 causes the AF calculation unit 71 to perform an AF calculation (detection of the focus adjustment state) based on the pixel signal B for the AF calculation area. As a result, the focus of the imaging optical system 10 can be adjusted. Since the AF calculation area is undecided in the first frame after the power-on operation, the AF calculation is performed on substantially the entire image pickup surface of the image pickup device 100.

In step S107, the control unit 70 sends an instruction to the image processing unit 30 to extract the main subject area from the live view image. In step S108, the control unit 70 sets the area including the main subject as the area of interest 80, determines the area of interest 80 as the AF calculation area, and proceeds to step S109. The AF calculation area is applied to the live view image acquired in the next frame.

In step S109, the control unit 70 determines whether or not the release operation has been performed. The release operation is used as the main imaging instruction for the imaging device 1. When the release button (not shown) is pressed, the control unit 70 positively determines step S109 and proceeds to step S110. If the pressing operation is not performed, the control unit 70 negatively determines step S109 and proceeds to step S112. move on.

When the tap operation on the release icon displayed on the liquid crystal display panel 51 is detected to determine the main imaging instruction, step S109 may be positively determined when the released release icon being displayed is tapped.

In step S110, the control unit 70 sends an instruction to the drive unit 21 and controls parameters (exposure time, gain, etc.) required for the exposure conditions for shooting determined based on the first data (pixel signal A (= a)). Is set and the process proceeds to step S111.

In step S111, the control unit 70 executes a shooting process, stores the acquired image data in a memory card or the like by the recording unit 60, and returns to step S102. In the shooting process, a control parameter for shooting that is set to be common (same) in all areas of the image sensor 100 is applied to acquire and record a still image of one frame (main imaging), and after acquiring the still image. Also gets a multi-frame image. Then, slow playback moving image data is generated and recorded based on the images of a plurality of frames acquired during a predetermined time before and after the release operation. The slow reproduction moving image data refers to moving image data reproduced at a frame rate (for example, 15 fps) slower than the frame rate (for example, 30 fps) acquired by the image sensor 100.

The control unit 70 generates slow playback moving image data as follows. That is, from before the shooting time (for example, 0.6 seconds before) before the release operation time (referred to as tx) to the time tx, the first is temporarily stored in the work memory 40 for displaying the above-mentioned live view image. Multiple frame images based on one data (pixel signal A (= a)) (for example, when the frame rate is acquired at 30 fps, 0.6 seconds is 18 frames), and the time tx is taken after the time tx. A plurality of frame images (for example, 0 when the frame rate is acquired at 30 fps) based on the first data (pixel signal A (= a)) stored in the work memory 40 by a time (for example, 0.4 seconds later). .Slow playback moving image data is generated based on (4 seconds is 12 frames). As a result, playback is performed based on a plurality of frame images (30 images in total) stored in the work memory 40 for 1 second (0.6 seconds before the time tx to 0.4 seconds after the time tx) sandwiching the time tx. Generates slow playback video data with a time of about 2 seconds. In this way, slow playback moving image data based on the frame images acquired before and after the release operation can be obtained. The slow playback moving image data is generated as MPEG data or JPEG data by the image processing unit 30.

In step S112 in which the above-mentioned step S109 is negatively determined and the process proceeds, the control unit 70 determines whether or not the power-off operation has been performed. The control unit 70 positively determines step S112 when the power-off operation of the ON-OFF switch (not shown) is performed, performs a predetermined power-off process, and ends the process according to FIG. If the ON-OFF switch (not shown) is not operated to turn off the power, the control unit 70 makes a negative determination in step S112 and returns to step S102. When returning to step S102, the above-described processing is repeated.

According to the first embodiment described above, the following effects can be obtained.
(1) The image pickup element 100 includes a demultiplexer 413 that inputs data from a plurality of pixels that perform photoelectric conversion, a pixel memory 414 that stores the input data for each pixel, data stored in the pixel memory 414, and the above. An adder 416 that adds input data for each pixel and an arithmetic circuit 415 that outputs the input data or the data added by the adder 416 are provided. As a result, the photoelectric conversion data can be appropriately processed, so that the usability of the image sensor 100 can be improved.

(2) Since the pixel memory 414 stores the data added by the adder 416 for each pixel, the input data can be integrated. As a result, the photoelectric conversion data can be integrated and output, or can be output without integration.

(3) Since the adder 416 adds data having different photoelectric conversion times for each pixel, data corresponding to a case where the photoelectric conversion time is changed for a long time can be obtained.

(4) Since the drive control unit 417 for reading the data stored in the pixel memory 414 and adding the data to the adder 416 at the timing when the data from the pixels is input is provided, the drive control unit 417 is provided at an appropriate timing. Can be controlled.

(5) When the data output from the arithmetic circuit 415 is the above input data, the exposure calculation, the determination of the white balance adjustment value, and the display of the live view image are performed, and the data output from the arithmetic circuit 415 is the addition data. Since the focus detection process is performed in a certain case, the photoelectric conversion data and the process using the photoelectric conversion data can be appropriately combined.

(6) Since the image composed of the input data is analyzed to determine whether to output the input data or the addition data from the arithmetic circuit 415, an appropriate photoelectric conversion is determined according to the image. Data can be output.

(7) In the above determination, the addition data is output for the pixels corresponding to the main subject area of the image, and the input data is output for the pixels corresponding to the area other than the main subject area of the image. The converted data can be output.

(8) Since the image pickup chip 113 having a plurality of pixels and the memory chip 112 for processing photoelectric conversion data are laminated, the image pickup element 100 can be compactly configured without enlarging each chip in the plane direction.

(9) Since the image pickup device 1 which is an example of an electronic device includes a memory chip 112 for processing photoelectric conversion data and an image pickup chip 113 having a plurality of pixels, it is easy to use by appropriately processing the photoelectric conversion data. A good imaging device 1 can be provided.

(Second embodiment)
In the first embodiment, an example has been described in which the screen is divided into a region of interest 80 and a peripheral region 90, and images having different accumulation times in the region of interest 80 and the peripheral region 90 are obtained before the instruction of the main imaging. In the second embodiment, the screen is divided into a bright region 85 and a dark region 95, and after the instruction of the main imaging, images having different accumulation times are obtained in the bright region 85 and the dark region 95. Here, the bright region 85 is, for example, a region composed of pixel signals having a value higher than a predetermined pixel signal value (higher than a predetermined luminance value) by analyzing a live view image. The dark region 95 is a region other than the bright region 85.

The image pickup apparatus 1 according to the second embodiment acquires an image based on the pixel signal accumulated in the first storage time in the bright region 85 and an image based on the pixel signal stored in the dark region 95 in the second storage time. , Both images are combined to obtain an image with a wide dynamic range. Therefore, the control unit 70 sends an instruction to the drive unit 21 when acquiring the image for recording, and the pixel signal after the lapse of the first accumulation time and the second from the dark area 95 of the image sensor 100 (image sensor 113). The pixel signal after the accumulation time has elapsed is read out in a plurality of times.

<Reading pixel signals with different storage times>
The read timing of the pixel signal, the stored signal in the image pickup chip 113, and the pixel signal read out from the image pickup device 100 via the arithmetic circuit 415 will be described with reference to FIG.

The drive unit 21 controls the image sensor 100 as follows. That is, the accumulation start time t0 to the time t1 of the recording image after the main imaging instruction is defined as the first accumulation time, and the time t0 to the time t2 is defined as the second accumulation time. At time t0, the drive unit 21 starts charge accumulation for the pixels included in the bright region 85 and the dark region 95. Then, at time t1, the pixel signal is output from the bright region 85 and the dark region 95 while controlling the pixel memory 414 so as not to read the pixel signal stored in the pixel memory n illustrated in FIG. As a result, the pixel signal a accumulated during the first storage time (time t0 to time t1) is output from the demultiplexer 413 and is output as it is as the signal A via the arithmetic circuit 415. This pixel signal A (= a) is also stored in the pixel memory n.

Further, when the drive unit 21 reads out the pixel signal at time t1, the drive unit 21 immediately starts charge accumulation for the pixels included in the dark area 95. Then, at time t2, the pixel signal is output from the dark area 95 while controlling the pixel memory 414 so as to read the pixel signal a stored in the pixel memory n illustrated in FIG. As a result, the pixel signal b accumulated between the time t1 and the time t2 is output from the demultiplexer 413, and the pixel signal b and the pixel signal a read from the pixel memory n are added by the adder n. Will be done. The added pixel signals a + b are output as signal B via the arithmetic circuit 415. Since the pixel signal B (= a + b) is the sum of the pixel signals accumulated from time t0 to time t1 and from time t1 to time t2, it is accumulated during the second accumulation time (time t0 to time t2). Corresponds to the pixel signal to be generated.

As described above, in the second embodiment, since the image of the dark area 95 is formed by the pixel signal B accumulated in the second accumulation time longer than the first accumulation time, the accumulation time is short and the signal level is low. Compared with the case where the image of the dark area 95 is formed in A, blackout of the image is less likely to occur. Further, since the image of the bright region 85 is formed by the pixel signal A accumulated in the first accumulation time shorter than the second accumulation time, the image of the bright region 85 is formed by the pixel signal B having a long accumulation time and a high signal level. Overexposure of the image is less likely to occur as compared with the case of forming. As a result, an image having a wide dynamic range from the bright part to the dark part can be obtained by performing the accumulation process from the time t0 to the time t2 only once. Further, since the accumulation start time is the same for the first accumulation time and the second accumulation time (both time t0), a moving subject is different from the conventional technique of synthesizing a plurality of images having different disclosure accumulation start times. It is also suitable for taking pictures of.

<Explanation of flowchart>
FIG. 10 is a flowchart illustrating a flow of a shooting operation executed by the control unit 70 of the imaging device 1 in the second embodiment. The control unit 70 repeatedly activates the process according to FIG. 10 when an ON-OFF switch (not shown) is turned on and each part of the image pickup apparatus 1 is energized. In step S201 of FIG. 10, the control unit 70 determines control parameters such as the frame rate and gain for the bright region 85 and the dark region 95, and proceeds to step S202. For example, the values to be applied in steps S202, S205, and S208, which will be described later, are read from the program data and prepared.

In step S202, the control unit 70 sends an instruction to the drive unit 21 to start live view imaging by the image pickup unit 20. The acquisition of the live view image started in step S202 is performed by setting the control parameters for the dark area 95 for substantially the entire area of the image pickup surface of the image pickup device 100, for example.

In step S203, the control unit 70 determines the bright region 85 and the dark region 95 based on the live view image, and proceeds to step S204. In step S204, the control unit 70 determines whether or not the release operation has been performed. The release operation is used as the main imaging instruction for the imaging device 1. When the release button (not shown) is pressed, the control unit 70 positively determines step S204 and proceeds to step S205. If the pressing operation is not performed, the control unit 70 negatively determines step S204 and proceeds to step S211. move on.

When the tap operation on the release icon displayed on the liquid crystal display panel 51 is detected to determine the main imaging instruction, step S204 may be positively determined when the released release icon being displayed is tapped.

In step S205, the control unit 70 sends an instruction to the drive unit 21 to start recording imaging by the image pickup unit 20. The acquisition of the recording image started in step S205 is performed, for example, by setting the control parameters for the bright region 85 for substantially the entire area of the imaging surface of the image sensor 100.

In step S206, the control unit 70 sends an instruction to the drive unit 21 to transfer the first data from the image pickup unit 20. The first data corresponds to the pixel signal A (= a) read out at the time t1. In step S207, the control unit 70 sends an instruction to the image processing unit 30, temporarily stores the data corresponding to the bright region 85 in the first data in the work memory 40, and proceeds to step S208.

In step S208, the control unit 70 sends an instruction to the drive unit 21 to transfer the second data from the image pickup unit 20. The second data corresponds to the pixel signal B (= a + b) read out at the time t2. In step S209, the control unit 70 sends an instruction to the image processing unit 30, temporarily stores the second data (that is, the data corresponding to the dark area 95) in the work memory 40, and proceeds to step S210.

In step S210, the control unit 70 sends an instruction to the image processing unit 30, and image-synthesizes the first data and the second data temporarily stored in the work memory 40. Specifically, in the image based on the first data, the region corresponding to the dark area 95 is replaced with the image based on the second data. As a result, a composite image is obtained in which the dark region 95 is composed of the pixel signal B (= a + b) and the bright region is composed of the pixel signal A (= a). The control unit 70 returns to step S202 when the image composition is completed. When returning to step S202, the above-described processing is repeated.

In step S211 in which the above-mentioned step S204 is negatively determined and the process proceeds, the control unit 70 determines whether or not the power-off operation has been performed. The control unit 70 positively determines step S211 when the power-off operation of the ON-OFF switch (not shown) is performed, performs a predetermined power-off process, and ends the process according to FIG. If the ON-OFF switch (not shown) is not operated to turn off the power, the control unit 70 makes a negative determination in step S211 and returns to step S202.

According to the second embodiment described above, the following effects can be obtained.
(1) The image pickup element 100 forms a bright region 85 of the image when the data output from the arithmetic circuit 415 is the data input to the demultiplexer 413, and the data output from the arithmetic circuit 415 is an adder. Since the dark region 95 of the image is formed when the data is added by 416, the photoelectric conversion data and the processing using the photoelectric conversion data can be appropriately combined to obtain an image having a wide dynamic range.

(2) Since the image composed of the input data is analyzed to determine whether to output the input data or the addition data from the arithmetic circuit 415, an appropriate photoelectric conversion is determined according to the image. Data can be output.

(3) In the above determination, the input data is output for the pixels corresponding to the region where the brightness of the image is higher than the predetermined value, and the added data is output for the pixels corresponding to the region where the brightness of the image is lower than the predetermined value. Therefore, it is possible to output appropriate photoelectric conversion data.

(Modification example 1)
The image pickup apparatus 1 according to the first embodiment and the second embodiment described above may be configured by a high-performance mobile phone or a tablet terminal. In this case, the camera unit mounted on the high-performance mobile phone (or tablet terminal) is configured by using the stacked image sensor 100.

(Modification 2)
In the second embodiment described above, an example in which the image processing unit 30 synthesizes an image using the first data and the second data temporarily stored in the work memory 40 has been described. Instead, the pixel memory 414 and the adder 416 illustrated in FIGS. 4 and 5 may be used to perform image composition. For example, when the pixel signal is read out at time t1, only the pixel signal A corresponding to the bright region 85 is selectively read out from the image sensor 100. Further, when reading out the pixel signal at time t2, control is performed so that only the pixel signal B corresponding to the dark region 95 is selectively read out from the image sensor 100.

(Modification example 3)
In the first embodiment and the second embodiment described above, the pixel signal after the lapse of the first storage time and the pixel signal after the lapse of the second storage time have passed from a predetermined region of the image sensor 100 (imaging chip 113). An example of reading out in two steps has been described. The number of times of reading a plurality of times may be not only the above-mentioned two times but also three times, four times, or more.

In addition, for the images read out in multiple times before the instruction of the main imaging, the image used for AF calculation, the exposure calculation, the image used for determining the white balance adjustment value, the image used for extracting the main subject, and the live view display are displayed. The use may be divided as an image to be used for. Further, the images read out in a plurality of times after the instruction of the main imaging may be classified into an extremely bright region, a bright region, a dark region, an extremely dark region, and the like, respectively, and their uses may be divided according to the brightness of the image. ..

The above description is merely an example, and is not limited to the configuration of the above embodiment. The configurations of the above embodiments and the modifications may be combined as appropriate.

1 ... Image sensor 10 ... Imaging optical system 20 ... Imaging unit 30 ... Image processing unit 40 ... Work memory 50 ... Display unit 51 ... Liquid crystal display panel 52 ... Touch panel 60 ... Recording unit 70 ... Control unit 71 ... AF calculation unit 72 ... AE , AWB arithmetic unit 100 ... image sensor 109 ... bump 111 ... signal processing chip 112 ... memory chip 113 ... imaging chip 131 ... unit area 413 ... demultiplexer 414 ... pixel memory 415 ... arithmetic circuit 416 ... adder 417 ... drive control unit

Claims (13)

Multiple pixels arranged side by side in the row and column directions,
A first conversion unit that converts a first signal output from the first pixel to the first output wiring among the plurality of pixels into a digital signal, and a first conversion unit.
Among the plurality of pixels, a second conversion unit that converts a second signal output from the second pixel arranged at a position on the column direction side from the first pixel to the second output wiring into a digital signal, and a second conversion unit.
A first adder that performs addition processing on the first signal converted into a digital signal by the first conversion unit, and
A second adder that performs addition processing on the second signal converted into a digital signal by the second conversion unit, and
An image sensor comprising. In the image pickup device according to claim 1,
A first storage unit that stores the first signal that has been subjected to addition processing by the first adder, and a first storage unit.
A second storage unit that stores the second signal that has been subjected to addition processing by the second adder, and a second storage unit.
An image sensor comprising. In the image pickup device according to claim 1 or 2.
The first adder performs an addition process on the first signal using a signal generated before the first signal is output to the first output wiring in the first pixel.
The second adder is an image pickup device that performs addition processing on the second signal using a signal generated before the second signal is output to the second output wiring in the second pixel. In the image pickup device according to any one of claims 1 to 3.
The first adder performs addition processing on a third signal output from a third pixel arranged at a position on the row direction side from the first pixel among the plurality of pixels.
The second adder is an image pickup device that performs addition processing on a fourth signal output from a fourth pixel arranged at a position on the row direction side from the second pixel among the plurality of pixels. In the image pickup device according to claim 4,
The third signal is output from the first output wiring.
The second output wiring is an image sensor that outputs the fourth signal. In the image pickup device according to claim 4 or 5.
A third storage unit that stores the third signal that has been subjected to addition processing by the first adder, and a third storage unit.
A fourth storage unit that stores the fourth signal that has been subjected to addition processing by the second adder, and a fourth storage unit.
An image sensor comprising. The image sensor according to any one of claims 4 to 6.
The first adder performs addition processing on the third signal using the signal generated before outputting the third signal to the first output wiring in the third pixel.
The second adder is an image sensor that performs addition processing on the fourth signal using a signal generated before the fourth signal is output to the second output wiring in the fourth pixel. In the image pickup device according to any one of claims 1 to 7.
The plurality of pixels are arranged on an imaging chip on which light is incident, and the plurality of pixels are arranged on an imaging chip.
The first conversion unit and the second conversion unit are image pickup devices arranged on a signal processing chip connected to the image pickup chip. In the image pickup device according to claim 8,
The first adder and the second adder are image pickup devices arranged on a memory chip connected to the signal processing chip. An image pickup apparatus comprising the image pickup device according to any one of claims 1 to 9. In the imaging device according to claim 10,
An imaging device including a control unit that displays an image based on the first signal stored in the first storage unit and the second signal stored in the second storage unit on the display unit. In the imaging device according to claim 11,
Light from an optical system including a focus lens is incident on the image sensor.
The control unit is an imaging device that drives the focus lens using at least one signal of the first signal stored in the first storage unit and the second signal stored in the second storage unit. .. In the imaging apparatus according to claim 11 or 12.
The control unit is an imaging device that causes a recording unit to record image data generated based on the first signal stored in the first storage unit and the second signal stored in the second storage unit.

Images