JP2014179778A - Signal processing apparatus, imaging device, imaging apparatus, and electronic apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly process photoelectric conversion data.SOLUTION: A signal processing apparatus 112 comprises: a data input unit 413 which receives data inputted from plural pixels for photoelectric conversion; a memory 414 which stores the data inputted to the data input unit 413 on pixel-to-pixel basis; an addition unit 416 which adds the data stored in the memory 414 and the data inputted to the data input unit 413 on pixel-to-pixel basis; and a data output unit 415 which outputs the data inputted to the data input unit 413 or the data added by the addition unit 416.

Description

  The present invention relates to a signal processing device, an imaging device, an imaging device, and an electronic apparatus.

  There has been proposed an electronic apparatus including an imaging element (hereinafter, referred to as a multilayer imaging element) in which a back-illuminated imaging chip and a signal processing chip are stacked (see Patent Document 1). The multilayer imaging element is laminated so that the back-illuminated imaging chip and the signal processing chip are connected to each other through micro bumps.

JP 2006-49361 A

  There are not many proposals to divide an image into blocks having one or more of the above-described areas and acquire a captured image for each block in an electronic apparatus including a conventional stacked image sensor, and the multilayer image sensor is provided. The usability of electronic devices was not sufficient. Furthermore, the conventional technology that combines a plurality of images captured separately under different exposure conditions is not suitable for shooting a moving subject.

  A signal processing device according to the present invention includes a data input unit that inputs data from a plurality of pixels that perform photoelectric conversion, a memory that stores data input to the data input unit for each pixel, data stored in the memory, and An addition unit that adds the data input to the data input unit for each pixel, and a data output unit that outputs the data input to the data input unit or the data added by the addition unit are provided.

  According to the present invention, photoelectric conversion data can be appropriately processed.

It is sectional drawing of a multilayer type image pick-up element. It is a figure explaining the pixel arrangement | sequence and unit area | region of an imaging chip. It is a circuit diagram corresponding to the unit area | region of an imaging chip. It is a block diagram which shows the functional structure of an image pick-up element. It is a figure explaining the flow of the pixel signal per pixel. It is a block diagram which illustrates the composition of the imaging device which has an image sensor. It is a figure which illustrates the attention field and peripheral field in an image sensor. It is a figure explaining the pixel signal read from an image sensor via a read timing, an accumulation signal, and an arithmetic circuit. It is a flowchart explaining the flow of the imaging | photography operation | movement which the control part of 1st embodiment performs. It is a flowchart explaining the flow of the imaging | photography operation | movement which the control part of 2nd embodiment performs.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
(First embodiment)
<Description of Laminated Image Sensor>
First, the multilayer imaging element 100 mounted on the electronic apparatus (for example, the imaging apparatus 1) according to the first embodiment of the present invention will be described. The multilayer image sensor 100 is described in Japanese Patent Application No. 2012-139026 filed earlier by the applicant of the present application. FIG. 1 is a cross-sectional view of the multilayer image sensor 100. The imaging device 100 includes a backside illumination type imaging chip 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are stacked, and are electrically connected to each other by a conductive bump 109 such as Cu.

  As shown in the figure, incident light is incident mainly in the positive direction of the Z-axis indicated by a white arrow. In the present embodiment, in the imaging chip 113, the surface on the side where incident light is incident is referred to as a back surface. Further, as shown in the coordinate axes, the left direction of the paper orthogonal to the Z axis is the X axis plus direction, and the front side of the paper orthogonal to the Z axis and the X axis is the Y axis plus direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

  An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is disposed on the back side of the wiring layer 108. The PD layer 106 includes a plurality of PDs (photodiodes) 104 that are two-dimensionally arranged and store charges corresponding to incident light, and transistors 105 that are provided corresponding to the PDs 104.

  A color filter 102 is provided on the incident side of incident light in the PD layer 106 via a passivation film 103. The color filter 102 has a plurality of types that transmit different wavelength regions, and has a specific arrangement corresponding to each of the PDs 104. The arrangement of the color filter 102 will be described later. A set of the color filter 102, the PD 104, and the transistor 105 forms one pixel.

  On the incident light incident side of the color filter 102, a microlens 101 is provided corresponding to each pixel. The microlens 101 condenses incident light toward the corresponding PD 104.

  The wiring layer 108 includes a wiring 107 that transmits the pixel signal from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multilayer, and a passive element and an active element may be provided.

  A plurality of bumps 109 are disposed on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the opposing surfaces of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed and aligned. The bumps 109 are joined and electrically connected.

  Similarly, a plurality of bumps 109 are disposed on the mutually facing surfaces of the signal processing chip 111 and the memory chip 112. The bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressurized, so that the aligned bumps 109 are joined and electrically connected.

  The bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, and micro bump bonding by solder melting may be employed. Further, for example, about one bump 109 may be provided for one unit region described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Further, a bump larger than the bump 109 corresponding to the pixel region may be provided in a peripheral region other than the pixel region where the pixels are arranged.

  The signal processing chip 111 has a TSV (silicon through electrode) 110 that connects circuits provided on the front and back surfaces to each other. The TSV 110 is preferably provided in the peripheral area. The TSV 110 may also be provided in the peripheral area of the imaging chip 113 and the memory chip 112.

  FIG. 2 is a diagram for explaining the pixel array and the unit region 131 of the imaging chip 113. In particular, a state where the imaging chip 113 is observed from the back side is shown. For example, 20 million or more pixels are arranged in a matrix in the pixel region. In the present embodiment, for example, 16 pixels of adjacent 4 pixels × 4 pixels form one unit region 131. The grid lines in the figure indicate the concept that adjacent pixels are grouped to form a unit region 131. The number of pixels forming the unit region 131 is not limited to this, and may be about 1000, for example, 32 pixels × 64 pixels, or more or less.

  As shown in the partially enlarged view of the pixel region, the unit region 131 includes four so-called Bayer arrays, which are composed of four pixels of green pixels Gb, Gr, blue pixels B, and red pixels R, vertically and horizontally. The green pixel is a pixel having a green filter as the color filter 102, and receives light in the green wavelength band of incident light. Similarly, a blue pixel is a pixel having a blue filter as the color filter 102 and receives light in the blue wavelength band, and a red pixel is a pixel having a red filter as the color filter 102 and receiving light in the red wavelength band. Receive light.

  In the present embodiment, a plurality of blocks are defined so as to include at least one unit region 131 per block, and each block can control pixels included in each block with different control parameters. That is, it is possible to acquire imaging signals having different imaging conditions between a pixel group included in a certain block and a pixel group included in another block. Examples of the control parameters are a frame rate, a gain, a thinning rate, the number of addition rows or addition columns to which pixel signals are added, the charge accumulation time or accumulation count, the number of digitization bits, and the like. Furthermore, the control parameter may be a parameter in image processing after obtaining an image signal from a pixel.

  FIG. 3 is a circuit diagram corresponding to the unit region 131 of the imaging chip 113. In FIG. 3, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least some of the transistors described below correspond to the transistor 105 in FIG.

  As described above, the unit region 131 is formed of 16 pixels. The 16 PDs 104 corresponding to the respective pixels are respectively connected to the transfer transistors 302, and the gates of the transfer transistors 302 are connected to the TX wiring 307 to which transfer pulses are supplied. In the present embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

  The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to a Vdd wiring 310 to which a power supply voltage is supplied, and the gate thereof is connected to a reset wiring 306 to which a reset pulse is supplied. In the present embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

  The drain of each amplification transistor 304 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. The source of each amplification transistor 304 is connected to the drain of each corresponding selection transistor 305. Each gate of the selection transistor 305 is connected to a decoder wiring 308 to which a selection pulse is supplied. In the present embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. The source of each selection transistor 305 is connected to a common output wiring 309. The load current source 311 supplies current to the output wiring 309. That is, the output wiring 309 for the selection transistor 305 is formed by a source follower. Note that the load current source 311 may be provided on the imaging chip 113 side or may be provided on the signal processing chip 111 side.

  Here, the flow from the start of charge accumulation to pixel output after the end of accumulation will be described. When a reset pulse is applied to the reset transistor 303 through the reset wiring 306 and simultaneously a transfer pulse is applied to the transfer transistor 302 through the TX wiring 307, the potentials of the PD 104 and the floating diffusion FD are reset.

  When the application of the transfer pulse is canceled, the PD 104 converts the incident light to be received into electric charge and accumulates it. Thereafter, when the transfer pulse is applied again without the reset pulse being applied, the accumulated charge is transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after the charge accumulation. . When a selection pulse is applied to the selection transistor 305 through the decoder wiring 308, a change in the signal potential of the floating diffusion FD is transmitted to the output wiring 309 through the amplification transistor 304 and the selection transistor 305. Thereby, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

  As shown in FIG. 3, in this embodiment, the reset wiring 306 and the TX wiring 307 are common to the 16 pixels forming the unit region 131. That is, the reset pulse and the transfer pulse are simultaneously applied to all 16 pixels. Accordingly, all the pixels forming the unit region 131 start charge accumulation at the same timing and end charge accumulation at the same timing. However, the pixel signal corresponding to the accumulated charge is selectively output from the output wiring 309 by sequentially applying the selection pulse to each selection transistor 305. Further, the reset wiring 306, the TX wiring 307, and the output wiring 309 are provided separately for each unit region 131.

  Thus, by configuring the circuit with the unit region 131 as a reference, the charge accumulation time can be controlled for each unit region 131. In other words, pixel signals having different frame rates can be output between the unit regions 131. Further, while one unit region 131 is caused to perform charge accumulation once, the other unit region 131 is caused to repeatedly accumulate charges several times and each time a pixel signal is output, so that Each frame for moving images can be output at a different frame rate between the unit areas 131.

  FIG. 4 is a block diagram illustrating a functional configuration of the image sensor 100. The analog multiplexer 411 sequentially selects the 16 PDs 104 forming the unit area 131 and outputs each pixel signal to the output wiring 309 provided corresponding to the unit area 131. The multiplexer 411 is formed on the imaging chip 113 together with the PD 104.

  The pixel signal output via the multiplexer 411 is supplied to the signal processing chip 111 by a signal processing circuit 412 that performs correlated double sampling (CDS) / analog / digital (A / D) conversion. D conversion is performed. The A / D converted pixel signal is delivered to the demultiplexer 413. The pixel signal output from the demultiplexer 413 is input to the adder 416 corresponding to each pixel. The adder 416 adds the pixel signal output from the demultiplexer 413 and the pixel signal read from the pixel memory 414 to correspond to each pixel, and outputs the pixel signal after the addition to the pixel memory 414 again. To do.

  The pixel memory 414 stores the pixel signal from the adder 416. Each of the pixel memories 414 has a capacity capable of storing the pixel signal after addition. The demultiplexer 413, the adder 416, and the pixel memory 414 are formed in the memory chip 112.

  FIG. 5 is a diagram illustrating the flow of pixel signals per pixel. In FIG. 5, the pixel signal S output from the demultiplexer 413 is input to the corresponding adder n among the adders 416. At this time, the pixel signal P stored in the corresponding memory n of the pixel memory 414 is read from the memory n and input to the adder n.

  The adder n adds the input pixel signal S and the pixel signal P, and outputs the pixel signal S + P after the addition to the pixel memory n. The pixel memory n stores the input pixel signal S + P and waits for reading to the arithmetic circuit 415. Here, when the addition is performed by the adder n, only the pixel signal S input to the adder n is controlled by controlling the pixel memory 414 so as not to read the pixel signal P stored in the pixel memory n. Can be directly output from the adder n to the pixel memory n. That is, the pixel signal S from the imaging chip 113 can be read from the memory n to the arithmetic circuit 415 without being added by the adder n.

  The arithmetic circuit 415 processes the pixel signal stored in the pixel memory 414 and passes it to the subsequent image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or may be provided in the memory chip 112.

  The drive control unit 417 is configured to send pixel signals from the imaging chip 113 to the signal processing chip 111 and the memory chip 112, read and store timings of the pixel signals in the pixel memory 414, and addition timings of the pixel signals in the adder 416. In order to synchronize the pixel signal delivery timing to the arithmetic circuit 415, a timing control signal is generated.

  Note that FIG. 4 shows connections for one unit region 131, but actually these exist for each unit region 131 and operate in parallel. However, the arithmetic circuit 415 does not have to exist for each unit region 131. For example, one arithmetic circuit 415 may perform sequential processing while sequentially referring to the values of the pixel memory 414 corresponding to each unit region 131. Good.

  As described above, the output wiring 309 is provided corresponding to each of the unit regions 131. Since the image pickup device 100 has the image pickup chip 113, the signal processing chip 111, and the memory chip 112 laminated, by using electrical connection between the chips using the bump 109 for the output wiring 309, each chip is arranged in the surface direction. Wiring can be routed without increasing the size.

<Description of imaging device>
FIG. 6 is a block diagram illustrating the configuration of the imaging apparatus 1 having the imaging element 100 described above. In FIG. 6, the imaging apparatus 1 includes an imaging optical system 10, an imaging unit 20, an image processing unit 30, a work memory 40, a display unit 50, a recording unit 60, and a control unit 70.

  The imaging optical system 10 includes a plurality of lenses, and guides a light beam from the object scene to the imaging unit 20. The imaging optical system 10 may be configured integrally with the imaging device 1 or may be configured to be replaceable with respect to the imaging device 1. Further, the imaging optical system 10 may include a focus lens or a zoom lens.

  The imaging unit 20 includes the above-described imaging device 100 and a driving unit 21 that drives the imaging device 100. The image sensor 100 is driven and controlled by a control signal output from the drive unit 21, so that independent storage control can be performed in units of blocks as described above. The controller 70 gives instructions to the drive unit 21 such as the position, shape, and range of the block.

  The image processing unit 30 performs image processing on the image data imaged by the imaging unit 20 in cooperation with the work memory 40. In the present embodiment, the image processing unit 30 performs detection processing of a main subject included in an image in addition to normal image processing (color signal processing, gamma correction, etc.). Detection of the main subject by the image processing unit 30 can be performed using a known face detection function. In addition to face detection, a human body included in an image may be detected as a main subject as described in, for example, Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940).

  The work memory 40 temporarily stores image data before and after JPEG compression and before and after MPEG compression. The display unit 50 includes, for example, a liquid crystal display panel 51, and displays images (still images and moving images) and various information captured by the imaging unit 20, and displays an operation input screen. The display unit 50 has a configuration in which a touch panel 52 is laminated on the display surface of the liquid crystal display panel 51. The touch panel 52 outputs a signal indicating a position where the user touches the liquid crystal display panel 51.

  The recording unit 60 stores various data such as image data in a storage medium such as a memory card. The control unit 70 has a CPU and controls the overall operation of the imaging apparatus 1. In the present embodiment, the control unit 70 divides the imaging surface of the imaging device 100 (imaging chip 113) into a plurality of blocks, and acquires images with different frame rates (accumulation times) and gains between the blocks. For this purpose, the control unit 70 instructs the drive unit 21 of the block position, shape, range, and control parameters for each block.

  In addition, the control unit 70 calculates the focus adjustment state by the imaging optical system 10 by the AF calculation unit 71. The control unit 70 further performs exposure calculation by the AE and AWB calculation unit 72 so that proper exposure can be obtained.

<Attention area and surrounding area>
In the present embodiment, the concept of a region of interest and a peripheral region is introduced in the screen to correspond to the plurality of blocks. FIG. 7 is a diagram illustrating a region of interest 80 and a peripheral region 90 in the image sensor 100 (imaging chip 113). The control unit 70 determines the positions of the attention area 80 and the peripheral area 90 in the image sensor 100 (imaging chip 113) through scene recognition based on the live view image.

  Here, the live view image is also called a preview image before the main imaging is performed, and refers to a monitor image acquired by the imaging device 100 at a predetermined frame rate (for example, 30 fps). In FIG. 7, a person is included on the right side of the screen of the live view image, and a tree is included on the left side of the screen of the live view image. The control unit 70 sends an instruction to the image processing unit 30 to perform a known scene recognition process on the live view image data. The image processing unit 30 performs a scene recognition process to analyze the live view image and extract a main subject area.

  The image processing unit 30 sets a range including the human body detected as described above as a main subject area. Note that not only a person but also an animal such as a pet may be detected, and a range including the animal may be set as a main subject area. Then, the control unit 70 sets the main subject area as the attention area 80 and sets the area other than the attention area 80 as the peripheral area 90.

  In the state where the live view image is displayed on the liquid crystal display panel 51 in the display unit 50, the region of the main subject corresponding (displayed) corresponding to the position where the user viewing the live view image touches the touch panel 52 is displayed. The attention area 80 may be used.

  For example, when the live view image is acquired, the control unit 70 determines whether the automatic exposure calculation and the white balance adjustment value are determined based on the pixel signal output from the attention area 80 and the peripheral area 90 after the accumulation of the first accumulation time. This is performed by the AWB calculation unit 72. The AE and AWB calculation unit 72 calculates exposure (exposure time, gain, etc.) so that, for example, the average level of the pixel signal approaches a predetermined level. In addition, the AE / AWB calculation unit 72 determines a white balance adjustment value for expressing a white color white.

  Furthermore, the control unit 70 generates a monitor image based on the pixel signal output after accumulation of the first accumulation time from the attention area 80 and the peripheral area 90, and displays it on the display unit 50 as the live view image. Let

  For example, when the live view image is acquired, the control unit 70 further changes the focus adjustment state by the imaging optical system 10 based on the pixel signal output from the attention area 80 after accumulation of the second accumulation time, and an AF (autofocus) calculation unit. 71 to calculate. In the present embodiment, the second accumulation time is controlled to be longer than the first accumulation time.

  The AF calculation unit 71 detects the focus adjustment state by, for example, a contrast detection method. Specifically, the position of the focus lens of the imaging optical system 10 is adjusted so as to increase the contrast of an image composed of pixel signals output from the attention area 80 while moving the position of the focus lens of the imaging optical system 10. To do.

  Note that the focus detection process may be performed by a phase difference detection method. In this case, focus detection pixels are provided in advance in the imaging device 100 (imaging chip 113). The focus adjustment state (specifically, the defocus amount) by the imaging optical system 10 is detected by performing a phase difference detection calculation using an output signal from a focus detection pixel included in the attention area 80. The focus detection pixel and the phase difference detection calculation are well known as described in, for example, Japanese Patent Application Laid-Open No. 2009-94881, and thus detailed description thereof is omitted.

  The control unit 70 sends an instruction to the drive unit 21 when acquiring the live view image, and the pixel signals (that is, the first accumulation) having different accumulation times as described above from the attention area 80 of the imaging element 100 (imaging chip 113). The pixel signal after the elapse of time and the pixel signal after the elapse of the second accumulation time are read out in a plurality of times. Here, the accumulation control is performed separately for the attention area 80 and the peripheral area 90 before an instruction for main imaging (still image recording or moving image recording) is performed by operating a release switch (not shown).

  That is, until the main imaging instruction is performed, exposure calculation, determination of the white balance adjustment value, and display of the live view image are performed based on the images acquired in the attention area 80 and the peripheral area 90, and the attention area 80 is displayed. A focus detection process is performed based on the acquired image.

<Reading out pixel signals with different accumulation times>
With reference to FIG. 8 illustrating pixel signal readout timing, an accumulation signal in the imaging chip 113, and a pixel signal read out from the image sensor 100 via the arithmetic circuit 415, readout of pixel signals having different accumulation times will be described. To do.

  The drive unit 21 controls the image sensor 100 as follows. That is, in each frame of the live view image, the accumulation start time t0 to time t1 is defined as the first accumulation time, and the period from time t0 to time t2 is defined as the second accumulation time. The drive unit 21 starts charge accumulation for the pixels included in the region of interest 80 and the peripheral region 90 at time t0. At time t1, pixel signals are output from the region of interest 80 and the peripheral region 90 while controlling the pixel memory 414 so as not to read out the pixel signals stored in the pixel memory n illustrated in FIG. As a result, the pixel signal a accumulated during the first accumulation time (from time t0 to time t1) is output from the demultiplexer 413, and is directly output as the signal A via the arithmetic circuit 415. This pixel signal A (= a) is also stored in the pixel memory n.

  Further, when the driving unit 21 reads out the pixel signal at time t1, the driving unit 21 immediately starts charge accumulation for the pixels included in the attention area 80. At time t2, the pixel signal is output from the region of interest 80 while controlling the pixel memory 414 so as to read out the pixel signal a stored in the pixel memory n illustrated in FIG. As a result, the pixel signal b accumulated from time t1 to time t2 is output from the demultiplexer 413, and this pixel signal b and the pixel signal a read from the pixel memory n are added by the adder n. Is done. The pixel signal a + b after the addition is output as the signal B through the arithmetic circuit 415. Since the pixel signal B (= a + b) is the sum of the pixel signals accumulated from the time t0 to the time t1 and from the time t1 to the time t2, it is accumulated during the second accumulation time (time t0 to time t2). This corresponds to the pixel signal to be processed.

  As described above, in the first embodiment, since the pixel signal B having a high signal level accumulated in the second accumulation time longer than the first accumulation time is used for the focus detection process, the pixel signal A having a low signal level is focused. Compared with the case where it is used for detection processing, high focus detection accuracy can be obtained. In addition, since the pixel signal A accumulated in the first accumulation time shorter than the second accumulation time is used for exposure calculation, white balance adjustment value determination, and live view image display, it is possible to calculate with high accuracy while avoiding the influence of noise. A live view image that is easy to view without being too bright can be obtained.

<Description of flowchart>
FIG. 9 is a flowchart for describing the flow of the photographing operation executed by the control unit 70 of the imaging device 1 in the first embodiment. The control unit 70 repeatedly activates the processing of FIG. 9 when an unillustrated ON-OFF switch is turned on and energization is performed for each unit of the imaging apparatus 1. In step S101 of FIG. 9, the control unit 70 determines control parameters such as a frame rate and a gain for the attention area 80 and the peripheral area 90, and proceeds to step S102. For example, values to be applied in steps S102 and S110, which will be described later, are read out from the program data and prepared.

  In step S <b> 102, the control unit 70 sends an instruction to the drive unit 21 to start live view imaging by the imaging unit 20. Acquisition of the live view image starting in step S102 is performed, for example, by setting control parameters for the peripheral region 90 for substantially the entire imaging surface of the imaging device 100.

  In step S <b> 103, the control unit 70 sends an instruction to the drive unit 21 to transfer the first data from the imaging unit 20. The first data corresponds to the pixel signal A (= a) read out at the time t1. In step S <b> 104, the control unit 70 causes the image processing unit 30 to perform image processing on live view image data based on the pixel signal A output from the imaging unit 20, and then causes the display unit 50 to display the live view image data.

  In step S <b> 105, the control unit 70 sends an instruction to the drive unit 21 to transfer the second data from the imaging unit 20. The second data corresponds to the pixel signal B (= a + b) read out at the time t2. In step S <b> 106, the control unit 70 causes the AF calculation unit 71 to perform AF calculation (detection of a focus adjustment state) based on the pixel signal B for the AF calculation region. Thereby, the focus of the imaging optical system 10 can be adjusted. Note that since the AF calculation area is not determined in the first frame after the power-on operation, the AF calculation is performed on substantially the entire imaging surface of the image sensor 100.

  In step S107, the control unit 70 sends an instruction to the image processing unit 30 to extract the main subject region from the live view image. In step S108, the control unit 70 sets an area including the main subject as the attention area 80, determines the attention area 80 as an AF calculation area, and proceeds to step S109. The AF calculation area is applied to the live view image acquired in the next frame.

  In step S109, the control unit 70 determines whether a release operation has been performed. The release operation is used as a main imaging instruction for the imaging apparatus 1. When a release button (not shown) is pressed, the control unit 70 makes a positive determination in step S109 and proceeds to step S110. If no press operation is performed, the control unit 70 makes a negative determination in step S109 and proceeds to step S112. move on.

  Note that when a tap operation on the release icon displayed on the liquid crystal display panel 51 is detected to determine the main imaging instruction, an affirmative determination may be made in step S109 when the displayed release icon is tapped.

  In step S110, the control unit 70 sends an instruction to the drive unit 21, and control parameters (exposure time, gain, etc.) necessary for the exposure conditions for photographing determined based on the first data (pixel signal A (= a)). Is set and the process proceeds to step S111.

  In step S111, the control unit 70 performs a photographing process, stores the acquired image data in a memory card or the like by the recording unit 60, and returns to step S102. In the shooting process, after applying a shooting control parameter set in common (same) to all areas of the image sensor 100 to acquire (main image) and record a single frame, and after acquiring a still image Also get multi-frame images. Then, based on the images of a plurality of frames acquired during a predetermined time before and after the release operation, slow reproduction moving image data is generated and recorded. Slow playback moving image data refers to moving image data that is played back at a frame rate (for example, 15 fps) slower than the frame rate (for example, 30 fps) obtained by the image sensor 100.

  The control unit 70 generates slow playback moving image data as follows. That is, the first time temporarily stored in the work memory 40 for displaying the above-described live view image from the release operation time (tx) before the pre-shooting time (for example, 0.6 seconds before) to the time tx. A plurality of frame images based on one data (pixel signal A (= a)) (for example, 0.6 frames when the frame rate is acquired at 30 fps is 18 frames), and later shots from time tx to time tx 0 when a plurality of frame images (for example, the frame rate is acquired at 30 fps) based on the first data (pixel signal A (= a)) stored in the work memory 40 by the time (for example, after 0.4 seconds) Slow playback moving image data is generated on the basis of 12 frames for 4 seconds). As a result, playback is performed based on a plurality of frame images (a total of 30 images) stored in the work memory 40 for one second (0.6 seconds before time tx to 0.4 seconds after time tx) with time tx interposed therebetween. Slow playback moving image data having a time of about 2 seconds is generated. In this way, slow playback moving image data based on frame images acquired before and after the release operation is obtained. The slow playback moving image data is generated by the image processing unit 30 as MPEG data or JPEG data.

  In step S112, which proceeds after making a negative determination in step S109 described above, the control unit 70 determines whether or not a power-off operation has been performed. The controller 70 makes an affirmative determination in step S112 when an unillustrated ON-OFF switch is turned off, performs a predetermined power-off process, and ends the process of FIG. If the unillustrated ON-OFF switch is not operated to turn off the power, the controller 70 makes a negative determination in step S112 and returns to step S102. When returning to step S102, the above-described processing is repeated.

According to the first embodiment described above, the following operational effects can be obtained.
(1) The image sensor 100 includes a demultiplexer 413 that inputs data from a plurality of pixels that perform photoelectric conversion, a pixel memory 414 that stores the input data for each pixel, data stored in the pixel memory 414, and the above An adder 416 for adding the input data for each pixel and an arithmetic circuit 415 for outputting the input data or the data added by the adder 416 are provided. Thereby, since photoelectric conversion data can be processed appropriately, the usability of the image sensor 100 can be improved.

(2) Since the pixel memory 414 stores the data added by the adder 416 for each pixel, the input data can be integrated. Thereby, photoelectric conversion data can be integrated and output, or can be output without integration.

(3) Since the adder 416 adds data with different photoelectric conversion times for each pixel, data corresponding to a case where the photoelectric conversion time is changed to be longer can be obtained.

(4) Since the drive control unit 417 that reads the data stored in the pixel memory 414 and adds the adder 416 to the data at the timing when the data from the pixel is input is provided. Can be controlled.

(5) When the data output from the arithmetic circuit 415 is the input data, exposure calculation, determination of the white balance adjustment value, and display of the live view image are performed, and the data output from the arithmetic circuit 415 is addition data. Since focus detection processing is performed in some cases, photoelectric conversion data and processing using the data can be appropriately combined.

(6) Since it is determined by analyzing the image constituted by the input data whether to output the input data or the addition data from the arithmetic circuit 415, appropriate photoelectric conversion according to the image Data can be output.

(7) In the above determination, addition data is output for pixels corresponding to the main subject area of the image, and input data is output for pixels corresponding to areas other than the main subject area of the image. Conversion data can be output.

(8) Since the imaging chip 113 having a plurality of pixels and the memory chip 112 for processing photoelectric conversion data are stacked, the imaging element 100 can be configured compactly without increasing the size of each chip in the surface direction.

(9) Since the imaging device 1 as an example of the electronic device includes the memory chip 112 that processes the photoelectric conversion data and the imaging chip 113 having a plurality of pixels, the photoelectric conversion data is appropriately processed and is easy to use. A good imaging device 1 can be provided.

(Second embodiment)
In the first embodiment, an example in which the screen is divided into the attention area 80 and the peripheral area 90 and images having different accumulation times are obtained in the attention area 80 and the peripheral area 90 before the instruction for the main imaging has been described. In the second embodiment, the screen is divided into a bright area 85 and a dark area 95, and images having different accumulation times are obtained in the bright area 85 and the dark area 95 after an instruction for the main imaging. Here, for example, the bright area 85 is an area constituted by pixel signals having a value higher than a predetermined pixel signal value (higher than a predetermined luminance value) by analyzing a live view image. The dark area 95 is an area other than the bright area 85.

  The imaging device 1 according to the second embodiment acquires an image based on pixel signals accumulated in the bright area 85 during the first accumulation time and an image based on pixel signals accumulated in the dark area 95 during the second accumulation time. The two images are combined to obtain an image with a wide dynamic range. For this reason, the control unit 70 sends an instruction to the driving unit 21 when acquiring the recording image, and the pixel signal after the first accumulation time has passed from the dark region 95 of the imaging device 100 (imaging chip 113) and the second. The pixel signal after the accumulation time has elapsed is read out in a plurality of times.

<Reading out pixel signals with different accumulation times>
The pixel signal readout timing, the accumulation signal in the imaging chip 113, and the pixel signal read out from the imaging device 100 via the arithmetic circuit 415 will be described with reference to FIG.

  The drive unit 21 controls the image sensor 100 as follows. That is, the recording image storage start time t0 to t1 after the main imaging instruction is set as the first storage time, and the time t0 to time t2 is set as the second storage time. The drive unit 21 starts charge accumulation for the pixels included in the bright region 85 and the dark region 95 at time t0. At time t1, pixel signals are output from the bright area 85 and the dark area 95 while controlling the pixel memory 414 so as not to read out the pixel signals stored in the pixel memory n illustrated in FIG. As a result, the pixel signal a accumulated during the first accumulation time (from time t0 to time t1) is output from the demultiplexer 413, and is directly output as the signal A via the arithmetic circuit 415. This pixel signal A (= a) is also stored in the pixel memory n.

  Furthermore, when the driving unit 21 reads out the pixel signal at time t1, the driving unit 21 immediately starts charge accumulation for the pixels included in the dark region 95. At time t2, the pixel signal is output from the dark area 95 while controlling the pixel memory 414 so as to read out the pixel signal a stored in the pixel memory n illustrated in FIG. As a result, the pixel signal b accumulated from time t1 to time t2 is output from the demultiplexer 413, and this pixel signal b and the pixel signal a read from the pixel memory n are added by the adder n. Is done. The pixel signal a + b after the addition is output as the signal B through the arithmetic circuit 415. Since the pixel signal B (= a + b) is the sum of the pixel signals accumulated from the time t0 to the time t1 and from the time t1 to the time t2, it is accumulated during the second accumulation time (time t0 to time t2). This corresponds to the pixel signal to be processed.

  As described above, in the second embodiment, since the image of the dark area 95 is formed by the pixel signal B accumulated in the second accumulation time longer than the first accumulation time, the pixel signal having a short accumulation time and a low signal level is formed. Compared with the case where an image of the dark area 95 is formed with A, the image is less likely to be black. Further, since the image of the bright area 85 is formed by the pixel signal A accumulated in the first accumulation time shorter than the second accumulation time, the image of the bright area 85 is formed by the pixel signal B having a long accumulation time and a high signal level. As compared with the case of forming, whiteout of an image is less likely to occur. As a result, an image having a wide dynamic range from the bright part to the dark part can be obtained by performing the accumulation process from time t0 to time t2 only once. In addition, since the first accumulation time and the second accumulation time have the same accumulation start time (both are time t0), the moving subject differs from the conventional technique in which a plurality of images having different disclosure accumulation start times are combined. It is also suitable for shooting.

<Description of flowchart>
FIG. 10 is a flowchart for describing the flow of the photographing operation executed by the control unit 70 of the imaging device 1 in the second embodiment. The control unit 70 repeatedly activates the process of FIG. 10 when an unillustrated ON-OFF switch is turned on and energization is performed for each unit of the imaging apparatus 1. In step S201 in FIG. 10, the control unit 70 determines control parameters such as a frame rate and a gain for the bright region 85 and the dark region 95, and proceeds to step S202. For example, values to be applied in steps S202, S205, and S208 described later are read out from the program data and prepared.

  In step S <b> 202, the control unit 70 sends an instruction to the drive unit 21 to start live view imaging by the imaging unit 20. The acquisition of the live view image starting in step S202 is performed by setting control parameters for the dark region 95, for example, over substantially the entire imaging surface of the imaging device 100.

  In step S203, the control unit 70 determines the bright area 85 and the dark area 95 based on the live view image, and proceeds to step S204. In step S204, the control unit 70 determines whether a release operation has been performed. The release operation is used as a main imaging instruction for the imaging apparatus 1. If a release button (not shown) is pressed, the control unit 70 makes a positive determination in step S204 and proceeds to step S205. If no release operation is performed, the control unit 70 makes a negative determination in step S204 and proceeds to step S211. move on.

  When a tap operation on the release icon displayed on the liquid crystal display panel 51 is detected to determine the main imaging instruction, an affirmative determination may be made in step S204 when the currently displayed release icon is tapped.

  In step S <b> 205, the control unit 70 sends an instruction to the driving unit 21 to start recording imaging by the imaging unit 20. Acquisition of the recording image starting in step S205 is performed by setting control parameters for the bright area 85, for example, over substantially the entire imaging surface of the imaging device 100.

  In step S <b> 206, the control unit 70 sends an instruction to the driving unit 21 to transfer the first data from the imaging unit 20. The first data corresponds to the pixel signal A (= a) read out at the time t1. In step S207, the control unit 70 sends an instruction to the image processing unit 30, temporarily stores the data corresponding to the bright area 85 in the first data in the work memory 40, and proceeds to step S208.

  In step S <b> 208, the control unit 70 sends an instruction to the drive unit 21 to transfer the second data from the imaging unit 20. The second data corresponds to the pixel signal B (= a + b) read out at the time t2. In step S209, the control unit 70 sends an instruction to the image processing unit 30, temporarily stores the second data (that is, data corresponding to the dark area 95) in the work memory 40, and proceeds to step S210.

  In step S210, the control unit 70 sends an instruction to the image processing unit 30, and synthesizes the first data and the second data temporarily stored in the work memory 40. Specifically, the area corresponding to the dark area 95 in the image based on the first data is replaced with the image based on the second data. Thereby, the dark part area | region 95 is comprised by the pixel signal B (= a + b), and the synthetic | combination image comprised by the bright part area | region is comprised by the pixel signal A (= a) is obtained. When the image composition is completed, the control unit 70 returns to step S202. When returning to step S202, the above-described processing is repeated.

  In step S211, which proceeds after making a negative determination in step S204 described above, the control unit 70 determines whether or not a power-off operation has been performed. The controller 70 makes an affirmative determination in step S211 when an unillustrated ON-OFF switch is turned off, performs a predetermined power-off process, and ends the process of FIG. When the unillustrated ON-OFF switch is not operated to turn off the power, the control unit 70 makes a negative determination in step S211 and returns to step S202.

According to the second embodiment described above, the following operational effects can be obtained.
(1) When the data output from the arithmetic circuit 415 is the data input to the demultiplexer 413, the image sensor 100 forms the bright area 85 of the image, and the data output from the arithmetic circuit 415 is the adder. Since the dark area 95 of the image is formed in the case of the data added by 416, an image with a wide dynamic range can be obtained by appropriately combining the photoelectric conversion data and the processing using the data.

(2) Since the operation circuit 415 determines whether to output the input data or the addition data by analyzing the image constituted by the input data, appropriate photoelectric conversion according to the image Data can be output.

(3) In the determination, the input data is output for the pixels corresponding to the region where the luminance of the image is higher than the predetermined value, and the added data is output for the pixels corresponding to the region where the luminance of the image is lower than the predetermined value. Therefore, appropriate photoelectric conversion data can be output.

(Modification 1)
You may comprise the imaging device 1 which concerns on 1st embodiment mentioned above and 2nd embodiment with a highly functional mobile telephone or a tablet terminal. In this case, a camera unit mounted on a high-function mobile phone (or tablet terminal) is configured using the multilayer image sensor 100.

(Modification 2)
In the second embodiment described above, the example in which the image processing unit 30 performs image synthesis using the first data and the second data temporarily stored in the work memory 40 has been described. Instead of this, image composition may be performed using the pixel memory 414 and the adder 416 illustrated in FIGS. 4 and 5. For example, when reading the pixel signal at time t1, control is performed so that only the pixel signal A corresponding to the bright area 85 is selectively read from the image sensor 100. Further, when reading the pixel signal at time t2, control is performed so that only the pixel signal B corresponding to the dark area 95 is selectively read from the image sensor 100.

(Modification 3)
In the first embodiment and the second embodiment described above, the pixel signal after the first accumulation time and the pixel signal after the second accumulation time have elapsed from the predetermined region of the image sensor 100 (image pickup chip 113). The example in which and are read in two steps has been described. The number of times of reading a plurality of times is not limited to two times as described above, but may be three times, four times, or more.

  In addition, for images read out multiple times before the main imaging instruction, an image used for AF calculation, an exposure calculation, an image used for determining a white balance adjustment value, an image used for extracting a main subject, a live view display, respectively The application may be divided as an image used for the image. In addition, for images read out in a plurality of times after the instruction for the main imaging, the usage may be divided according to the brightness of the image as a very bright region, a bright region, a dark region, a very dark region, etc. .

  The above description is merely an example, and is not limited to the configuration of the above embodiment. The configurations of the above embodiments and modifications may be combined as appropriate.

DESCRIPTION OF SYMBOLS 1 ... Imaging device 10 ... Imaging optical system 20 ... Imaging part 30 ... Image processing part 40 ... Work memory 50 ... Display part 51 ... Liquid crystal display panel 52 ... Touch panel 60 ... Recording part 70 ... Control part 71 ... AF calculating part 72 ... AE , AWB arithmetic unit 100 ... imaging element 109 ... bump 111 ... signal processing chip 112 ... memory chip 113 ... imaging chip 131 ... unit area 413 ... demultiplexer 414 ... pixel memory 415 ... arithmetic circuit 416 ... adder 417 ... drive control part

Claims (13)

A data input unit that inputs data from a plurality of pixels that perform photoelectric conversion;
A memory for storing the data input to the data input unit for each pixel;
An adder for adding the data stored in the memory and the data input to the data input unit for each pixel;
A data output unit that outputs data input to the data input unit or data added by the adder; and
A signal processing apparatus comprising: The signal processing device according to claim 1,
The memory further stores the data added by the adder for each pixel. The signal processing device according to claim 1 or 2,
The signal processing apparatus according to claim 1, wherein the adder adds the photoelectrically converted data for different pixels. In the signal processing device according to any one of claims 1 to 3,
A signal processing apparatus comprising: a timing control unit that reads data stored in the memory at a timing when data from the pixel is input to the data input unit and causes the addition unit to perform addition. . In the signal processing device according to any one of claims 1 to 4,
When the data output from the data output unit is the data input to the data input unit, the first process is performed, and the data output from the data output unit is the data added by the adding unit A signal processing apparatus comprising a signal processing unit that performs second processing in some cases. The signal processing device according to claim 5,
The signal processing apparatus characterized in that the second processing is a focus detection calculation. The signal processing device according to claim 5 or 6,
The signal processing apparatus, wherein the first processing is an exposure calculation. In the signal processing device according to any one of claims 1 to 7,
The data output unit analyzes whether the data input to the data input unit is output or the data added by the adding unit is output, and analyzes an image composed of data from the plurality of pixels. A signal processing apparatus comprising a determination unit that determines the The signal processing device according to claim 8,
The determining unit outputs data added by the adding unit for pixels corresponding to a main subject region of the image, and is input to the data input unit for pixels corresponding to regions other than the main subject region of the image. A signal processing device characterized in that the data is output. The signal processing device according to claim 8,
The determination unit outputs data input to the data input unit for pixels corresponding to a region where the luminance of the image is higher than a predetermined value, and the pixel corresponding to a region where the luminance of the image is lower than the predetermined value. A signal processing apparatus that outputs data added by an adding unit. A processing chip having the signal processing device according to any one of claims 1 to 10,
An imaging chip having the plurality of pixels;
An image pickup element characterized by being laminated.   An imaging device comprising the imaging device according to claim 11. The signal processing device according to any one of claims 1 to 10,
An imaging unit having a plurality of pixels;
An electronic device comprising:

Images