JP6089419B2 - Imaging circuit, imaging method, imaging apparatus, and program - Google Patents

Description

  The present invention relates to an image pickup circuit, an image pickup method, an image pickup apparatus, and a program for picking up an image of a subject with an image pickup element.

  Currently, imaging devices are used in various terminals, most of which employ CMOS (Complementary Metal Oxide Semiconductor) image sensors. This CMOS image sensor is advantageous in terms of package size reduction and high-speed transfer compared with a CCD image sensor, but has a problem that focal plane distortion occurs due to rolling shutter driving. The focal plane distortion is a distortion of a captured image that occurs due to exposure of the image sensor in units of lines and sequential reading.

In the case of a terminal such as a digital camera equipped with a mechanical shutter, even when a CMOS image sensor is installed, global shutter driving can be realized in still image shooting, and focal plane distortion can be removed.
However, when the mechanical shutter is mounted on the terminal, the lens unit itself becomes large.
Therefore, Patent Document 1 discloses a technique for correcting focal plane distortion without using a mechanical shutter.

JP 2007-142929 A

  However, in the technique described in Patent Document 1, while determining the motion vector of the motion region, the imaging interval of the frame, the exposure start time difference based on the position in the frame, and the exposure start order based on the position in the frame, The image is corrected by software processing. Therefore, there is a problem that complicated software processing is required.

Even when a mechanical shutter is installed, if a mechanical shutter such as moving image shooting or high-speed continuous shooting cannot be used, shooting is performed using a rolling shutter, and thus focal plane distortion may occur in the same manner.
As described above, in the conventional technique, it is difficult to suppress the distortion of the captured image resulting from the exposure and reading of the image sensor in units of lines with a simple configuration.

  The present invention has been made in view of such a situation, and an object of the present invention is to suppress distortion of a captured image that occurs due to exposure and reading of an image sensor in units of lines with a simple configuration.

In order to achieve the above object, an imaging circuit of one embodiment of the present invention is an imaging circuit of an imaging element, and the pixels included in the imaging element are second pixels of the same color that are close to each other on the same line as the first pixels. When the exposure of the first pixel by the first exposure means and the first exposure by the first exposure means are completed, the second exposure is completed. A second exposure unit that starts exposure with all the pixels simultaneously, and at the time when the first line exposure by the first exposure unit is completed, the first exposure unit and the second exposure unit from the first line. the reading of the image data that has been exposed by the exposure means, La sequentially reading means for performing in-units, wherein after the exposure by the first exposure unit, the first pixel and the second exposure image data that has been exposed in Compensation with image data exposed by means Comprising a correction means for the said is perpendicular to the direction of thinning out the direction and the read image data read, characterized in that.

  ADVANTAGE OF THE INVENTION According to this invention, the distortion of the captured image which generate | occur | produces in order to perform exposure and reading of an image pick-up element per line can be suppressed with a simple structure.

It is a block diagram which shows the structure of the hardware of the imaging device which concerns on one Embodiment of this invention. It is a mimetic diagram for explaining an image sensor concerning one embodiment of the present invention. It is a circuit diagram which shows the imaging circuit at the time of making it operate | move in correction | amendment mode. It is a schematic diagram for demonstrating the read-out procedure of pixel data. . It is a functional block diagram which shows the functional structure for performing an imaging process among the functional structures of the imaging device of FIG. It is a schematic diagram for demonstrating the image data comprised by the correction pixel. 6 is a flowchart illustrating a flow of imaging processing executed by the imaging apparatus of FIG. 1 having the functional configuration of FIG. 5. It is a flowchart explaining the flow of a correction | amendment imaging process among imaging processes. It is a schematic diagram which shows the modification which changed the magnitude | size of the effective pixel and cancellation pixel in an image sensor.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of an imaging apparatus 1 according to an embodiment of the present invention.
The imaging device 1 is configured as a digital camera, for example.
Further, in the present embodiment, the imaging apparatus 1 generates captured image data by a focal plane shutter system that captures an image using a shutter installed in the immediate vicinity of the image plane. Note that the imaging apparatus 1 may employ various mechanisms such as a drum type and a square type as long as it is a focal plane shutter type shutter.

  The imaging apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an imaging unit 16, and an operation unit 17. A display unit 18, a storage unit 19, a communication unit 20, a drive 21, and a battery 22.

  The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 19 to the RAM 13.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

  The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output interface 15 is also connected to the bus 14. An imaging unit 16, an operation unit 17, a display unit 18, a storage unit 19, a communication unit 20, and a drive 21 are connected to the input / output interface 15.

  The imaging unit 16 includes an optical lens unit (not shown), an image sensor (not shown), and an imaging circuit 100. Details of the imaging circuit 100 will be described later.

The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS type photoelectric conversion element. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. Through various signal processing, a digital signal is generated and output as an output signal of the imaging unit 16.
Such an output signal of the imaging unit 16 is hereinafter referred to as “captured image data”. The captured image data is appropriately supplied to the CPU 11.

The operation unit 17 includes various buttons and the like, and inputs various types of information according to user instruction operations.
The display unit 18 includes a display, a speaker, and the like, and outputs an image and sound.
The storage unit 19 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 20 controls communication performed with other devices (not shown) via a network including the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 21. The program read from the removable medium 31 by the drive 21 is installed in the storage unit 19 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19.

  The battery 22 is constituted by, for example, a lithium ion secondary battery, and is a power supply source that supplies driving power to the entire imaging apparatus 1.

Next, an image sensor mounted on such an image pickup apparatus 1 will be described.
FIG. 2 is a schematic diagram for explaining an image sensor according to an embodiment of the present invention.
FIG. 2A is a schematic diagram illustrating a pixel array of the image sensor.
FIG. 2B is a schematic diagram for explaining the number of pixels of the image sensor.

In the imaging device according to the present embodiment, pixels of the same area to which color filters of the same color are attached are arranged in pairs. The image sensor has pixels of each color arranged in the same pattern in units of two lines.
First, in the first line of the imaging pixels, a red (RED) pixel pair ([R], [r]) and a green (GREEN) pixel pair ([G], [g]) are present. Alternatingly arranged.
In the second line of the imaging pixel, a pair of green (GREEN) pixels ([G], [g]) and a pair of blue (BLUE) pixels ([B], [b]) are alternately displayed. Has been placed.

  In the imaging device configured in this way, each pixel ([R], [r], [G], [g], [B], [b]) is a pixel (hereinafter, “ And “effective pixel”) function as a pixel used to correct the displayed pixel, that is, to subtract (subtract) the value from the pixel value of the effective pixel (hereinafter referred to as “cancel pixel”). Configured as follows.

  In this embodiment, among all the pixels, the pixels [R], [G], and [B] function only as effective pixels, and the pixels [r], [g], and [b] Accordingly, the pixel functions as an effective pixel or a cancel pixel.

  In the imaging apparatus 1 according to the present embodiment, when all the pixels function as effective pixels, [r], [g], and [b] that form a pair of [R], [G], and [B] pixels. The case where these pixels function as cancel pixels can be selected as a mode. Specifically, a mode in which all pixels function as effective pixels (hereinafter referred to as “normal shooting mode”) and a pair of [R], [G], and [B] pixels [r], A mode in which the pixels [g] and [b] function as cancel pixels (hereinafter referred to as “correction mode”) is configured as a selectable mode.

  Incidentally, as shown in FIG. 2B, the image sensor is configured by the number of pixels of 2m pixels × n pixels (m and n are natural numbers). When the image sensor is operated in a normal shooting mode in which all pixels function as effective pixels (number of pixels: 2 m × n), among all the pixels, [R], [G], [B ] [R], [g], and [b], which are paired with pixels, are operated in a correction mode in which the pixels function as cancel pixels (number of pixels: m × n). It will operate with a number of effective pixels.

Next, the imaging circuit 100 when operated in the correction mode will be described.
FIG. 3 is a circuit diagram illustrating the imaging circuit 100 when operated in the correction mode.

  As illustrated in FIG. 3, the imaging circuit 100 includes two types of photodiodes 101 and 102 that can be controlled differently, and an effective pixel reset switch 103 and a cancel pixel reset switch 104 that are provided in each of the photodiodes 101 and 102. An amplifier circuit 105 and a read switch 106 are provided.

  The photodiodes 101 and 102 correspond to the [R], [G], and [B] pixels, respectively, and the effective pixel photodiode 101 that functions as an effective pixel and the [r], [g], and [b] pixels. Correspondingly, it is constituted by a cancel pixel photodiode 102 that functions as a cancel pixel.

The effective pixel photodiode 101 operates to start exposure in response to an imaging instruction.
That is, the effective pixel photodiode 101 is connected to the effective pixel reset switch 103 on the cathode side, and when the effective pixel reset switch 103 is turned on, it is reset by releasing the accumulated charge to the power source. When the effective pixel reset switch 103 is turned off, the charge is again accumulated and exposure is started. The effective pixel photodiode 101 has a cathode connected to the + side input terminal of the differential amplifier circuit 105, and outputs the charge (pixel data) accumulated by exposure as a + side input signal of the differential amplifier circuit 105.

The cancel pixel photodiode 102 operates to start exposure at the timing when the exposure of the effective pixel photodiode 101 is completed.
That is, the cancel pixel photodiode 102 is connected to the cancel pixel reset switch 104 on the cathode side, and when the cancel pixel reset switch 104 is turned on, it is reset by releasing the accumulated charge to the power source. When the cancel pixel reset switch 104 is turned off, the charge is accumulated again and exposure is started. Further, the cancel pixel photodiode 102 is connected to the negative input terminal of the differential amplifier circuit 105 on the cathode side, and a voltage value (pixel value) corresponding to the charge accumulated by exposure is applied to the negative input signal of the differential amplifier circuit 105. Output as.

The effective pixel reset switch 103 operates to reset the effective pixel photodiode 101 in response to the start of imaging.
The effective pixel reset switch 103 has a control terminal connected to the effective pixel reset line 107, and in accordance with an input signal to the control terminal, the cathode side of the effective pixel photodiode 101 and the + side input terminal of the differential amplifier circuit 105. Switch the continuity with.

The cancel pixel reset switch 104 operates to reset the cancel pixel photodiode 102 at the timing when the exposure of the effective pixel photodiode 101 is completed.
The cancel pixel reset switch 104 is connected to a cancel pixel reset line 108, and the continuity between the cathode side of the cancel pixel photodiode 102 and the negative side input terminal of the differential amplifier circuit 105 is determined in accordance with an input signal to the control terminal. Switch.

The difference amplifying circuit 105 performs a subtraction process for subtracting the pixel value of the cancel pixel of the cancel pixel photodiode 102 paired with the effective pixel from the pixel value of the effective pixel of the effective pixel photodiode 101 as an exposure correction process. Generated pixels (hereinafter referred to as “correction pixels”) are generated. As a result of the subtraction, image data composed of correction pixels is output.
The differential amplifier circuit 105 is connected to the read switch 106.

The readout switch 106 is a switch for reading out correction pixels. The read switch 106 operates so as to read each pixel for each line of the image sensor.
The read switch 106 has a control terminal connected to the line selection line 109, and switches conduction between the output terminal of the differential amplifier circuit 105 and the signal line 110 in accordance with an input signal to the control terminal.

  In the imaging device 1, hereinafter, the effective pixel photodiode 101 and the effective pixel reset switch 103 that controls resetting of the effective pixel photodiode 101 are referred to as “effective pixel exposure unit”. That is, the effective pixel exposure unit 111 includes the effective pixel photodiode 101 and the effective pixel reset switch 103.

The cancel pixel photodiode 102 and the cancel pixel reset switch 104 that controls resetting of the cancel pixel photodiode 102 are referred to as a “cancel pixel exposure unit”. That is, the cancel pixel exposure unit 112 includes the cancel pixel photodiode 102 and the cancel pixel reset switch 104.
The differential amplifier circuit 105 and the readout switch 106 that controls the output of the pixel are referred to as an “exposure correction unit”. That is, the exposure correction unit 113 includes the differential amplifier circuit 105 and the readout switch 106.

In the imaging circuit 100 configured as described above, when there is an imaging instruction, the effective pixel photodiode 101 is first reset by the effective pixel reset switch 103. Thereafter, exposure is started in the reset effective pixel photodiode 101.
When the exposure of the effective pixel photodiode 101 is completed, the cancel pixel photodiode 102 is reset by the cancel pixel reset switch 104. Thereafter, exposure is started in the reset cancel pixel photodiode 102.

  When the exposure is completed in each of the effective pixel photodiode 101 and the cancel pixel photodiode 102, the readout switch 106 reads out the pixels for each line of the image sensor. Then, by reading out pixel data, the effective pixel photodiode 101 on the same line and the signal output from the cancel pixel photodiode 102 (signals indicating the effective pixel and the cancel pixel) are input to the differential amplifier circuit 105. Is done.

When a signal indicating the effective pixel and the cancel pixel is input to the differential amplifier circuit 105, the correction pixel is generated by subtracting the pixel value of the cancel pixel corresponding to the effective pixel from the pixel value of the effective pixel. The
As a result, in the imaging circuit 100, pixel data is output via the signal line 110.

Next, a pixel data reading procedure and image data generated as a result of reading will be described.
FIG. 4 is a schematic diagram for explaining a pixel data reading procedure.
FIG. 4A is a schematic diagram for explaining a read pattern of pixel data.
FIG. 4B is a schematic diagram for explaining a relationship between reset and readout of each pixel for outputting one frame of image data.

Pixel data is read for each line as shown in FIG. That is, in the pixel data, effective pixels and cancel pixels on the same line are read out in the same readout period.
In the case of such a pixel readout pattern, the exposure time of the image sensor of each line is between the first line (first line) readout and the last line readout. There is a difference. That is, while one line is being read out, the line that has not been read out is exposed, so that a difference in the amount of exposure that occurs is generated. The image data generated by this difference in exposure amount has a difference in luminance between the upper area where the readout time is early and the lower area where the readout time is late, or the image data ( (Focal plane distortion) may occur. That is, the captured image is distorted due to exposure of the image sensor in line units and sequential readout.

For this reason, in the imaging circuit 100 according to the present embodiment, the excess exposure amount in the effective pixel is detected as the exposure amount of the cancel pixel, and correction processing for subtracting the cancel pixel from the effective pixel is performed, thereby correcting the corrected image data. Is output.
In detail, in the imaging circuit 100, as shown in FIG. 4B, first, the effective pixels of all lines are reset, the exposure of effective pixels is started, and the exposure of effective pixels is completed. The cancel pixels of all lines are reset and exposure of the cancel pixels is started. Next, in the imaging circuit 100, when the exposure of the cancel pixel is started, readout of the pixels in each line unit, that is, readout of the pixels from the first line to the last line is sequentially performed at predetermined time intervals.

Thereby, in the cancel pixel, the exposure is performed from the end of the exposure of the effective pixel to the end of the readout, and at the time of readout, the charge for the time from the start of readout is accumulated in the cancel pixel.
By subtracting the pixel value of the cancel pixel that has accumulated the charge for the time from the start of reading from the pixel value of the effective pixel, the exposure state corresponding to the case where the reading start time is the same for all the effective pixels of the line It becomes. That is, when the image data is composed of correction pixels, there is no difference in the exposure time of the image sensor between the upper area where the readout time is early and the lower area where the readout time is late. Therefore, it is possible to output image data without distortion of the captured image such as focal plane distortion that occurs due to exposure of the image sensor in line units and sequential readout.

Note that when the imaging circuit 100 is used in the normal shooting mode, the output signal of the effective pixel photodiode 101 and the output signal of the cancel pixel photodiode 102 are each independently extracted to the signal line 110.
That is, in addition to the configuration shown in FIG. 3, a switch (not shown) that switches conduction between the cathode side of the effective pixel photodiode 101 and the signal line 110 according to an input signal to the control terminal, and an input signal to the control terminal Accordingly, a switch (not shown) that switches conduction between the cathode side of the cancel pixel photodiode 102 and the signal line 110 is provided. By controlling the conduction of these switches, the pixel values of the effective pixel photodiode 101 and the cancel pixel photodiode 102 can be extracted as effective pixels. When the imaging circuit 100 is used in the correction mode, these switches are always turned off.

Next, a functional configuration for executing the imaging process among the functional configurations of the imaging apparatus 1 will be described.
FIG. 5 is a functional block diagram illustrating a functional configuration for executing the imaging process among the functional configurations of the imaging apparatus 1 in FIG. 1.
The imaging process is a series of processes for outputting image data by performing exposure control of an imaging element corresponding to the imaging mode from selection of the imaging mode.

When executing the imaging process, in the CPU 11, as shown in FIG. 5, the mode operation control unit 41, the imaging control unit 42, the captured image acquisition unit 43, and the display control unit 44 function.
A captured image storage unit 61 is provided as an area of the storage unit 19 of the imaging device 1. Note that the captured image storage unit 61 is provided as an area of the storage unit 19, and may be provided as an area of the removable medium 31.

The mode operation control unit 41 selects a mode by a mode selection operation from the operation unit 17 and performs an operation according to the selected mode. The imaging circuit 100 (effective pixel exposure unit 111, cancel pixel exposure unit 112, The exposure correction unit 113) is controlled.
The mode operation control unit 41 selects a normal shooting mode or a correction mode.
When the normal shooting mode is selected, the mode operation control unit 41 controls the effective pixel exposure unit 111 and the cancel pixel exposure unit 112 to cause all the pixels to function as effective pixels. In the normal photographing mode, image data having twice the number of pixels (2m × n) as the correction mode is output.
On the other hand, when the correction mode is selected, the mode operation control unit 41 controls the effective pixel exposure unit 111, the cancel pixel exposure unit 112, and the exposure correction unit 113 so as to control the effective pixel and the effective pixel. A pixel of the same color adjacent to the pixel is caused to function as a cancel pixel. In the correction mode, image data of the number of pixels (m × n) is output in order to set half of the pixels as cancel pixels that are paired with effective pixels.
As described above, the mode operation control unit 41 controls each switch of the imaging circuit 100 according to the set mode, and controls exposure and reading of image data in units of lines.

  The imaging control unit 42 performs imaging control such as an imaging instruction on the imaging unit 16. Specifically, an instruction to start exposure or the like is issued, and image data is output from the imaging unit 16.

  The captured image acquisition unit 43 acquires captured image data output from the imaging unit 16. Then, the captured image acquisition unit 43 performs control to store the acquired image data in the captured image storage unit 61. In addition, the captured image acquisition unit 43 outputs to the display control unit 44 that the captured image data has been acquired.

  The display control unit 44 receives the captured image acquired by the captured image acquisition unit 43, and stores the captured image (captured image captured in the normal capture mode and captured in the correction mode) stored in the captured image storage unit 61. Control is performed to display on the display unit 18 any one of the captured images in which the effective pixels are corrected. As a result, the display unit 18 displays either a captured image captured in the normal capturing mode or a captured image in which effective pixels captured in the correction mode are corrected.

Next, image data output from the imaging device 1 described above will be described.
FIG. 6 is a schematic diagram for explaining image data composed of correction pixels.
FIG. 6A is a schematic diagram illustrating an example of image data in a case where only effective pixels are configured.
FIG. 6B is a schematic diagram illustrating an example of image data in a case where only cancellation pixels are configured.
FIG. 6C is a schematic diagram illustrating an example of image data in the case where the correction pixel is used.
In this example, the following description will be made assuming that reading is performed sequentially from the top of the subject image.

In the example of FIG. 6A in which the image data is composed of only effective pixels, a luminance error and a shape distortion occur in the lower part of the subject image (a line with a later acquisition time) due to the readout time lag.
Further, in the example of FIG. 6B in which the image data is composed only of cancel pixels, it consists only of luminance error and shape distortion components due to the readout time lag, compared to the example of FIG. The image is emphasized in luminance error and shape distortion.
In the imaging circuit 100, image data is configured from correction pixels obtained by subtracting cancel pixels from effective pixels. Accordingly, an image having no distortion or luminance difference as shown in FIG. 6C displayed as a subtraction result obtained by subtracting FIG. 6B from FIG. 6A is output.

Next, it is a flowchart explaining the flow of the imaging process which the imaging device 1 which has a functional structure as mentioned above performs.
FIG. 7 is a flowchart for explaining the flow of imaging processing executed by the imaging device 1 of FIG. 1 having the functional configuration of FIG.

  The imaging process is executed, for example, when the user performs an operation for starting the imaging process on the operation unit 17.

In step S1, the mode operation control unit 41 determines whether or not the current setting is a correction mode. That is, the mode operation control unit 41 determines whether the mode is the correction mode or the normal shooting mode according to the mode setting operation performed by the user on the operation unit 17.
If it is not the correction mode, that is, if it is the normal photographing mode, NO is determined in step S1, and the process proceeds to step S5. The processing after step S5 will be described later.

  On the other hand, in the correction mode, YES is determined in step S1, and the process proceeds to step S2.

  In step S <b> 2, the mode operation control unit 41 operates the cancel pixel as a cancel pixel to expose the image sensor. That is, the mode operation control unit 41 operates the cancel pixel exposure unit 112 (specifically, pixels of [r], [g], and [b]) to function as a cancel pixel.

In step S3, the mode operation control unit 41 performs a correction imaging process.
The corrected imaging process is a process of outputting a corrected pixel by subtracting the pixel value of the pixel imaged by the cancel pixel exposure unit 112 from the pixel value of the pixel imaged by the effective pixel exposure unit 111 by the exposure correction unit 113. is there. The flow of the correction imaging process will be described later.

In step S4, the mode operation control unit 41 outputs image data composed of correction pixels. The image data output at this time is image data having the number of pixels of m × n.
Specifically, image data as shown in FIG. 6C is output.
After such a process in step S4, the imaging process ends.

  In step S5, the mode operation control unit 41 operates the cancel pixel as an effective pixel to expose the image sensor. That is, the mode operation control unit 41 operates the cancel pixel exposure unit 112 (specifically, pixels of [r], [g], and [b]) to function as effective pixels.

  In step S6, the mode operation control unit 41 resets effective pixels (all pixels) in all lines, and starts exposure of effective pixels. That is, the mode operation control unit 41 operates the effective pixel reset switch 103 and the cancel pixel reset switch 104 to reset all the pixels (effective pixels and cancel pixels).

In step S7, the mode operation control unit 41 outputs image data including pixels treated as all effective pixels. The image data output at this time is image data having the number of pixels of 2m × n.
After such a process of step S7, the imaging process ends.

Next, the flow of the corrected imaging process executed in step S3 in the imaging process will be described.
FIG. 8 is a flowchart for explaining the flow of the corrected imaging process in the imaging process.

  In step S <b> 21, the mode operation control unit 41 resets effective pixels on all lines and starts exposure of effective pixels. That is, the mode operation control unit 41 operates the effective pixel reset switch 103 to reset the effective pixels.

  In step S <b> 22, the mode operation control unit 41 resets the cancel pixels for all lines at the exposure end timing of the effective pixels, and starts exposure of the cancel pixels. That is, the mode operation control unit 41 operates the cancel pixel reset switch 104 to reset the cancel pixel.

  In step S23, the mode operation control unit 41 outputs the pixel values of the effective pixels and the cancellation pixels in order from the first line to the exposure correction unit 113 at the timing of the end of exposure of the effective pixels, and starts reading out the pixels.

  In step S <b> 24, the mode operation control unit 41 operates the exposure correction unit 113 so as to subtract the cancel pixels that are exposed by the difference in exposure time from the first line. That is, the mode operation control unit 41 operates the readout switch 106, and as a result, the exposure correction unit 113 operates to perform correction for subtracting the cancel pixel from the effective pixel. Thereafter, the process returns to the imaging process.

As described above, the imaging circuit 100 includes the effective pixel exposure unit 111, the cancel pixel exposure unit 112, and the exposure correction unit 113.
The pixels constituting the image sensor of the imaging circuit 100 are a pair of cancel pixels of the same color that are adjacent to each other on the same line as the effective pixels.
The effective pixel exposure unit 111 starts exposure with effective pixels simultaneously for all lines.
The cancel pixel exposure unit 112 starts the exposure with the cancel pixel at the same time when the exposure of the first line by the effective pixel exposure unit 111 is completed.
When the exposure of the first line by the effective pixel exposure unit 111 is completed, the exposure correction unit 113 reads the image data exposed by the effective pixel exposure unit 111 and the cancel pixel exposure unit 112 from the first line in units of lines. Do it sequentially.
In addition, after the exposure by the effective pixel exposure unit 111 is completed, the exposure correction unit 113 has exposed the image data (pixel value) exposed by the effective pixels while being read by the exposure correction unit 113 by the cancel pixel exposure unit 112. Correct with image data (pixel value).
For this reason, in the imaging circuit 100, exposure is started at different timings for a pair of effective pixels and cancel pixels, and image data (effective pixels) is generated with an exposure amount (pixel value of the cancel pixel) corresponding to an unnecessary exposure time. Therefore, it is possible to suppress the distortion of the captured image that occurs due to the exposure and reading of the image sensor in units of lines with a simple configuration.

Further, the exposure correction unit 113 corrects the focal plane distortion caused by the exposure time by the effective pixel exposure unit 111 in which the readout order becomes longer as the later line is corrected by the image data exposed by the cancel pixel exposure unit 112.
Therefore, in the imaging circuit 100, it is possible to remove the focal plane distortion caused by the shake of the subject that could not be removed conventionally and the focal plane distortion caused by the shake required for the shake sensor with a simple configuration.

The exposure correction unit 113 corrects the focal plane distortion by subtracting the pixel value of the pixel exposed by the effective pixel exposure unit 111 from the pixel value of the pixel exposed by the cancel pixel exposure unit 112.
Therefore, the imaging circuit 100 can remove the focal plane distortion by a simple process called a subtraction process.

Further, the image data exposed by the cancel pixel exposure unit 112 is image data having only a focal plane distortion component.
Therefore, in the image pickup circuit 100, the cancel pixel becomes image data used only for correction, and therefore the process of generating the imaged image data can be simplified.

In addition, the imaging apparatus 1 according to the present embodiment performs imaging by switching the function of the imaging element of the imaging circuit 100 according to the shooting mode. That is, the imaging apparatus 1 includes a mode operation control unit 41.
The mode operation control unit 41 controls whether the cancel pixel is operated as the cancel pixel or operated in the same operation as the effective pixel according to a plurality of mounted shooting modes.
For this reason, in the imaging apparatus 1, functionality and operability with a high degree of freedom can be obtained by properly using the operation of the cancel pixel.

Further, the mode operation control unit 41 operates the cancel pixel as the cancel pixel in the shooting mode for correcting the focal plane distortion among the plurality of shooting modes, and operates in the normal shooting mode with the same operation as the effective pixel. Take control.
For this reason, in the imaging device 1, according to the imaging | photography mode selected, distortion suppression and the output of the image data of a high pixel number can be used properly.

Next, modified examples of the effective pixel and the cancel pixel of this embodiment will be described.
[First Modification]
In the above-described embodiment, two pixels having the same area are paired to function as effective pixels and cancel pixels. In the present modification, the effective pixel and the cancel pixel are configured to have a predetermined area ratio.
FIG. 9 is a schematic diagram illustrating a first modification in which the sizes of the effective pixel and the cancel pixel are changed.
FIG. 9A is a schematic diagram illustrating a configuration of an image sensor in which the area ratio of the effective pixel and the cancel pixel to be paired is 2: 1.
FIG. 9B is a schematic diagram illustrating a configuration of an image sensor in which the area ratio of the effective pixel and the cancel pixel that are paired is 3: 1.
As shown in FIGS. 9A and 9B, the size (area ratio) of the effective pixel and the cancel pixel can be variously different.

When configured as in this example, control is performed to correct the value of the cancel image so as to be a conversion value equivalent to that of the effective pixel, in accordance with the set area ratio.
Thereby, even if the effective pixel and the cancel pixel are different in size, it is possible to perform appropriate correction.
Note that the effective pixel and the cancel pixel do not necessarily have a corresponding positional relationship. Even in the case where the positions of the image sensor arrays are adjacent to each other, it is considered that the error can be neglected in view of the recent increase in pixel density.

[Second Modification]
In the above-described embodiment, pixels having the same color are provided adjacent to the same line of the image sensor, and each function as an effective pixel and a cancel pixel. In this modification, the function of the present invention is realized using an image sensor having a normal Bayer array. Specifically, using the nearest line having the same array pattern in the image sensor as a pair, the pixel of the same color in one line is set as an effective pixel, and the other pixel of the same color is set as a cancel pixel. Can be made.
In this case, although the existing CMOS sensor can be used, the accuracy of correction and the number of effective pixels are reduced. Therefore, as a method for suppressing a deterioration in image quality (particularly jaggies), it is considered effective to use pixels of the same color arranged side by side on the same line as cancel pixels.

In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.
For example, in the above-described embodiment, the exposure correction unit 113 is provided in the image sensor. However, the effective pixel exposure unit 111 and the cancel pixel exposure unit 112 are provided in the image sensor, and effective pixels are used in subsequent processing. It is also possible to subtract the pixel value of the cancel pixel from this pixel value. In this case, a CPU that performs pixel value subtraction processing functions as the exposure correction unit 113.

  In the above-described embodiment, the exposure correction unit 113 is provided in the imaging circuit 100. However, the present invention is not limited to this, and the exposure correction unit 113 may be provided outside the imaging circuit 100.

In the above-described embodiment, the imaging apparatus 1 to which the present invention is applied has been described using a digital camera as an example, but is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having a corrected imaging processing function. Specifically, for example, the present invention can be applied to a notebook personal computer, a printer, a television receiver, a video camera, a portable navigation device, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 5 is merely an example, and is not particularly limited. That is, it is sufficient that the imaging apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional block is used to realize this function is not particularly limited to the example of FIG.
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which a program is recorded, the hard disk included in the storage unit 19 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalents thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
An imaging circuit of an imaging element,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
First exposure means for simultaneously starting exposure by the first pixel on all lines;
Second exposure means for starting exposure by the second pixels at the same time for all lines at the time when exposure of the first line by the first exposure means is completed;
When the exposure of the first line by the first exposure unit is completed, the image data exposed by the first exposure unit and the second exposure unit is sequentially read from the first line in units of lines. Means,
A correction unit that corrects the image data exposed by the first pixel with the image data exposed by the second exposure unit after the exposure by the first exposure unit;
An imaging circuit comprising:
[Appendix 2]
The correction means uses the image data exposed by the second exposure means as the image data exposed by the first pixel while the reading means reads out after the exposure by the first exposure means. The imaging circuit according to appendix 1, wherein correction is performed.
[Appendix 3]
The reading means simultaneously reads the first pixel and the second pixel located on the same line, and sequentially reads the first pixel and the second pixel located on different lines at a predetermined time interval. The imaging circuit according to appendix 1 or 2, characterized by:
[Appendix 4]
The correction means corrects the focal plane distortion due to the exposure time of the first exposure means which becomes longer as the subsequent line is corrected with the image data exposed by the second exposure means. The imaging circuit according to any one of the above.
[Appendix 5]
The correction means corrects the focal plane distortion by subtracting the image data read by the reading means from the image data exposed by the second exposure means. Imaging circuit.
[Appendix 6]
6. The imaging circuit according to appendix 5, wherein the image data exposed by the second exposure means is image data of only the focal plane distortion component.
[Appendix 7]
The imaging circuit according to any one of appendices 1 to 6, wherein the first pixel and the second pixel have different sizes.
[Appendix 8]
The imaging circuit according to appendix 7, wherein the correction unit controls a correction amount based on the image data exposed by the second exposure unit based on an area ratio between the first pixel and the second pixel. .
[Appendix 9]
An imaging method for controlling an imaging circuit of an imaging element,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
A first exposure step for starting exposure by the first pixel simultaneously for all lines;
A second exposure step of starting the exposure by the second pixels at the same time for all the lines at the time when the exposure of the first line by the first exposure step is completed;
When the exposure of the first line in the first exposure step is completed, the readout of the image data exposed in the first exposure step and the second exposure step from the first line is sequentially performed in line units. Steps,
A correction step of correcting the image data exposed in the first pixel with the image data exposed in the second exposure step after completion of the exposure in the first exposure step;
An imaging method comprising:
[Appendix 10]
An imaging apparatus that performs imaging using an imaging element of an imaging circuit,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
The imaging circuit
First exposure means for simultaneously starting exposure by the first pixel on all lines;
Second exposure means for starting exposure by the second pixels at the same time for all lines at the time when exposure of the first line by the first exposure means is completed;
When the exposure of the first line by the first exposure unit is completed, the image data exposed by the first exposure unit and the second exposure unit is sequentially read from the first line in units of lines. Means,
A correction unit that corrects the image data exposed by the first pixel with the image data exposed by the second exposure unit after the exposure by the first exposure unit;
With
The imaging device
Operation control means for controlling whether the second pixel is operated as the second pixel or operated in the same operation as the first pixel in accordance with a plurality of shooting modes mounted on the imaging apparatus; An imaging apparatus comprising:
[Appendix 11]
The operation control means causes the second pixel to operate as the second pixel in a shooting mode for correcting focal plane distortion among the plurality of shooting modes, and in the normal shooting mode, The imaging apparatus according to appendix 10, wherein control is performed to perform an equivalent operation.
[Appendix 12]
A program executed by a computer mounted on an imaging device that performs imaging using an imaging element of an imaging circuit,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
The imaging circuit
First exposure means for simultaneously starting exposure by the first pixel on all lines;
Second exposure means for starting exposure by the second pixels at the same time for all lines at the time when exposure of the first line by the first exposure means is completed;
When the exposure of the first line by the first exposure unit is completed, the image data exposed by the first exposure unit and the second exposure unit is sequentially read from the first line in units of lines. Means,
A correction unit that corrects the image data exposed by the first pixel with the image data exposed by the second exposure unit after the exposure by the first exposure unit;
With
An operation control function for controlling whether to operate the second pixel as the second pixel or an operation equivalent to the first pixel in accordance with a plurality of photographing modes mounted on the imaging apparatus;
A program that functions as
[Appendix 13]
A program executed by a computer that controls imaging of an imaging device,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
A first exposure control function for simultaneously starting exposure by the first pixel on all lines;
A second exposure control function for starting the exposure by the second pixels at the same time for all lines at the time when the exposure of the first line by the first exposure control function is completed;
When the exposure of the first line by the first exposure control function is completed, the readout of image data exposed by the first exposure control function and the second exposure control function from the first line is read in line units. Sequential read control function;
A correction function for correcting the image data exposed by the first pixel with the image data exposed by the second exposure control function after completion of the exposure by the first exposure control function;
A program that functions as

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Bus, 15 ... Input-output interface, 16 ... Imaging part, 17 ... Operation unit, 18 ... display unit, 19 ... storage unit, 20 ... communication unit, 21 ... drive, 22 ... battery, 31 ... removable media, 41 ... mode operation control , 42... Imaging control unit, 43... Captured image acquisition unit, 44... Display control unit, 61. Diode 102: Cancel pixel photodiode 103: Effective pixel reset switch 104 ... Cancel pixel reset switch 105 ... Differential amplifier circuit 106: Read switch 111 ... Effective pixel Light unit, 112 ... cancellation pixel exposure unit, 113 ... exposure correction unit

Claims (11)

An imaging circuit of an imaging element,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
First exposure means for simultaneously starting exposure by the first pixel on all lines;
Second exposure means for starting exposure by the second pixels at the same time for all lines at the time when exposure of the first line by the first exposure means is completed;
When the exposure of the first line by the first exposure means is completed, the reading of the image data that has been exposed by the from the first line the first exposure means and said second exposing means, successively in line units Reading means for performing;
A correction unit that corrects the image data exposed by the first pixel with the image data exposed by the second exposure unit after the exposure by the first exposure unit;
Equipped with a,
The reading direction of the image data and the direction of thinning out the reading are orthogonal to each other,
An imaging circuit characterized by the above.   The correction means uses the image data exposed by the second exposure means as the image data exposed by the first pixel while the reading means reads out after the exposure by the first exposure means. The imaging circuit according to claim 1, wherein correction is performed.   The reading means simultaneously reads the first pixel and the second pixel located on the same line, and sequentially reads the first pixel and the second pixel located on different lines at a predetermined time interval. The imaging circuit according to claim 1 or 2.   2. The correction unit corrects a focal plane distortion due to an exposure time by the first exposure unit, which becomes longer as a subsequent line is corrected by image data exposed by the second exposure unit. 4. The imaging circuit according to any one of items 3.   The correction means corrects the focal plane distortion by subtracting the image data read by the reading means from the image data exposed by the second exposure means. Imaging circuit.   6. The imaging circuit according to claim 5, wherein the image data exposed by the second exposure means is image data of only the focal plane distortion component. An imaging method for controlling an imaging circuit of an imaging element,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
A first exposure step for starting exposure by the first pixel simultaneously for all lines;
A second exposure step of starting the exposure by the second pixels at the same time for all the lines at the time when the exposure of the first line by the first exposure step is completed;
When the exposure of the first line by the first exposure step is completed, the reading of the image data exposed in the first exposure step and said second exposure step from the first line, successively in line units A reading step to perform;
A correction step of correcting the image data exposed in the first pixel with the image data exposed in the second exposure step after completion of the exposure in the first exposure step;
Including
The reading direction of the image data and the direction of thinning out the reading are orthogonal to each other,
An imaging method characterized by the above. An imaging apparatus that performs imaging using an imaging element of an imaging circuit,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
The imaging circuit
First exposure means for simultaneously starting exposure by the first pixel on all lines;
Second exposure means for starting exposure by the second pixels at the same time for all lines at the time when exposure of the first line by the first exposure means is completed;
When the exposure of the first line by the first exposure means is completed, the reading of the image data that has been exposed by the from the first line the first exposure means and said second exposing means, successively in line units Reading means for performing;
A correction unit that corrects the image data exposed by the first pixel with the image data exposed by the second exposure unit after the exposure by the first exposure unit;
With
The reading direction of the image data and the direction of thinning out the reading are orthogonal,
The imaging device
Operation control means for controlling whether the second pixel is operated as the second pixel or operated in the same operation as the first pixel in accordance with a plurality of shooting modes mounted on the imaging apparatus; An imaging apparatus comprising: The operation control means causes the second pixel to operate as the second pixel in a shooting mode for correcting focal plane distortion among the plurality of shooting modes, and in the normal shooting mode, The imaging apparatus according to claim 8 , wherein control is performed so as to perform an equivalent operation. A program executed by a computer mounted on an imaging device that performs imaging using an imaging element of an imaging circuit,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
The imaging circuit
First exposure means for simultaneously starting exposure by the first pixel on all lines;
Second exposure means for starting exposure by the second pixels at the same time for all lines at the time when exposure of the first line by the first exposure means is completed;
When the exposure of the first line by the first exposure means is completed, the reading of the image data that has been exposed by the from the first line the first exposure means and said second exposing means, successively in line units Reading means for performing;
A correction unit that corrects the image data exposed by the first pixel with the image data exposed by the second exposure unit after the exposure by the first exposure unit;
With
The reading direction of the image data and the direction of thinning out the reading are orthogonal,
An operation control function for controlling whether to operate the second pixel as the second pixel or an operation equivalent to the first pixel in accordance with a plurality of photographing modes mounted on the imaging apparatus;
A program to realize A program executed by a computer that controls imaging of an imaging device,
The pixels constituting the image sensor are paired with second pixels of the same color that are close to each other on the same line as the first pixels.
A first exposure control function for simultaneously starting exposure by the first pixel on all lines;
A second exposure control function for starting the exposure by the second pixels at the same time for all lines at the time when the exposure of the first line by the first exposure control function is completed;
Wherein when the first exposure for the first line by the exposure control function has ended, the reading of the image data that has been exposed by the first exposure control and the second exposure control functions from the first line, line Read control function sequentially performed in units,
A correction function for correcting the image data exposed by the first pixel with the image data exposed by the second exposure control function after completion of the exposure by the first exposure control function;
Realized,
The reading direction of the image data and the direction of thinning out the reading are orthogonal to each other,
A program characterized by that.