JP5343736B2 - Digital camera - Google Patents

Description

  The present invention relates to a digital camera.

  In a camera using a rolling shutter (also referred to as a slit rolling shutter) image sensor that controls charge accumulation for each row, distortion (rolling shutter distortion) occurs in the captured image if the camera shakes in the horizontal direction during imaging. The image signal is read out and stored from a wide area on the imaging surface of the image sensor, and the central portion of the read area is used as an image area for display or the like. There is known a digital camera that performs blur correction by shifting an area (see, for example, Patent Document 1).

See JP 2007-264074 A

  However, in the conventional digital camera described above, an image signal in a wide readout area must be once taken into the memory, so that a memory with a large storage capacity is required and the center of the image signal in the wide readout area taken into the memory is required. Since only a part is used as an image region, signal processing must be performed even for an image signal that is not used as an image, and there is a problem that a load is imposed on the signal processing.

In the present invention, a plurality of pixels including an effective pixel column that receives a subject image and outputs an image signal and a non-effective pixel column that outputs a black level signal for determining a black level for the image signal are arranged in a row direction. an imaging element for outputting the images signals arranged in a column direction of the two-dimensionally, an image signal output from the image pickup device, before passing to the subsequent processing circuit, a predetermined pre-stage correction on the image signal a pre-processing circuit for performing processing, a detection means for detecting displacement in the row direction of the camera, a pre-processing circuit is in either et picture image signal of the image signal of each line output from the imaging device captures The setting means for setting the range in units of pixel columns for each row according to the displacement detected by the detection means, and the signal of the capture range set by the setting means from the output signal for each row of the image sensor after I write takes, non-perforated And capture means for capturing a signal of a predetermined range from the signals of the pixel column, characterized in that it comprises a.

  According to the present invention, the capacity of a memory for storing an image signal captured from an image sensor can be reduced, and the minimum image signal necessary for creating an image out of signals output from the image sensor. It is only necessary to process the signal, and the burden of signal processing can be reduced.

The figure which shows the structure of the digital camera of one embodiment A timing chart showing a sequence for fetching an image signal for one row necessary for forming an image from an image signal (raw data (RAW DATA)) output for each row from the image sensor. FIG. 2 is a diagram in which the image signals of each row shown in FIG. 2 are arranged vertically with the tips of the image signals aligned. The figure which shows the state which displayed the image signal of 1 line required in order to form an image side by side on a display screen When the rolling shutter distortion as shown in FIGS. 2 to 4 occurs, the image (a) displayed on the display screen when the distortion correction according to the embodiment is not performed, and the distortion according to the embodiment. The figure which shows the image (b) displayed on a display screen when correction | amendment is implemented The flowchart which shows the taking-in processing of the image sensor output of one embodiment

  FIG. 1 is a diagram illustrating a configuration of a digital camera according to an embodiment. In FIG. 1, illustration and description of devices, circuits, and apparatuses of a digital camera that are not directly related to the present invention are omitted. The present invention can be applied to any digital camera such as a compact digital camera, a single-lens reflex digital camera, and a digital video camera.

  A configuration of a digital camera according to an embodiment will be described with reference to FIG. The image sensor 1 is an imaging element in which pixels are arranged in a two-dimensional manner in the row direction and the column direction. The image sensor 1 captures an image of a subject formed on the imaging surface by the imaging optical system 2 and converts it into an image signal. In this embodiment, a CMOS image sensor will be described as an example of a rolling shutter imaging device that performs charge accumulation control for each row. The shake detection sensor 3 is a horizontal (lateral) shake amount (displacement amount) Δxi (i is a pixel array row number of the image sensor 1 when the camera is held at the normal position, and i = 1, 2, 3,...), And an acceleration sensor or the like can be used. As described above, the image sensor 1 performs charge accumulation control (charge accumulation start (reset of accumulated charge) and termination, output of accumulated charge (image signal), etc.) for each row. The direction of the row is the same direction as the direction of motion detection of the motion detection sensor 3, ie, the horizontal direction.

  A digital front end (DFE) 4 is a signal processing IC, and performs signal processing before image processing by a subsequent signal processing circuit (ASIC) 5, that is, clamping processing and AGC processing (automatic processing for the image signal Sig from the image sensor 1). This is a pre-stage processing circuit that performs pre-stage correction processing such as gain adjustment) and shading processing on the image signal. A signal processing circuit (ASIC) 5 is an image processing IC, and processing related to an image, that is, white balance processing, gamma correction, color interpolation, and processing on the image signal after processing by the processing circuit in the previous stage, that is, the digital front end (DFE) 4, This is a subsequent processing circuit that performs edge enhancement, vignette correction, compression / decompression, and the like. A TG (timing generator) 6 controls the timing of various processes such as charge accumulation, readout of accumulated signals, A / D conversion, and image processing for the image sensor 1, the DFE 4, and the signal processing circuit 5.

  The recording medium 7 is a detachable memory such as a memory card, and records an image. The frame memory 8 is a memory for storing frame images. The monitor 9 is a color LCD that displays a through image and a reproduced image. The CPU 10 controls the DFE 4, the signal processing circuit 5, the TG 6, the recording medium 7, the frame memory 8, and the monitor 9, performs various calculations of the camera, sequence control, display control, etc., and executes an image sensor output capturing program described later. Then, processing for capturing the output signal of the image sensor 1 is performed while correcting the rolling shutter distortion. The operation device 11 includes various operation members for the user to operate the camera.

  Next, a method for capturing the output signal of the image sensor 1 while correcting the rolling shutter distortion according to the embodiment will be described with reference to FIGS. FIG. 2 is a timing chart showing a sequence for fetching an image signal for one horizontal line necessary for forming an image from an image signal (raw data (RAW DATA)) output from the image sensor 1 for each line. It is. The image sensor 1 synchronizes with the falling edge of the horizontal synchronization signal HD (see FIG. 1) pulse shown in FIG. 2A in accordance with the accumulated charge in the pixel column of each row as shown in FIG. A signal (image signal) Sig is output. Then, after outputting the image signals from the first row to the n-th row (i = n) and outputting the image signal for one frame, the first row is again synchronized with the vertical synchronization signal VD pulse (see FIG. 1). Return to the line and output the image signal of the next frame.

In FIG. 2, the horizontal direction necessary for forming an image from the image signals (raw data (RAW DATA)) Sig of the first row, i-th row and (i + k) -th row output from the image sensor 1. A state in which an image signal for one row is captured is illustrated, and illustration of capturing an image signal in another row is omitted. Here, there is no camera shake due to camera shake or the like at the time of capturing the image signal of the first row, and a camera shake amount Δxi in the right direction of the camera occurs at the time of capturing the image signal of the i-th row. It is assumed that a camera shake amount Δxi + k in the left direction of the camera occurs at the time when the image signal of i + k) row is captured. As shown in FIG. 1, the signal processing circuit (ASIC) 5 inputs a blur amount Δxi (i = 1 to n) [m] via the CPU 10, and a rolling shutter caused by the horizontal blur amount Δxi by the following equation. A shift amount Zi [pix] for each pixel for correcting the distortion is calculated.
Zi = g (Δxi) (1)
In equation (1), g () is a function for converting the horizontal blur amount Δxi [m] into the distortion correction shift amount Zi [pix]. In the example shown in FIG. 2, the shift amount Zi is calculated for the blur amount Δxi of the i-th row, and the shift amount Zi + k is calculated for the blur amount Δxi + k of the (i + k) -th row. .

  Camera shake caused by camera shake occurs in the “right and left directions” in the horizontal direction, and in response to this, the image signal capture range for one horizontal line of the image is reduced to the “left direction in the row direction. It is necessary to shift each “right”. Therefore, in order to capture an image signal for one horizontal line from the central portion of the output signal sequence for each pixel array row of the image sensor 1, and to shift the image signal capturing range left and right to correct rolling shutter distortion. The shift margin is provided on the left and right of the output signal train of the image sensor 1.

  In FIG. 2 (d), Zh represents the range of the image signal required for one horizontal line of the image (the range of the area indicated by the diagonal line rising to the right in FIG. 2 (d)) in units of the number of pixels. is there. When the camera shake amount Δxi is 0 (in the case of the first row in FIG. 2), it is not necessary to shift the image signal capture range Zh to the left and right, and the shift amount Zi is 0. In this case, the capturing of the image signal for one horizontal line necessary for forming an image is started from the time when the predetermined number of pixels Zo is delayed from the falling edge of the horizontal synchronizing signal HD pulse. When the image signal is captured, the capture of the image signal is terminated. The regions before and after the image signal capturing range for Zh pixels are shift allowances for shifting the image signal capturing range to the left and right in order to correct rolling shutter distortion.

  In FIG. 2B, HD_strt is a pulse signal for starting the capturing of image signals for one horizontal line. When the camera shake amount Δxi is 0 (in the case of the first row in FIG. 2), the signal processing circuit (ASIC) 5 generates an HD_strt pulse when it is delayed by Zo pixels from the falling edge of the horizontal synchronization signal HD pulse. However (see FIG. 2B), the DFE 4 starts capturing the image signals for one horizontal line necessary for forming an image in synchronization with the falling edge of the HD_strt pulse.

  Next, in the i-th row in which the camera is shaken to the right side by Δxi, the signal processing circuit 5 calculates the shift amount Zi by the above equation (1). Then, as shown in FIG. 2B, the signal processing circuit 5 generates an HD_strt pulse at a time point delayed by (Zo−Zi) pixels from the falling edge of the HD pulse, and the DFE 4 follows the falling edge of the HD_strt pulse. Acquisition of image signals for one horizontal line (image signals for Zh pixels) necessary for forming an image in synchronization is started. That is, the image signal capture range is shifted to the “left side” in the row direction by Zi pixels, compared to when the blur amount Δx is zero.

  On the contrary, in the (i + k) th row in which the camera is shaken to the “left side” by Δxi + d, the signal processing circuit 5 calculates the shift amount Zi + k by the above equation (1). Then, as shown in FIG. 2B, the signal processing circuit 5 generates an HD_strt pulse at a time point delayed by (Zo + Zi + k) pixels from the falling edge of the HD pulse, and the DFE 4 follows the falling edge of the HD_strt pulse. Acquisition of image signals for one horizontal line (image signals for Zh pixels) necessary for forming an image in synchronization is started. That is, the image signal capturing range is shifted to the “right side” in the row direction by Zi + k pixels, compared to when the blur amount Δx is 0.

  FIG. 3 is a diagram in which the image signals in each row shown in FIG. 2 are arranged vertically with the tips of the image signals aligned. When the camera shake amount Δx is 0 (here, the first row), the camera signal shake amount Δx and the camera shake direction with reference to the image signal capture range Zh (the range indicated by the oblique line rising to the right in FIG. 3) Zh Accordingly, the image signal capture range Zh shifts to the left or right in the row direction.

  FIG. 4 shows a state in which image signals for Zh pixels in each row are displayed on the display screen 9 a of the monitor 9. The image signal for Zh pixels in each row is taken in the horizontal direction of the camera when the image signal for one row in the horizontal direction of the image (for Zh pixels) is captured from the image signals output from the image sensor 1 for each row. The image signal capture range is shifted by the shift amount Zi in the row direction according to the blur amount Δxi and the blur direction. Therefore, one image generated by arranging the image signals for Zh pixels in each row vertically with reference to the left end of the display screen 9a is an image in which rolling shutter distortion due to camera shake is corrected.

  FIG. 5 shows an image (a) displayed on the display screen 9a when the rolling shutter distortion as shown in FIGS. 2 to 4 occurs and the distortion correction according to the embodiment described above is not performed. FIG. 7 shows a comparison with the image (b) displayed on the display screen 9a when the distortion correction according to the embodiment described above is performed. When the rolling shutter distortion is not corrected, the image is distorted according to the camera shake amount Δxi and the camera shake direction.

  Thus, in one embodiment, an image is formed by the signal processing circuit (ASIC) 5 according to the camera shake amount Δxi and the camera shake direction from the image signals output from the image sensor 1 for each row. The image signal capturing range for one horizontal line necessary for the determination is determined, only the image signal in the capturing range is captured by the DFE 4, and various processes are performed on the captured image signal by the signal processing circuit 5 and stored in the frame memory 8. To do. Accordingly, various processes are performed only on the image signal necessary for one horizontal line of the image provided for display or the like among the image signals output from the image sensor 1 for each row, so that the image is processed as in the conventional case. Compared to extracting the image signal for one horizontal line necessary for forming an image after performing various processes on all the image signals output from the sensor 1, the processing load on the image signal is reduced. Is done.

  In addition, since only the image signal for one horizontal line necessary for forming an image is captured and stored in the frame memory 8, all the image signals for each row output from the image sensor are stored in the frame memory as in the prior art. Compared to storing, the storage capacity of the frame memory 8 can be reduced.

  Next, a method of capturing the output of an optical black pixel (OB pixel) used mainly for determining the black level (also used for removing dark current included in the image signal) will be described. In the image sensor 1, in addition to effective pixels that output an image signal corresponding to the subject image, OB pixels (non-effective pixels) that output a black level signal for determining a black level for the image signal of the effective pixel are arranged. Has been. The image sensor 1 outputs the black level signal of the OB pixel after outputting the image signal of the effective pixel for each row. As described above, all the signals output from the image sensor are once stored in the frame memory, and the image signal of the effective pixel is extracted from the signal, and the black level signal of the OB pixel is extracted. In one embodiment, only a range necessary for one horizontal line of an image used for display or the like among image signals of effective pixels is captured, so that the black level signal of the OB pixel is not captured as it is.

  Therefore, in this embodiment, a black level signal of an OB pixel in a predetermined range (period) in the output signal for each row of the image sensor 1 is captured. As shown in FIGS. 1 to 3, the signal processing circuit (ASIC) 5 receives a black level signal of the OB pixel after a predetermined time from the falling edge of the horizontal synchronizing signal HD, here after a predetermined number of pixels Bo converted into the number of pixels. An OB_strt pulse for starting capturing is output, and the DFE 4 starts capturing the black level signal of the image sensor 1 in synchronization with the falling edge of the OB_strt pulse, and captures a black level signal for a predetermined number of pixels Bh. Thereafter, the capturing of the black level signal is terminated. Such a black level signal capturing operation is repeated for each row.

  As a result, of the signals output from the image sensor 1 for each row, only the image signals necessary for one row in the horizontal direction of the image provided for display or the like are captured, and the minimum necessary for each row. The black level signal can be captured, and the determination of the black level signal included in the image signal of the effective pixel is performed while reducing the signal processing and image processing burden on the image signal and the black level signal and the storage capacity of the frame memory. Can do.

  FIG. 6 is a flowchart illustrating image sensor output capturing processing according to an embodiment. When the user sets the live view mode or the moving image mode using the operation device 11, the CPU 10 starts executing the program shown in FIG. In step 1, the camera shake detection sensor 3 detects the camera shake amount Δxi (i is the number of rows in the image sensor 1 in the horizontal direction, i = 1 to n) and the camera shake direction. In the subsequent step 2, the signal processing circuit (ASIC) 5 shifts the image signal capturing range shift amount Zi [pix for correcting the rolling shutter distortion caused by the camera shake amount Δxi in the horizontal direction by the above equation (1). ] (The sign of the shift amount indicates the shift direction).

  In step 3, the image signal for Zh pixels is captured by the DFE 4 from the shift position determined by the shift amount Zi [pix] and the shift direction of the calculation result among the image signals output from the image sensor 1. In step 4, the signal processing circuit 5 performs the above-described various signal processing and image processing on the captured image signals for Zh pixels and stores them in the frame memory 8. Next, in step 5, among the black level signals output from the image sensor 1, a black level signal for Bh pixels is captured from the predetermined position described above.

  In step 6, whether the capturing of the image signal and the black level signal for one frame image has been completed, that is, whether the capturing of the image signal and the black level signal for each row from the first row to the n-th row has been completed. If the capturing is not completed, the process returns to step 1 to repeat the above-described processing. When the capturing is completed, the sensor output capturing process is terminated.

  In the above-described embodiment, the shift amount Zi of the capturing range of the image signal of the i-th row is determined based on the camera shake amount Δxi and the shake direction detected in synchronization with the i-th horizontal synchronization signal HD. An example is shown in which the shift direction is calculated to shift the capture range of the image signal in the i-th row. However, since the camera shake cycle is actually slower than the line interval (the time interval of the horizontal sync signal HD), even if the line at the time of detecting the blur does not match the line that actually corrects the distortion. On the actual image, such distortion correction error is not recognized, and it is considered that there is no problem. That is, the shift amount Zi and the shift direction of the image signal capture range are calculated based on the camera shake amount Δxi and the shake direction detected in synchronization with the horizontal synchronization signal HD in the i-th row, for example, the (i + 1) -th row. Alternatively, there is no problem even if the capturing range of the (i + 2) th image signal is shifted. In other words, there is no problem even if the camera shake amount Δxi and the camera shake direction are not detected for each line (for each horizontal synchronization signal HD) but for every several lines.

  In the above-described embodiments and their modifications, all combinations of the embodiments and the modifications are possible.

According to the above-described embodiment and its modifications, the following operational effects can be achieved. First, a plurality of pixels are arranged two-dimensionally in the row direction and the column direction, and output from the image sensor 1 that outputs an image signal of a subject image captured and photoelectrically converted by the plurality of pixels. Before the image signal is transferred to the signal processing circuit (ASIC) 5 in the subsequent stage, a digital front end (DFE) 4 that performs a predetermined upstream correction process on the image signal and the displacement in the row direction of the camera are detected. And a digital front end (DFE) 4 that outputs image signals necessary for forming an image of a subject image from image signals for each row output from the image sensor 1. In accordance with the displacement detected by the blur detection sensor 3, the capture range is set from the processing for setting the pixel column unit for each row and the output signal for each row of the image sensor 1. Since the processing for capturing the signal in the captured range is performed, the minimum signal necessary for forming an image among the signals output from the image sensor 1 can be captured. In addition to reducing the capacity of the frame memory 8 for storing the image signal captured from the image sensor, only the minimum image signal necessary for creating an image out of the signals output from the image sensor 1 is processed. It is possible to reduce the burden of signal processing in the digital front end DFE 4 and the signal processing circuit (ASIC) 5.
In addition, since the digital front end 4 has a setting range for the capturing function and a function for capturing a signal within the set capturing range, the image sensor 1 side remains in the conventional configuration (complexity, large size). The above functions can be realized without increasing the cost and cost.

  In addition, according to the embodiment and the modification thereof, for each row of the image sensor 1, the signal of the capture range is captured from the signal of the effective pixel column, and the signal of the OB pixel column (ineffective pixel column) is captured. Since a predetermined range of signals is captured from the inside, it is possible to determine the black level signal for the image signal of the effective pixel while reducing the signal processing and image processing load for the image signal and the black level signal and the storage capacity of the frame memory. It can be carried out.

DESCRIPTION OF SYMBOLS 1; Image sensor, 3; Shake detection sensor, 4; Digital front end (DFE), 5; Signal processing circuit (ASIC), 6; Timing generator (TG), 10; CPU

Claims (4)

A plurality of pixels including a valid pixel column that receives a subject image and outputs an image signal and a non-valid pixel column that outputs a black level signal for determining a black level for the image signal are arranged in a row direction and a column direction. an imaging element for outputting the images signals arranged in two dimensions,
A pre-processing circuit that performs a predetermined pre-correction processing on the image signal before passing the image signal output from the image sensor to a post-processing circuit;
Detecting means for detecting displacement in the row direction of the camera,
The pre-processing circuit is
Setting the capture range in either et picture image signal of the image signal of each of the lines output from the imaging device, in accordance with the detected said displaced by said detecting means, for setting the pixel column basis for each said row Means,
From the output signal of each of the rows of the image sensor, after I Captures signals set the capture range by the setting unit captures to capture the signal of a predetermined range from among the signals of the effective pixel columns And a digital camera. The digital camera according to claim 1 , wherein
The setting means includes
If the displacement is not detected, set the range including the vicinity of the center of the signal of the effective pixel column as the capture range,
When the displacement is detected, the capturing range is shifted in the row direction according to the displacement amount and the displacement direction. The digital camera according to claim 2 ,
The image sensor starts outputting a signal for each row in response to a horizontal synchronization signal for each row,
The setting means includes
If the displacement is not detected, set the capture range after a predetermined time from the generation of the horizontal synchronization signal,
When the displacement is detected, the predetermined time is changed according to the amount and direction of the displacement. The digital camera according to claim 2 or claim 3 ,
The digital camera according to claim 1, wherein the capturing unit captures the signal of the ineffective pixel at a predetermined position in the plurality of rows of the image sensor regardless of the presence or absence of the displacement.

Images