JP2008166974A - Image blur detection method, imaging apparatus, and image blur detection program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform detection of image blur of a photographed image in high precision and high speed. <P>SOLUTION: When the apparatus photographs an object image by solid imaging elements 21 which has a first pixel group (G pixel group) which is arranged and formed, for example, in checkers shape in an imaging area, and second pixel groups (R pixel group and B pixel group) which are arranged and formed in remained checkers positions of the imaging area, it makes exposure time of the first pixel group shorter than an exposure time of the second pixel groups, and detects motion vectors of the object image by performing comparison processing of image data (G image frame) and image data (RB image frame) read out from the second pixel groups. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image blur detection method, an imaging apparatus, and an image blur detection program that can improve the accuracy of image blur detection.

  When a subject is imaged by an imaging device such as a digital still camera or a digital video camera, camera shake that occurs when the hand holding the imaging device cannot be fixed or subject blur that occurs due to the subject moving faster than the shutter speed occurs. These appear in the captured image as image blur.

  FIG. 6 is an explanatory diagram of the principle of image blur detection. If the image in the block 2 indicated by the predetermined address in the image 1 of the Nth frame shown in FIG. 6A is a reference image, the same predetermined in the image 3 of the (N + 1) th frame shown in FIG. 6B. If the comparison image cut out by the block 4 indicated by the address is the same as the reference image, the comparison image is not blurred with respect to the reference image.

  However, if camera shake or subject blurring has occurred, the reference image in block 2 and the comparative image cut out in block 4 do not match. Therefore, the block 4 in the image 3 is shifted by 4a, 4b, 4c,... In the X direction (horizontal direction) and the Y direction (vertical direction) one pixel or several pixels at a time while shifting each block 4a, 4b, 4c. ,... Are compared with the reference image, and the block position where the comparison image having the highest correlation with the reference image is cut out is obtained.

  In the example shown in FIG. 6B, if the comparison image having the highest correlation with the reference image of block 2 is cut out at block 4c, the difference between the reference image of block 2 and the comparison image of block 4c is the vector “ k ". This vector k is a movement vector of the reference image, that is, a blur vector.

  If this movement vector k can be detected on average throughout the entire screen, it can be determined that this movement vector k is a movement vector due to camera shake, and after image 1 is displayed, image 3 is displayed in the opposite direction to movement vector k. If the image is moved and displayed, an image without camera shake can be displayed.

  Further, if this movement vector k can be detected only on a part of the screen, it can be determined that it is a movement vector due to subject blur of the reference image, and if the reference image portion is appropriately corrected, the captured image data that suppresses subject blurring. Can be obtained.

  As a conventional technique related to the movement vector detection technique, there is Patent Document 1 below.

Japanese Patent Laid-Open No. 61-201581

  As described with reference to FIG. 6, when image blur is detected, a reference image cut out from an image of a certain frame and a number of comparison images cut out from the next frame image are compared and calculated, for example, 30 frames per second. In an image pickup apparatus that picks up an image, even if the time required for the comparison calculation process is subtracted before the detection of the image blur, a time delay of [1/30] seconds occurs at the minimum, and the detection accuracy cannot be improved. There's a problem.

  An object of the present invention is to provide an image blur detection method, an imaging apparatus, and an image blur detection program that can shorten the time until image blur is detected and improve the accuracy of image blur detection.

  According to the image blur detection method of the present invention, a pixel group arranged in a two-dimensional array in the imaging region of a solid-state imaging device is divided into a first pixel group and a remaining second pixel group arranged by repeating a predetermined pattern. When capturing a subject image, the exposure time of the first pixel group is shorter than the exposure time of the second pixel group, and the image data read from the first pixel group and the second pixel group are read. A motion vector of a subject image is detected by performing a comparison process with image data.

  The image blur detection method of the present invention is characterized in that the first pixel group is a pixel group arranged in a checkered pattern in the imaging region, and the second pixel group is a pixel group arranged in the remaining checkered positions. To do.

  An imaging device according to the present invention includes a solid-state imaging device having a pixel group arranged in a two-dimensional array in an imaging region, a first pixel group in which the pixel group is arranged by repeating a predetermined pattern, and the remaining second pixels. An image sensor driving means for making an exposure time of the first pixel group shorter than an exposure time of the second pixel group when capturing a subject image divided into groups; image data read from the first pixel group; and Computational processing means for detecting a motion vector of the subject image by comparing the image data read from the second pixel group is provided.

  The imaging device of the present invention is characterized in that the first pixel group is a pixel group arranged in a checkered pattern in the imaging region, and the second pixel group is a pixel group arranged in the remaining checkered positions.

  The arithmetic processing means of the imaging apparatus of the present invention is characterized in that the motion vector is detected as a motion vector distribution within one screen.

  The arithmetic processing unit of the imaging apparatus according to the present invention is characterized in that motion estimation of a subject image is performed based on image data read from the first pixel group.

  The image blur detection program of the present invention divides a pixel group arranged in a two-dimensional array in an imaging region of a solid-state imaging device into a first pixel group and a remaining second pixel group arranged by repeating a predetermined pattern. An image blur detection program for processing captured image data obtained by making an exposure time of the first pixel group shorter than an exposure time of the second pixel group when capturing a subject image, The motion vector of the subject image is detected by comparing the read image data with the image data read from the second pixel group.

  The image blur detection program of the present invention is characterized in that the first pixel group is a pixel group arranged in a checkered pattern in the imaging region, and the second pixel group is a pixel group arranged in the remaining checkered positions. To do.

  According to the present invention, since the image data of the same frame is composed of the first image frame and the second image frame having different exposure times, the first image frame and the second image frame are compared and calculated, so that the short exposure is performed. A blur vector resulting from the time difference can be calculated, and blur correction, blur prediction, motion prediction, moving object detection, moving object tracking, and the like can be performed with high accuracy and high speed.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of a digital camera according to an embodiment of the present invention. This digital camera includes a solid-state imaging device 21, an analog signal processing unit 22 that performs analog processing such as automatic gain adjustment (AGC) and correlated double sampling processing on analog image data output from the solid-state imaging device 21, and an analog signal. An analog / digital conversion unit (A / D) 23 that converts analog image data output from the processing unit 22 into digital image data, and an A / D 23, analog signal processing unit according to instructions from a system control unit (CPU) 29 described later 22, a drive control unit (including a timing generator) 24 that controls the drive of the solid-state imaging device 21, and a flash 25 that emits light in response to an instruction from the CPU 29.

  The digital camera of the present embodiment further includes a digital signal processing unit 26 that takes in digital image data output from the A / D 23 and performs interpolation processing, white balance correction, RGB / YC conversion processing, and the like. A compression / decompression processing unit 27 that compresses or decompresses the image data, a display unit 28 that displays a menu or the like, displays a through image or a captured image, and a system control unit that performs overall control of the entire digital camera ( CPU) 29, an internal memory 30 such as a frame memory, and a media interface (I / F) unit 31 that performs interface processing between a recording medium 32 that stores JPEG image data and the like, and a bus that interconnects them 40, and an operation unit 33 for inputting an instruction from the user is connected to the system control unit 29. It has been.

  FIG. 2 is a schematic diagram of the surface of the solid-state imaging device 21 shown in FIG. The solid-state image pickup device 21 in the illustrated example is a CCD (charge coupled device) type solid-state image pickup device, and includes a plurality of photodiodes 51 arranged in a square lattice pattern on a semiconductor substrate, and the right side of each photodiode row. Provided at the vertical charge transfer path (VCCD) 52 provided, the horizontal charge transfer path (HCCD) 53 provided connected to the end of each vertical charge transfer path 52, and the output end of the horizontal charge transfer path 53 The amplifier 54 is provided.

  “R”, “G”, and “B” described on each photodiode 51 indicate the colors (R = red, G = green, B = blue) of the color filters stacked on the photodiode, and are so-called Bayer arrays. This is a color filter array. When the photodiode 51 with the R filter is called “R pixel”, the photodiode 51 with the G filter is called “G pixel”, and the photodiode with the B filter is called “B pixel”, the solid state of the present embodiment. In the imaging device 21, G pixels are arranged in a checkered pattern in the imaging region, and R pixels and B pixels are arranged in the remaining checkered positions.

  A read gate (read electrode film) 55g also serving as a vertical transfer electrode film is provided between the G pixel 51 and the right vertical charge transfer path 52, and the R pixel 51 and the B pixel 51 and the right vertical charge transfer path. A read gate (read electrode film) 55rb also serving as a vertical transfer electrode film is provided between the gate electrode 52 and the gate electrode 52. The arrangement of the vertical transfer electrode films in the vertical charge transfer path 52 is a well-known configuration capable of progressive readout of all pixels in the present embodiment, but may be a configuration capable of interlace readout.

  In the solid-state imaging device 21 of the present embodiment, the signal charges are read out to the vertical charge transfer path 52 by the separate readout pulses for the G pixels and R pixels in the same row, and the G pixels and B pixels in the same row are separated. The signal charge is read to the vertical charge transfer path 52 by the read pulse, but the signal charge is read to the vertical charge transfer path 52 in the R pixel and the B pixel at the same timing.

  When the subject image is captured by the digital camera of FIG. 1 equipped with the solid-state image sensor 21 having such a configuration, the system control unit 29 shown in FIG. 1 issues a command to the drive unit 24, as shown in FIG. First, an electronic shutter pulse is output to the solid-state imaging device 21 and unnecessary charges are discarded on the substrate side. As a result, the electronic shutter is “open” and exposure of each pixel R, G, B is started.

  When signal charges are read out from each pixel to the vertical charge transfer path 52, the signal charge read timing from the R pixel and the B pixel is different from the signal charge read timing from the G pixel. For example, if the shutter time determined according to the exposure is [1/60] second, the signal charge read timing from the R pixel and B pixel is set to [1/60] second after the electronic shutter is opened, The readout timing of the signal charge from the G pixel is earlier than that, for example, [1/90] seconds after the electronic shutter is opened.

  As a result, the signal charge of the G pixel is read out and held in the vertical charge transfer path 52 [1/90] seconds after the electronic shutter is opened, and after the electronic shutter is opened [1/60]. After two seconds, the signal charges of the R pixel and B pixel are read out to the vertical charge transfer path 52. Then, after the signal charges of all the pixels are read onto the vertical charge transfer path, each signal charge is transferred in the direction of the horizontal charge transfer path 53, and then transferred to the output amplifier 54 along the horizontal charge transfer path 53. Then, the output amplifier 54 outputs a voltage value signal corresponding to the amount of transferred signal charges to the analog signal processing unit 22 in FIG. 1 as captured image data. The above operation is performed for each vertical synchronization signal when capturing a moving image or displaying a through image.

  The captured image data captured by the analog signal processing unit 22 is recorded on the recording medium 32 as, for example, JPEG image data by a known digital camera image processing technique, and the through image data is displayed on the display unit 28.

  At this time, since the signal charge amount of the G pixel is a signal charge amount that is less than the shutter time of the R pixel and the B pixel, image processing is performed using the corrected signal charge amount. For example, in the above example, the shutter time [1/90] seconds for the G pixel is [2/3] compared to the shutter time [1/60] seconds for the R pixel and the B pixel. Correction is performed by multiplying the read signal charge amount by [3/2] to make a picture of the subject image.

  When making this picture, the digital camera according to the present embodiment detects image blur using the difference in exposure time between the R pixel, the B pixel, and the G pixel as described below.

  FIG. 4 is a functional configuration diagram of the image blur detection processing device 60 that detects image blur with the digital camera shown in FIG. 1. In this image blur detection processing device 60, when the user inputs an instruction to perform image blur correction processing from the operation unit 33 shown in FIG. 1, the system control unit 29 activates a movement vector (blur vector) detection program. Thus, the system control unit 29 is configured using the subordinate digital signal processing unit 26, the internal memory 30, and the like.

  The image blur detection processing device 60 includes a thinning processing unit 61 that thins out image data output from the solid-state imaging device 21 to generate a reduced image, and R pixels and B pixels of the thinned-out reduced image data. A separation processing unit 62 that separates image data (hereinafter referred to as an RB image frame) and image data of G pixels (hereinafter referred to as a G image frame), and a first that stores the RB image separated by the separation processing unit 62 A memory 63 and a second memory 64 for storing the G image.

  The image blur detection processing device 60 of the present embodiment further includes a first estimation processing unit 65 that captures an RB image frame and performs motion distribution estimation processing, and a second estimation that captures a G image frame and performs motion distribution estimation processing and motion prediction. A processing unit 66, a control unit 67 that controls output to the subsequent stage of the G image frame based on a processing result of the second estimation processing unit 66, an estimation result of the first estimation processing unit 65, an RB image frame, and a second estimation process An image blur distribution estimation processing unit 68 that estimates an integrated blur distribution of a captured image from the processing result of the unit 66 and the G image frame, and an output control unit 69 that performs output control based on the processing result of the blur distribution estimation processing unit 68 With.

  The output control unit 69 performs camera shake correction processing or subject blur correction when outputting captured image data captured from the solid-state imaging device 21 to the internal memory 30 for display or recording.

  FIG. 5A is an explanatory diagram of a movement vector (blur vector) distribution estimation process. In the illustrated embodiment, a captured image of one screen is divided into a total of 16 partial screen data of 4 rows and 4 columns, and movement vector calculation processing is performed for each of the divided screens # 1 to # 16. This calculation process is basically the same as the calculation process described in FIG. 6, and a reference image is cut out from the divided screen, and a cut-out position in the divided screen of the comparative image having a high correlation with the reference image is obtained. A movement vector in the divided screen is obtained.

  Thereby, in this embodiment, as shown in FIG.5 (b), a total of 16 partial movement vectors K1-K16 are calculated. In the illustrated example, only K7 is different from the other partial vectors K1 to K6 and K8 to K16. For this reason, a moving object is photographed in the divided screen # 7, and it can be estimated that the other partial movement vectors K1 to K6 and K8 to K16 are movement vectors due to camera shake.

  Therefore, the shake distribution estimation processing unit 68 outputs the average vector of the partial movement vectors K1 to K6 and K8 to K16 to the output control unit 69 as a hand shake vector. Accordingly, when outputting the N + 1 frame image after outputting the N frame image, the output control unit 69 moves the N + 1 frame image in the direction opposite to the camera shake vector and outputs it. This makes it possible to obtain output data in which camera shake is suppressed.

  In the example shown in FIG. 5B, the shake distribution estimation processing unit 68 outputs the partial movement vector K7 as the subject shake vector to the output control unit 69, apart from the hand shake vector. Thus, when outputting the N + 1 frame image after outputting the N frame image, the output control unit 69 performs sharpness correction or the like on the partial image of the divided screen # 7 to suppress subject blurring.

  In the case of calculating such a camera shake vector and subject blur vector, conventionally, the calculation is performed by comparing the N frame image and the N + 1 frame image. However, in the present embodiment, the N frame image and the N + 1 frame image are calculated. In addition to performing a comparison operation, a comparison operation between the G image frame and the RB image frame in the N frame image can also be performed, and a comparison between the G image frame in the N frame image and the G image frame in the N + 1 frame image can be performed. The calculation can also be performed, and the blur vector detection accuracy is improved as compared with the conventional case.

  For example, in the above example, if the motion prediction result obtained from the N-1 frame image and the G image frame of the N frame image matches the RB image frame of the N frame image, the motion prediction result is correct. Therefore, it is possible to perform motion prediction and blur vector detection with higher accuracy.

  In order to shorten the time for performing the comparison operation between the reference image cut out from the N frame image and the comparison image cut out from the N + 1 frame image, it is effective to narrow the search range for cutting out the comparison image. According to this embodiment, the cut-out position of a highly correlated comparative image can be limited to a narrow range by the movement obtained from the G image frame and the RB image frame of the N frame image, and the movement vector of the movement vector can be limited with high speed and high accuracy. Calculation is possible.

  Further, according to the present embodiment, when a still image is captured, the difference between the G image frame and the RB image frame in the single frame image can be detected as a motion vector, so that camera shake correction in the still image is also performed with high accuracy. It becomes possible.

  The correlation between the reference image and the comparison image can be obtained by the sum of all the pixels of the difference between the image data of each pixel of the reference image and the comparison image (the total pixels in the block 2 described in FIG. 6). The comparative image with the smallest sum is considered to have the highest correlation with the reference image.

  In the case of the present embodiment, since the G image frame and the RB image frame are compared, if the comparison is performed with the same amount of image data, the difference becomes large. Therefore, the luminance component is obtained from the image data of the G image frame, and the luminance component is obtained from the image data of the RB image frame. Since the luminance component of the R image frame and the B image frame is smaller than the luminance component of the G image frame, normalization is performed so as to match the luminance component of the G image frame. Then, the difference between the luminance component of the G image frame and the normalized luminance component of the RB image frame is obtained, and the correlation is obtained from the sum of the total number of pixels of the difference, whereby the G image frame and the RB image frame are obtained. The correlation between can be calculated.

  In the above-described embodiment, in order to speed up the blur vector calculation process, a reduced image is generated by the thinning-out processing unit 61, and a comparison operation between the G image frame and the RB image frame in the reduced image is performed. However, if a high-performance arithmetic processing unit can be used and there is sufficient memory capacity, the blur vector can be calculated using the G image frame and the RB image frame in the original image data without performing the thinning process. Is possible.

  Further, the above-described embodiment describes an optimal embodiment in which the pixel group formed in the imaging region is divided into the G pixel group arranged in a checkered pattern and the R pixel and B pixel group in the remaining checkered positions. However, the pixel group arranged in a two-dimensional array in the imaging region is divided into a first pixel group and a remaining second pixel group that are arranged by repeating an arbitrary predetermined pattern. It is possible to calculate the motion vector of the subject image in the same manner even if it is formed in a stripe shape and divided into the G pixel group and the RB pixel group.

  Furthermore, in the above-described embodiment, the processing has been described as processing inside the digital camera. However, all the image data output from the solid-state imaging device 21 is recorded, and this is taken into an external personal computer or the like. The same processing can be performed by executing a blur vector detection program or an electronic camera shake correction program.

  The image blur detection method and the like according to the present invention can detect an image blur with high accuracy and high speed, and therefore, when applied to a digital camera or the like, a high-quality image can be taken and is useful.

It is a block block diagram of the digital camera which concerns on one Embodiment of this invention. It is a surface schematic diagram of the solid-state image sensor shown in FIG. FIG. 3 is a diagram illustrating drive timings of an R pixel, a G pixel, and a B pixel of the CCD solid-state imaging device illustrated in FIG. It is a functional block diagram of the image blur detection processing apparatus comprised by a program in the digital camera shown in FIG. It is distribution explanatory drawing of an image blur vector. It is image blurring process calculation explanatory drawing.

Explanation of symbols

21 Solid-state imaging device 24 Drive unit 26 Digital signal processing unit 29 System control unit 51 Photodiode (pixel)
52 vertical charge transfer path 53 horizontal charge transfer path 60 image blur detection processing device 62 separation processing unit 63 RB image frame storage memory 64 G image frame storage memory 65 RB image frame motion distribution estimation processing unit 66 G image frame motion distribution estimation processing unit 67 motion prediction unit 68 image blur distribution estimation processing unit 69 output control unit

Claims (8)

  When imaging a subject image by dividing a pixel group arranged in a two-dimensional array in an imaging region of a solid-state imaging device into a first pixel group and a remaining second pixel group that are arranged by repeating a predetermined pattern. By making the exposure time of the pixel group shorter than the exposure time of the second pixel group, the image data read from the first pixel group and the image data read from the second pixel group are compared. An image blur detection method comprising detecting a motion vector of a subject image.   2. The image blur according to claim 1, wherein the first pixel group is a pixel group arranged in a checkered pattern in the imaging region, and the second pixel group is a pixel group arranged in the remaining checkered positions. Detection method.   A solid-state imaging device having a pixel group arranged in a two-dimensional array in the imaging region, and dividing the pixel group into a first pixel group and a remaining second pixel group arranged by repeating a predetermined pattern, and subject images Image pickup device driving means for making the exposure time of the first pixel group shorter than the exposure time of the second pixel group when imaging, image data read from the first pixel group, and reading from the second pixel group An image processing apparatus comprising: an arithmetic processing unit that detects a motion vector of a subject image by performing a comparison process on the image data.   The imaging apparatus according to claim 3, wherein the first pixel group is a pixel group arranged in a checkered pattern in the imaging region, and the second pixel group is a pixel group arranged in the remaining checkered positions. .   The imaging apparatus according to claim 3, wherein the arithmetic processing unit detects the motion vector as a motion vector distribution in one screen.   The imaging apparatus according to claim 3, wherein the arithmetic processing unit performs motion estimation of a subject image based on image data read from the first pixel group.   When imaging a subject image by dividing a pixel group arranged in a two-dimensional array in an imaging region of a solid-state imaging device into a first pixel group and a remaining second pixel group that are arranged by repeating a predetermined pattern. An image blur detection program for processing captured image data obtained by making an exposure time of a pixel group shorter than an exposure time of the second pixel group, wherein the image data read from the first pixel group and the first An image blur detection program for detecting a motion vector of a subject image by performing comparison processing with image data read from a group of two pixels.   The image blur according to claim 7, wherein the first pixel group is a pixel group arranged in a checkered pattern in the imaging region, and the second pixel group is a pixel group arranged in the remaining checkered positions. Detection method.

Images