JP4687619B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program for correcting camera shake and subject shake of an image captured by an imaging device such as a CCD sensor.

  In general, when taking a picture without a supplementary light (strobe) in a dark environment such as a night view, the shutter speed becomes slow to compensate for the lack of light, so camera shake and subject shake (hereinafter referred to as camera shake) Is likely to occur. Such a camera shake problem becomes particularly apparent in a digital camera including an imaging device such as a CCD sensor having a large number of pixels.

  For example, using a digital camera equipped with a 640-pixel imaging device with a horizontal pixel count of VGA class, a subject with a brightness that requires a shutter speed of 1/100 seconds (exposure time of 10 milliseconds) is 5 degrees in the horizontal angle of view. Let us consider taking a picture with (this angle of view is equivalent to the angle of view at the telephoto end of a zoom lens of a popular digital camera). In this case, since the number of pixels per horizontal field angle is 640/5 = 128 pixels, even if it moves very slightly within an exposure time of 10 milliseconds, for example, even if it moves only 0.1 degrees at most, it is 0. As a result, a large camera shake of 1 × 128 = 12.8 pixels occurs.

  As described above, even a VGA class imaging device having a relatively small number of pixels generates a large (as much as 12.8 pixels) camera shake with very little movement (0.1 degree movement). In a digital camera equipped with an imaging device having a larger number of pixels as described above, naturally, a countermeasure against camera shake is indispensable.

<First prior art>
Japanese Patent Laid-Open No. 2004-228561 incorporates a camera shake correction unit (camera shake correction lens, its driving mechanism, etc.) in a part of the photographing optical system, and moves the camera shake correction lens to optically correct the camera shake. The technology of the camera shake correction camera is described. However, in this technique, it is necessary to incorporate a camera shake correction lens, a driving mechanism thereof, and the like in the photographing optical system, which causes the device to be complicated and large, and the cost is unavoidable.

<Second prior art>
The following Patent Document 2 describes a technology that solves the problems of Patent Document 1 described above by correcting camera shake by image processing. The principle of the second prior art is as follows. Consider a case where a subject having a brightness that requires a shutter speed of 1/100 second (exposure time 10 milliseconds) is photographed. Now, this exposure time (10 milliseconds) is sufficient to cause camera shake, but if this 1/10 exposure time (that is, 1 millisecond) is taken, this 1 millisecond is taken. Since the shutter speed is a high-speed shutter of 1/1000 second in terms of shutter speed, almost no camera shake should occur.

  However, simply shortening the exposure time (1 millisecond) will naturally lead to underexposure, so that an image with sufficient brightness cannot be obtained.

  Therefore, in the second prior art, the exposure time is set to 1 / n (10 milliseconds → 1 millisecond), n (n = 10) images are continuously shot, and these n images are combined and exposed. Make up for the shortage.

  Specifically, an image sensor that sequentially captures n preset images (corresponding to the above 10 images) with a short exposure time (corresponding to the above 1 millisecond) that is not affected by camera shake, The brightness correction of the n images and the positional deviation between two temporally adjacent images in the n images are accurate to 1 / m pixels (1 pixel or 1/2 pixel to 1/8 pixel). And a means for correcting the detected positional deviation, and a means for adding and averaging the n images corrected for the positional deviation.

JP-A-6-138529 JP 2000-224460 A

  However, in the second prior art, since the camera shake is corrected by image processing, the disadvantages of the first prior art (complexity of equipment, increase in size and cost increase) can be eliminated, but the following points There is a problem to be solved.

  According to the second prior art, the short exposure time that is not affected by camera shake is set to 1 millisecond, and the exposure time that may cause camera shake is set to 10 milliseconds. The imaging device sequentially captures n preset images with a short exposure time that is not affected by camera shake. Here, n is given by “exposure time that may cause camera shake” ÷ “short exposure time that is not affected by camera shake”, so that the number of images taken by the image sensor in the second prior art example ( n) is 10 milliseconds ÷ 1 millisecond = 10 [sheets].

  With this number of sheets, the required time is almost a moment, so there is no stress on the photographer.

  However, when shooting a darker subject, for example, when taking a subject with a brightness that requires a shutter speed of 1/10 second (exposure time 100 milliseconds), the number of images taken by the image sensor in this case Since (n) is also 100 milliseconds ÷ 1 millisecond = 100 [sheets], the photographer must be prepared for a waiting time of 10 times, and feels a great stress.

  As described above, in the second prior art, the number (n) of images taken by the image sensor is simply set by “exposure time that may cause camera shake” ÷ “exposure time that is not affected by camera shake”. As a result, the value of n when shooting a darker subject increases, and as a result, the technical problem to be solved is that the darker the subject, the greater the stress on the photographer. is there.

  Therefore, the present invention suppresses an increase in the number of shot images when shooting a darker subject by switching the image composition processing according to the accuracy of motion prediction, for example, the brightness of the image, and An object of the present invention is to provide an image processing apparatus, an image processing method, and a program for reducing stress.

According to the first aspect of the present invention, an imaging unit that periodically generates and outputs a plurality of images, a plurality of images output from the imaging unit are sequentially captured, and a motion between the images is predicted. A position correction unit that corrects the positional deviation between the two images, an image synthesis unit that applies a predetermined image synthesis process to sequentially synthesize the images after the positional deviation correction to generate and output a synthesized image, and the motion prediction An estimation unit for estimating accuracy, and when the estimation result of the estimation unit is high in accuracy, an image synthesis process prioritizing image resolution is selected as a predetermined image synthesis process used in the image synthesis unit, while the estimation unit And an image composition processing selection means for selecting image S / N priority image composition processing as the predetermined image composition processing used by the image composition means when the estimation result is low in accuracy. Image processing It is a device.
The invention according to claim 2 is the image processing apparatus according to claim 1, wherein the estimation unit estimates the accuracy of the motion prediction according to brightness of an image output from the imaging unit. .
According to a third aspect of the present invention, the position correction unit further includes a feature point extraction unit that extracts a feature point of an image output from the imaging unit for motion prediction, and the estimation unit includes the feature point. The image processing apparatus according to claim 1, wherein the accuracy of the motion prediction is estimated based on an output of the point extraction unit.
According to the fourth aspect of the present invention, an imaging step for periodically generating and outputting a plurality of images, a plurality of images output from the imaging step are sequentially captured, and motion between these images is predicted. A position correction step for correcting a positional deviation between the two images, an image synthesis step for applying a predetermined image synthesis process to sequentially synthesize the images after the positional deviation correction to generate and output a synthesized image, and the motion prediction An estimation step for estimating accuracy, and if the estimation result of the estimation step is high in accuracy, an image synthesis process prioritizing image resolution is selected as a predetermined image synthesis process used in the image synthesis step, while the estimation step If the estimation result of the image is low in accuracy, the image synthesis process selection is performed to select the image S / N priority image synthesis process as the predetermined image synthesis process used in the image synthesis step. The image processing method which comprises a step.
The invention according to claim 5 is the image processing method according to claim 4, wherein the estimation step estimates the accuracy of the motion prediction according to the brightness of the image output from the imaging step. .
According to a sixth aspect of the present invention, the position correction step further includes a feature point extraction step for extracting a feature point of an image output from the imaging step for motion prediction, and the estimation step includes the feature point. The image processing method according to claim 4, wherein the accuracy of the motion prediction is estimated based on an output of the point extraction step.
According to the seventh aspect of the present invention, an image pickup unit that periodically generates and outputs a plurality of images to a computer, and a plurality of images output from the image pickup unit are sequentially captured, and a motion between the images is detected. A position correcting unit that predicts and corrects a positional deviation between the two images; an image synthesizing unit that applies a predetermined image synthesis process to sequentially synthesize the images after the positional deviation correction to generate a synthesized image; and An estimation unit for estimating the accuracy of motion prediction, and when the estimation result of the estimation unit is high in accuracy, an image synthesis process prioritizing image resolution is selected as a predetermined image synthesis process used in the image synthesis unit, When the estimation result of the estimation means is low in accuracy, an image composition processing selection means for selecting an image S / N priority image composition process as a predetermined image composition process used by the image composition means is realized. Which is a program which is characterized.
The invention according to claim 8 is the program according to claim 7, wherein the estimation means estimates the accuracy of the motion prediction according to the brightness of the image output from the imaging means.
According to a ninth aspect of the present invention, the position correction unit further includes a feature point extraction unit that extracts a feature point of an image output from the imaging unit for motion prediction, and the estimation unit includes the feature point. 8. The program according to claim 7, wherein the accuracy of the motion prediction is estimated based on an output of the point extracting means.

In the present invention, when the accuracy of motion prediction is high, image synthesis with priority for image resolution is applied to perform image synthesis of the output image. On the other hand, when the accuracy of motion prediction is low, image synthesis with priority for image S / N. Is used to synthesize the output image, so a large number of captured images are required even when the accuracy of motion prediction is low (for example, the accuracy of motion prediction between considerably dark images is low). do not do.
Therefore, it is possible to provide an image processing apparatus, an image processing method, and a program that suppress an increase in the number of images to be taken when shooting a darker subject and reduce a photographer's stress.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, taking application to a digital camera as an example. It should be noted that the specific details or examples in the following description and the illustrations of numerical values, character strings, and other symbols are only for reference in order to clarify the idea of the present invention, and the present invention may be used in whole or in part. Obviously, the idea of the invention is not limited. In addition, a well-known technique, a well-known procedure, a well-known architecture, a well-known circuit configuration, and the like (hereinafter, “well-known matter”) are not described in detail, but this is also to simplify the description. Not all or part of the matter is intentionally excluded. Such well-known matters are known to those skilled in the art at the time of filing of the present invention, and are naturally included in the following description.

  FIG. 1 is a configuration diagram of a digital camera showing an example of an application system of the present embodiment. In this figure, the digital camera 1 is necessary for a digital camera including a CPU 2 and the like, which is typically a one-chip microprocessor, and a digital camera appropriately arranged around the controller 3. At least, the following parts are provided.

  The imaging unit 4 includes an optical unit 5 including a photographing lens, a diaphragm mechanism, a focusing mechanism (and a zoom mechanism if necessary), and an imaging device 6 including a two-dimensional image sensor such as a CCD sensor or a CMOS sensor. The Operation of the imaging unit 4 [aperture size and zoom magnification (adjusting the shooting angle of view), focusing, setting of the exposure time (shutter speed) of the imaging device 6, readout operation of the imaging signal, and imaging sensitivity (ISO sensitivity) The setting and the like are controlled by the control from the imaging control unit 7 that has received the imaging operation instruction from the control unit 3. The shooting operation instruction from the control unit 3 is, for example, an instruction to read a through image (an image for checking shooting composition; sometimes a low resolution image), a still image or a moving image (high resolution). For example), and various prior operation instructions such as aperture value setting and zoom magnification setting necessary for the reading operation.

  In response to the shooting operation instruction from the control unit 3, the imaging unit 4 generates a through image, a still image, or a moving image (hereinafter referred to as a frame image) at a predetermined frame rate (typically 30 frames per second). Is read periodically. These frame images are converted into digital signals by a digital converter (not shown) and then temporarily stored in the buffer area 8a of the memory unit 8.

  In addition to the buffer area 8a, the memory unit 8 includes at least each of an image processing area 8b, a work area 8c, and an addition method holding area 8d. These areas 8 a to 8 d including the buffer area 8 a are arbitrarily accessed from the control unit 3, the image processing unit 10, and the display unit 11 via the bus line 9.

  The image processing unit 10 transfers the frame image temporarily stored in the buffer area 8a to the image processing area 8b in accordance with an instruction from the control unit 3, and performs a required compression process (for example, JPEG) on the transferred frame image. Compression processing), and the stored data in the image processing area 8b is overwritten with the compressed image. The control unit 3 transfers the stored data (compressed image) in the image processing area 8b to the external storage unit 17 described later and stores it.

  Further, the image processing unit 10 transfers a compressed image read from an external storage unit 17 described later to the image processing area 8b in accordance with an instruction from the control unit 3, and performs a required decompression process (for the transferred compressed image). For example, JPEG decompression processing) is performed, and the stored data in the image processing area 8b is overwritten with the decompressed image.

 The display unit 11 is composed of a flat display device such as a liquid crystal display. Normally, the through image temporarily stored in the buffer area 8a is sequentially read and displayed for checking the shooting composition. Sometimes, a captured image before compression processing is read out from the image processing area 8b and displayed, or when an image is reproduced, a reproduced image after expansion processing is read out from the image processing area 8b and displayed. Furthermore, the display unit 11 displays various setting screens (for example, menu drawings) that are appropriately output from the control unit 3.

  The program code storage unit 12 holds various software resources. This software resource realizes an operating system (OS) corresponding to the architecture of the CPU 2 and various functional operations of the digital camera 1 (photographing operation, image reproduction operation and other management operations, etc.) in cooperation with the OS. Control program to be included. These software resources are read (loaded) from the program code storage unit 12 to the work area 8c of the memory unit 8 as necessary, and are executed by the CPU 2 of the control unit 2.

  The operation unit 13 includes various operators necessary for the user operation input interface of the digital camera 1, such as a shutter key 13a for performing still image shooting and moving image shooting, and a mode for switching between image shooting and image playback modes. Including a key 13b.

  The computer interface unit 14 is, for example, a data input / output unit compatible with a general-purpose protocol such as USB or IEEE 1394. It is possible to transfer and read captured images between them.

  The external storage interface unit 16 inputs and outputs image data and the like with the external storage unit 17, and the external storage unit 17 stores the image data and the like supplied from the external storage interface unit 16.

  The external storage unit 17 is composed of a non-volatile mass storage device such as a flash memory, a hard disk, or an optical disk (the stored contents are not lost even when the power is turned off). Used to store and store the generated images. Each image stored and stored is, for example, a compressed file such as a JPEG format or an uncompressed raw data file (so-called RAW file). The external storage unit 17 may be a fixed type, or may be a general-purpose memory device that can be detached from the digital camera 1 and attached to the external personal computer 15, for example.

  The power supply unit 18 includes a rechargeable secondary battery or a disposable primary battery, and supplies a power supply voltage necessary for the operation of each unit of the digital camera 1 including the control unit 3.

Here, details of the imaging device 6 will be described.
FIG. 2 is a structural diagram of the imaging device 6. Here, a CCD sensor having N columns × M rows of pixels is taken as an example. The imaging device 6 two-dimensionally arranges N × M photoelectric conversion elements 6a for accumulating charges according to the amount of incident light in a matrix, and N vertical transfer units 6b, one for each column. To form an imaging region 6c, and further, a horizontal transfer portion 6d is arranged on the lower side of the imaging region 6c in the drawing. The signal charges accumulated in the photoelectric conversion element 6a are taken into an adjacent vertical transfer unit 6b in response to a read signal (not shown), and the inside of the vertical transfer unit 6b is synchronized with a vertical transfer clock (not shown). Sequentially transferred downward in the drawing. The output ends of all the vertical transfer units 6b are connected to the horizontal transfer unit 6d. The horizontal transfer unit 6d sequentially takes in signal charges for one line in synchronization with the vertical transfer clock. The signal charge taken into the horizontal transfer unit 6d is sequentially transferred in the left direction in the drawing in synchronization with a horizontal transfer clock (not shown), and the signal charge that has reached the output end of the horizontal transfer unit 6d is provided at the same end. The electric charge detection unit 6e converts the electric signal into an electric signal, and the amplifier 6f amplifies the electric signal. Then, the electric signal is taken out from the terminal 6g as a pixel unit image signal (CCD output).

  Here, for convenience, O1, E1, O2, E2, O3, E3,... Are assigned to the horizontal alignments (scanning lines or lines) of the photoelectric conversion elements 6a. However, O is an abbreviation for odd numbers, and E is an abbreviation for even numbers. O1, O2, and O3 indicate odd lines, and E1, E2, and E3 indicate even lines. The interlaced CCD sensor can switch between two modes of field reading (also referred to as field accumulation) and frame reading (also referred to as frame accumulation) from the outside. In the field readout, the CCD output is in the order of O1 + E1, O2 + E2, O3 + E3,..., But in the frame readout, the CCD output is taken out in two times, an odd frame and an even frame. The output is in the order of O1, O2, O3,..., And the CCD output is in the order of E1, E2, E3,.

  As described above, the image pickup device 6 typified by a CCD sensor that two-dimensionally arranges a large number of photoelectric conversion elements 6 a and outputs an image signal corresponding to light from a subject outputs an image signal including color information as it is. Can not. This is because the photoelectric conversion element 6a merely generates a signal corresponding to the intensity of light brightness.

  In order to obtain color information, three image pickup devices are used, and red, green, and blue color filters are attached to each of the image pickup devices, and red, green, and blue image signals are extracted from the respective image pickup devices. A plate-type color imaging device is known. This imaging device is mainly used for studio TV cameras that emphasize color reproducibility, but for home video cameras and popular digital cameras, size, weight, and power consumption. A “single-plate color imaging device” in which one imaging device and a mosaic color filter are combined is used. The same applies to the imaging device 6 of the present embodiment.

  FIG. 3 is a diagram illustrating the imaging device 6 and the color filter 19. Each cell of the imaging device 6 is a pixel including one photoelectric conversion element 6 a, and each pixel has a one-to-one correspondence with the cell of the color filter 19. Each cell of the color filter 19 has a specific color, and various types are used depending on how the colors are selected and arranged.

FIG. E. It is a principle diagram of a color filter called a Bayer method (or a green checkered method) conceived by Bayer. This method is widely used because the color signal and the luminance signal are well balanced and good color reproducibility can be obtained without depending on the brightness of the subject. In this figure, Y is a filter for obtaining luminance information, C ₁ , C ₂
Is a filter for obtaining color information. The Y filter is arranged in a checkered pattern, the C ₁ filter is arranged in the gap between the odd lines, and the C ₂ line is arranged in the gap between the even lines. The reason why a large number of Y filters are arranged is that luminance information perceives better image resolution and outline sharpness than color information in human vision.

FIG. 5 is a configuration diagram of an actual color filter, where R is a red filter, G is a green filter, and B is a blue filter. Red (R), green (G), and blue (B) are the three primary colors of light. In particular, since green represents the brightness of the subject well, the G filter is also used as a filter for obtaining luminance information. That is, the G filter corresponds to the Y filter described above, and the R filter and the B filter are C ₁ , C ₂ , respectively.
It corresponds to a filter. In the illustrated example, only a 6 × 6 filter is shown, but this is for convenience of explanation. Actually, it is composed of much more filters (mesh).

  FIG. 6 is a pixel array diagram of an interlaced CCD sensor. As shown in this figure, in the case of an interlaced CCD sensor, the color filters are arranged in the same way for odd lines and even lines, but the read pixels are arranged regardless of the field readout or frame readout. If attention is paid, the arrangement (arrangement of color information in accordance with the Bayer method) in FIG. 5 is exactly the same. That is, O1 + E1, O2 + E2, and O3 + E3 combined readout is performed in field readout, while O1, O2, O3,..., E1, E2, and E3 interlaced readout are performed in frame readout. It is the same in that the G / R line and the G / B line are read sequentially.

  Therefore, in this specification, unless otherwise specified, the arrangement of pixels and color filters of the imaging device 6 will be described using the Bayer arrangement shown in FIG. 5 as an example.

  Next, items specific to this embodiment will be described. The technical problem to be solved by the present embodiment is to solve the problems of the conventional technique (the second conventional technique at the beginning) for correcting the camera shake and the subject shake of the image taken by the imaging device 6 such as a CCD sensor. There is. That is, in the second prior art, the exposure time is set to 1 / n (10 milliseconds → 1 millisecond), n (n = 10) images are continuously shot, and these n images are combined and exposed. Although the shortage is compensated for, when shooting a darker subject, for example, the number of images shot by the image sensor when shooting a subject with a brightness that requires a shutter speed of 1/10 second (exposure time 100 milliseconds) ( Since n) is also 100 milliseconds ÷ 1 millisecond = 100 [sheets], the photographer has to be prepared for a waiting time of 10 times, and there is a problem of feeling great stress. The aim of the present embodiment is to suppress an increase in the number of images taken when shooting a darker subject by switching the image composition processing according to the brightness of the image, and to reduce the photographer's stress. is there.

  FIG. 7 is a diagram showing a flowchart of a control program including characteristic items in the present embodiment. This control program is stored in advance in the program code storage unit 12, and is read into the work area 8c of the memory unit 8 in response to an operation action for starting shooting by the user (ie, a pressing operation of the shutter key 13a) ( Loaded) and executed by the CPU 2 of the control unit 3.

  When this flowchart is started, first, the buffer area 8a of the memory unit 8 is initialized (step S1). The initialization is, for example, an operation of writing all 0s in all bits (all pixels) of the buffer area 8a. Next, the imaging unit 4 is controlled to capture the first high-resolution frame image (hereinafter referred to as the first image) (step S2). The photographed first image is taken out from the imaging device 6 of the imaging unit 10, converted into a digital signal by an analog-digital converter (not shown), and then written in the buffer area 8 a of the memory unit 8.

  Next, feature points are extracted from the first image written in the buffer area 8a (step S3). The extracted feature point information is used for motion prediction of the second and subsequent images described later. As the feature point detector, for example, a well-known one such as a Harris corner detector or a corner detector called SUSAN can be used. What is necessary is just to select a thing more than a threshold value as a feature point using these detectors.

  Next, the accuracy of motion prediction is estimated (step S4). “The accuracy of motion prediction” refers to the degree of certainty of motion prediction between images between the first image and the second and later images described later. For example, when the brightness of the first image and the second and subsequent images described later are considerably insufficient (that is, when the image is considerably dark), the contrast of the respective images becomes low. The error of the feature point extracted in S3 becomes large. Therefore, in this case, it can be estimated that the accuracy of motion prediction deteriorates (low accuracy). On the other hand, when the brightness of the first image and the second and subsequent images described later are not so low (that is, when the images are not so dark), the contrast of each image can be obtained moderately. Therefore, the error of the feature point extracted in the above step S3 becomes small. Therefore, in this case, it can be estimated that the accuracy of motion prediction does not deteriorate (high accuracy).

  Thus, since it is considered that there is a certain relationship between the accuracy of motion prediction and the brightness of the image, here, the brightness of the input image (as a measure for estimating the accuracy of motion prediction ( Luminance value). As a method of measuring the luminance value, for example, the luminance Y may be obtained while thinning out the input image, and the average of the luminance Y may be the brightness of the image. The following formula (1) is an example of a calculation formula for obtaining the luminance Y from the RGB components of the image.

  For example, when the thinning-out number is “8” in both the vertical and horizontal directions, x and y in the coordinates (x, y) of the target pixel are expressed by the following equation (2).

  The color of the pixel at coordinates (x, y) in equation (2) is G (record). As shown in a pixel array diagram of the Bayer structure described later (FIG. 8), in the Bayer structure, the G pixel also serves as a pixel of luminance information. Therefore, in the pixel array of the Bayer structure, as shown in the following equation (3), the luminance Y is set to the pixel value (G) of (x, y), the pixel value (blue) of (x + 1, y), ( x, y + 1) and the pixel value (red).

  Here, P (x, y) is a pixel value of (x, y). The brightness Lumi of the image is obtained from the following equation (4) as an average of the luminance Y.

  Here, N is the total number of thinned pixels.

  If the accuracy of motion prediction is estimated from the brightness of the image in this way, then the estimation result of the accuracy of motion prediction is determined (step S5). For example, it is determined whether the brightness of the image is greater than a predetermined value. If the brightness of the image is greater than a predetermined value, it is determined that the accuracy is high (the accuracy of motion prediction is high), and otherwise it is determined that the accuracy is low (the accuracy of motion prediction is low).

  Then, an appropriate image composition process (here, image composition process for image addition) corresponding to each determination result is selected (step S6, step S7), and the selection result is stored in the addition method holding area 8d of the memory unit 8. Store.

  The selected image composition processing in the case of “low accuracy” relates to the addition method of “image S / N priority”, while the selected image composition processing in the case of “high accuracy” is the addition of “image resolution priority”. It is about the law. This is because when it is estimated that the image is not so dark and the motion prediction accuracy is good (high accuracy), camera shake correction may be performed with priority on the image resolution, but conversely, the image is dark and the motion prediction accuracy is good. When it is estimated that there is no image (low accuracy), the output image is produced with a small number of captured images at the expense of resolution, rather than performing camera shake correction (for which a large number of captured images are required). This is because camera shake correction with priority on S / N improvement is better.

  As described above, the accuracy of motion prediction is estimated from the brightness of the image, and the image synthesis process is selected. Information corresponding to the selection result (for example, “image S / N priority” → “0”, “image resolution priority”). "→" 1 ") is stored in the addition method holding area 8d of the memory unit 8. Next, after the motion correction parameter is initialized (no motion) (step S8), information (“0”, “1”) corresponding to the selection result is extracted from the addition method holding area 8d of the memory unit 8, It is determined whether the information is “1” (that is, “image resolution priority”) or “0” (that is, “image S / N priority”) (step S9).

  Here, the reason for initializing the motion correction parameter in step S8 (assuming no motion) is the image composition processing (by the image resolution priority addition method performed in step S10 or step S11, or the image S). This is because the first image is an object to be processed (reference image) and therefore there is no image to be added yet at this stage.

  Specifically, for example, when a model whose motion correction parameter is affine transformation is used, in the image synthesis process, conversion from coordinates (u, v) to coordinates (x, y) is performed using the following equation (5). ) Is used for processing. In addition, following Formula (5) is a homogeneous coordinate expression format.

  In the case of this example, the parameters when there is no motion are in the relationship of (x, y) = (u, v), so the parameters are “a = 1, b = 0, c = 0, d = 1. , E = 0, f = 0 ”.

  In this manner, in step S10 or step S11, the image is selected in accordance with the selection result ("image resolution priority" or "image S / N priority") held in the addition method holding area 8d of the memory unit 8. The processing unit 10 executes image composition processing by the image resolution priority addition method or image S / N priority addition method. The image processing unit 10 performs the coordinate conversion of the above equation (5), reads the data of the corresponding coordinates of the first photographed image recorded in the buffer area 8a of the memory unit 8, and stores the data in the buffer area 8a. Then, the addition processing is performed with the image addition data secured in another area. Since the buffer area 8a has been initialized to “0” in the initialization step S1, after the processing, the image addition data becomes an image reflecting only the first image. Details of each process will be described later.

  Next, it is checked whether the shutter key 13a is still being pressed (step S12). If the shutter key 13a is kept pressed, the second and subsequent high-resolution frame images (hereinafter, the second and subsequent images) are photographed in order to continue the synthesis process (step S16). The captured second and subsequent images are taken out from the imaging device 6 of the imaging unit 10 and converted into digital signals by an analog-to-digital converter (not shown), and then the memory unit 8. Are written in the buffer area 8a.

  Then, it tracks where the feature point of the reference image (the first image at this stage) has moved to the image captured this time (the second image at this stage). A correspondence list with corresponding coordinates of the image taken this time is created (step S17). As this tracking method, for example, a pixel position estimation method using a gradient method, a template matching function, or the like can be used. Then, from the corresponding list created by the tracking method in this way, for example, by using RANSAC (Random Sample Consensus Argorithm) or the like, an erroneous corresponding point is excluded, and from the remaining corresponding point list, the parameter (a) (a) ˜f) is obtained by the least square method (step S18).

  Next, using the motion correction parameter obtained in step S18, image addition is performed by the currently selected image addition method to obtain an output added image having no camera shake or subject shake and having sufficient contrast. The obtained output added image updates the added image data in the buffer area 8a of the memory (step S10 or step S11).

  In this way, while the shutter key 13a is kept pressed, the first, second, third,... Images are continuously shot (continuous shooting), and these images are shot. First, the first image is used as a reference image, and after the motion correction between the reference image and the second image is performed, the image addition of the reference image and the second image is performed. The shutter key 13a presses the reference image with the added image to correct the motion between the reference image and the third image, and to add the image between the reference image and the second image. The process is repeated until the user's finger leaves the shutter key 13a.

  When the user's finger is released from the shutter key 13a, the determination result in step S12 is NO, and necessary development processing for the output added image in the memory buffer area 8a at that time, for example, white balance adjustment or brightness adjustment is performed. Then, after performing gamma (γ) correction and tone correction (step S13), the image processing unit 10 transfers the developed image to the image processing area 8b and performs compression processing such as JPEG (step S14). Finally, the compressed image is stored in the external storage unit 17 or the like (step S15), and the flowchart ends.

  As described above, according to this flowchart, the first image is taken immediately after the shutter key 13a is depressed, the feature points of the image are extracted, and, for example, the accuracy of motion prediction is improved from the brightness of the image. Based on the estimation result, it is determined whether to select “image resolution priority” image synthesis processing or “image S / N priority” image synthesis processing. While the button 13a continues to be pressed, the second and subsequent images are sequentially photographed, and an erroneous corresponding point is identified from the feature point correspondence list between the reference image and the image captured subsequent to the reference image. Based on the removed correspondence list, the parameter of the previous equation (5) is obtained, and the image addition method determined above, that is, “image resolution priority” image synthesis processing or “image S / N superiority” is determined. By performing image addition using image synthesis processing ", a hand shake or subject blur is not, moreover, it is possible to obtain an output addition image of sufficient contrast.

<Image addition method>
Image addition accompanied by motion correction will be described. Here, image addition corrects a positional shift between two images taken continuously in time, and adds information of corresponding pixels of both images so as to overlap the images. That means.

  FIG. 8 is a pixel array diagram of the addition target image. Here, the thing of the Bayer structure demonstrated previously is shown. In this figure, the x-axis represents the column direction, the y-axis represents the row direction, and the coordinates of each pixel are (x, y). As described above, in the Bayer structure, rows of G, B, G, B,... And rows of R, G, R, G,. Array). R is a red pixel, G is a green pixel, and B is a blue pixel, and the ratio of green pixels (G) that are often felt by human eyes is increasing. This is because it also serves as a pixel for luminance information. In addition, a two-digit subscript attached to each of R, G, and B represents each pixel position (coordinate).

  The input image while the shutter key 13a is kept depressed has a pixel array with a Bayer structure as shown. For this reason, each pixel of the input image includes only one-channel color information (that is, color data of any one of R, G, and B). On the other hand, if the output image is in the RGB image format, each pixel of the output image has three channels of color data.

  The coordinate system of the output image is set to be the same as the coordinate system of the first image. First, consider the first image addition. Since the coordinate system of the output image is the same as that of the first image, the coordinates (x, y) of the input image corresponding to the coordinate point (u, v) of the output image are (u, v). ) Since pixel data corresponding to these coordinates has a Bayer structure in the input image, only one channel of color data exists for each pixel, and no other pixel data exists, so image data of corresponding colors in the vicinity. From this, interpolation is used.

  Now, consider the addition of the second and subsequent images. The corresponding coordinate (x, y) of the input image corresponding to the coordinate point (u, v) of the output image is obtained by using the motion correction parameter obtained in step S18 as the parameter of the previous equation (5). For this reason, the obtained coordinates (x, y) rarely become an integer value.

  FIG. 9 is a coordinate correspondence diagram between an input image and an output addition image. As shown in this figure, the color pixel data at the coordinates of q = (x, y) is obtained by interpolation from neighboring images.

  Various methods are conceivable for this interpolation method, but this embodiment is characterized in that the accuracy of the motion prediction parameter is estimated and this interpolation method (image synthesis processing) is switched according to the accuracy.

If the nearest coordinates of each color of the coordinates (x, y) of the input image are (Xr. Yr), (Xg, Yg), (Xb, Yb), these are R ₄₅ , G in the example of FIG. _44, the coordinate point of each pixel of the B _34. The pixel value of the coordinate (X, Y) of the input image is P (X, Y), and the RGB component of the coordinate (u, v) of the output image is r (u, v), g (u, v), b, respectively. Assuming (u, v), an example of an interpolation method giving priority to image resolution is given by the following equation (6), and an example of an interpolation method giving priority to image S / N is given by the following equations (7) to (10). However, the following equation (7) is an example of an interpolation method of red (R) and blue (B), which performs an averaging of nine neighboring points centered on the nearest pixel of the same color. Expression (9) is an example of the green (G) interpolation method, and performs the averaging of 13 neighboring points centered on the nearest pixel of the same color.

  FIG. 10 is a diagram illustrating an example of an image S / N priority interpolation method (R and B interpolation method). In this figure, each 5 × 5 cell represents a pixel, and −2, −1, 0, 1 and 2 attached in the horizontal direction are x coordinates, −2, −1, 0, and 2 attached in the vertical direction. 1 and 2 are y coordinates. A pixel at coordinates (0, 0) is the pixel of interest. In the previous equation (7), r (u, v), that is, R pixel interpolation is performed, but this equation (7) is also an B pixel interpolation equation. When the B pixel interpolation formula is used, r (u, v) is read as b (u, v), and Xr and Yr in formula (7) are read as Xb and Yb.

  The combinations of the x and y coordinates (1) to (9) listed at the right end in the figure are the coordinates corresponding to the nine elements (k, l) in the previous equation (8). The elements (1) to (9) are added to the corresponding portions of the 5 × 5 cell in the figure, and the elements are written. As a result, interpolation of the target pixel of r (u, v) or b (u, v) is performed for nine pixels in the vicinity of the same color as the target pixel, that is, pixel values of (1) to (9). It will be understood that it is given as an arithmetic average.

  FIG. 11 is a diagram illustrating an example of the image S / N priority interpolation method (G interpolation method). In this figure, as described above, each 5 × 5 cell represents a pixel, and −2, −1, 0, 1 and 2 attached in the horizontal direction are x coordinates, −2 attached in the vertical direction, -1, 0, 1 and 2 are y coordinates. A pixel at coordinates (0, 0) is the pixel of interest.

  The combinations of the x and y coordinates (1) to (13) listed at the right end in the figure are the coordinates corresponding to the 13 elements of (k, l) in the previous equation (10). The elements (1) to (13) are added to the corresponding portions of the 5 × 5 cell in the figure, and the elements are written. Thereby, the interpolation of the pixel of interest of g (u, v) is given by 13 pixels in the vicinity of the same color as the pixel of interest, that is, the addition average of the pixel values of (1) to (13). It will be understood.

Since it is as mentioned above, according to this embodiment, the following peculiar effects are acquired.
(A) Since the image composition processing (image addition) is switched according to the accuracy prediction of motion detection, specifically, the accuracy of motion detection deteriorates when the input image is considerably dark (composition accuracy is less Since image synthesis processing (image addition) that can obtain a sufficient image S / N with a small number of shots is employed, it is not necessary to take many images unnecessarily. A good image with good quality can be obtained without giving a great stress to the photographer.
(B) On the other hand, when the input image is not so dark, the S / N of the image is originally high, so the accuracy of motion detection does not decrease (compositing accuracy is good), and the number of images required for combining is small. In such a case, the image composition processing (image addition) giving priority to the resolution is selected. In this case, an image with a high resolution can be obtained while the number of shots is small, Subject shake can be corrected.
(C) Due to the above (a) and (b), the number of shots required for image composition is large regardless of whether the brightness level of the input image is large, that is, when the image is considerably dark to not so dark. Therefore, the number of shots can be made almost constant, and a composite image with satisfactory quality can be obtained without stressing the photographer.

  In the above-described embodiment, “image brightness” is used to estimate the accuracy of motion prediction, but the present invention is not limited to this. You may estimate using the other information or signal showing the precision of motion estimation. For example, it is possible to use the output value of the detector for detecting the feature point used when extracting the feature point in step S3 in FIG. This is because when a corner detector is used to detect a feature point, the corner detector shows a larger output value when a suitable feature point is detected. Therefore, if the output value is detected to be larger than a predetermined value, it is determined as “high accuracy”, and if not, it is determined as “low accuracy”.

  In the above-described embodiment, the application to the digital camera 1 is taken as an example. However, the present invention is not limited to this.

It is a block diagram of the digital camera which shows an example of the application system of this embodiment. 2 is a structural diagram of an imaging device 6. FIG. 2 is a diagram illustrating an imaging device 6 and a color filter 19. FIG. It is a principle diagram of a color filter called a Bayer method (or a green checkered method). It is a block diagram of an actual color filter. It is a pixel array diagram of an interlaced CCD sensor. It is a figure which shows the flowchart of the control program containing the characteristic matter in this embodiment. It is a pixel arrangement | sequence figure of an addition object image. It is a coordinate corresponding | compatible figure of an input image and an output addition image. It is a figure which shows an example (R and B interpolation method) of the image S / N priority interpolation method. It is a figure which shows an example (G interpolation method) of the image S / N priority interpolation method.

Explanation of symbols

1 Digital camera (image processing device)
2 CPU (position correction means, image composition means, estimation means, image composition processing selection means, feature point extraction means)
4 Imaging unit (imaging means)

Claims (9)

Imaging means for periodically generating and outputting a plurality of images;
A position correcting unit that sequentially captures a plurality of images output from the imaging unit, predicts a motion between the images, and corrects a positional deviation between the two images;
Image synthesizing means for applying a predetermined image synthesizing process and sequentially synthesizing the images after the positional deviation correction to generate a synthesized image;
Estimating means for estimating the accuracy of the motion prediction;
When the estimation result of the estimation means is high in accuracy, the image synthesis priority processing is selected as the predetermined image synthesis processing used in the image synthesis means, while the estimation result of the estimation means is low in accuracy. In this case, the image processing apparatus further comprises: an image composition processing selection unit that selects an image S / N priority image composition process as the predetermined image composition process used by the image composition unit.   The image processing apparatus according to claim 1, wherein the estimation unit estimates the accuracy of the motion prediction according to brightness of an image output from the imaging unit.   The position correcting unit further includes a feature point extracting unit that extracts a feature point of an image output from the imaging unit for motion prediction, and the estimating unit is based on an output of the feature point extracting unit. The image processing apparatus according to claim 1, wherein accuracy of the motion prediction is estimated. An imaging step for periodically generating and outputting a plurality of images;
A position correction step for sequentially capturing a plurality of images output from the imaging step, predicting a motion between the images, and correcting a positional deviation between both images;
An image synthesis step of applying a predetermined image synthesis process and sequentially synthesizing the images after the positional deviation correction to generate and output a synthesized image;
An estimation step for estimating the accuracy of the motion prediction;
When the estimation result of the estimation step is high in accuracy, the image synthesis priority processing is selected as the predetermined image synthesis processing used in the image synthesis step, while the estimation result of the estimation step is low in accuracy. In this case, the image processing method includes: an image composition process selection step for selecting an image composition process with priority for image S / N as the predetermined image composition process used in the image composition step.   5. The image processing method according to claim 4, wherein the estimating step estimates the accuracy of the motion prediction according to the brightness of the image output from the imaging step.   The position correction step further includes a feature point extraction step for extracting a feature point of the image output from the imaging step for motion prediction, and the estimation step is based on an output of the feature point extraction step. The image processing method according to claim 4, wherein the accuracy of the motion prediction is estimated. On the computer,
Imaging means for periodically generating and outputting a plurality of images;
A position correcting unit that sequentially captures a plurality of images output from the imaging unit, predicts a motion between the images, and corrects a positional deviation between the two images;
Image synthesizing means for applying a predetermined image synthesizing process and sequentially synthesizing the images after the positional deviation correction to generate a synthesized image;
Estimating means for estimating the accuracy of the motion prediction;
When the estimation result of the estimation means is high in accuracy, the image synthesis priority processing is selected as the predetermined image synthesis processing used in the image synthesis means, while the estimation result of the estimation means is low in accuracy. In this case, there is provided a program that realizes image composition processing selection means for selecting image S / N priority image composition processing as predetermined image composition processing used by the image composition means.   The program according to claim 7, wherein the estimation unit estimates the accuracy of the motion prediction according to brightness of an image output from the imaging unit.   The position correcting unit further includes a feature point extracting unit that extracts a feature point of an image output from the imaging unit for motion prediction, and the estimating unit is based on an output of the feature point extracting unit. The program according to claim 7, wherein accuracy of the motion prediction is estimated.

Images