JP2007306107A - Imaging apparatus, imaging method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily acquire an image where a moving body is deleted and exposure is appropriate. <P>SOLUTION: An imaging section 41 converts incident light photoelectrically to capture the image. While a user operates a release switch 31, an average image generation section 58 generates an average image where the pixel values of a plurality of images captured by the imaging section 41 at the exposure time of appropriate exposure are averaged. When the operation of the release switch 31 is stopped, an I/O control section 62 records the average image on a memory card 65 via a storage section 63 or a drive 64. The present invention can be applied to, for example a digital camera. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program, and more particularly, for example, an imaging apparatus and an imaging method that allow a user to easily acquire an image with proper exposure from which a moving object (moving object) is erased. And related to the program.

  For example, Patent Document 1 proposes image processing for estimating an output image based on a plurality of input images captured with less than appropriate exposure (exposure).

  In the image processing described in Patent Document 1, in the process of estimating an output image from a plurality of input images, the positional relationship between the plurality of input images is corrected, and each of the corrected input images is added (hereinafter referred to as correction addition). Process).

According to the correction addition process described in Patent Document 1, it is possible to obtain an output image in which a moving body portion on which a moving body is projected is deleted and only a stationary portion that is not a moving body portion is captured in an input image.
JP 2005-038396 A

  In the correction addition process described in Patent Document 1, the exposure amount of the finally obtained output image is determined in advance, a plurality of input images less than the appropriate exposure are captured, and the input images are added to obtain an appropriate value. Although an output image of exposure can be obtained, in this case, since the number of times of adding the input image is determined, the degree of erasure of the moving object cannot be adjusted.

  In addition, in the correction addition process described in Patent Document 1, by performing a process of adding a plurality of input images while confirming the degree of erasure of moving objects without determining the exposure amount of the finally obtained output image, Although it is possible to obtain an output image from which the moving object has been erased, in this case, the exposure amount of the finally obtained output image cannot be adjusted.

  The present invention has been made in view of such a situation, and enables a user to easily acquire an image of proper exposure from which a moving object has been erased.

  An imaging apparatus according to a first aspect of the present invention is an imaging apparatus that captures an image, an imaging unit that captures an image by photoelectrically converting incident light, an operation unit that is operated by a user, and the operation unit. Is operated, the average image generating means for generating an average image obtained by averaging the pixel values of a plurality of images captured by the imaging means with an exposure time of appropriate exposure, and the operation of the operating means are stopped. And a recording control means for recording the average image on a recording medium.

  The imaging apparatus according to the first aspect includes a display unit that displays an image, and a display control that causes the display unit to display a new average image each time a new average image is generated in the average image generation unit. Means.

  In the imaging device of the first aspect, while the operation unit is being operated, the imaging unit captures the position of the subject shown in each of a plurality of images captured by the imaging unit. Correction means for correcting an image can be further provided, and the average image generation means can average a plurality of images corrected by the correction means.

  The average image generating means in the imaging device of the first aspect includes an addition image obtained by adding n−1 images when the n-th image is captured by the imaging means, and the n-th image. And adding means for generating a new added image obtained by adding n images, and 1 / n multiplication for multiplying the pixel value of the new added image obtained by the adding means by 1 / n. Means.

  The average image generating means in the imaging device of the first aspect is 1 / n for increasing the pixel value of the n-th image by 1 / n when the n-th image is picked up by the imaging means. Multiplying means, (n-1) / n multiplying the pixel value of the average image generated from n-1 images by (n-1) / n, and the pixel value being 1 / n times And adding means for adding pixel values of the n-th image and the average image having a pixel value multiplied by (n−1) / n.

  An imaging method or program according to the first aspect of the present invention is an imaging method of an imaging apparatus including an imaging unit that images an image by photoelectrically converting incident light, or an image by photoelectrically converting incident light. A program that causes a computer to execute an imaging process of an imaging apparatus including an imaging unit that performs imaging, and a plurality of images captured by the imaging unit with an exposure time of appropriate exposure while an operation unit operated by a user is operated. Generating an average image obtained by averaging pixel values of the image, and recording the average image on a recording medium when the operation of the operation unit is stopped.

  In the first aspect of the present invention, pixel values of a plurality of images captured by the imaging unit with an appropriate exposure time are averaged while an operating unit operated by a user is operated. When an average image is generated and the operation of the operation means is stopped, the average image is recorded on a recording medium.

  An imaging apparatus according to a second aspect of the present invention is an imaging apparatus that captures an image. The imaging unit captures an image by photoelectrically converting incident light, and the imaging unit captures an image with an exposure time of appropriate exposure. Each time a new average image is generated in the average image generation means for generating an average image obtained by averaging pixel values of a plurality of images, a display means for displaying an image, and the average image generation means, a new image is generated. Display control means for displaying an average image on the display means.

  The image pickup apparatus according to the second aspect can further include recording control means for recording the average image on a recording medium.

  In the imaging device of the second aspect, while the operation unit is being operated, the imaging unit captures the position of the subject shown in each of the plurality of images captured by the imaging unit. Correction means for correcting an image can be further provided, and the average image generation means can average a plurality of images corrected by the correction means.

  The average image generating means in the imaging device of the second aspect includes an addition image obtained by adding n−1 images when the n-th image is captured by the imaging means, and the n-th image. And adding means for generating a new added image obtained by adding n images, and 1 / n multiplication for multiplying the pixel value of the new added image obtained by the adding means by 1 / n. Means.

  The average image generating means in the imaging device of the second aspect is 1 / n for increasing the pixel value of the n-th image by 1 / n times when the n-th image is picked up by the imaging means. Multiplying means, (n-1) / n multiplying the pixel value of the average image generated from n-1 images by (n-1) / n, and the pixel value being 1 / n times And adding means for adding pixel values of the n-th image and the average image having a pixel value multiplied by (n−1) / n.

  The imaging method or the program according to the second aspect of the present invention provides an imaging method of an imaging apparatus including an imaging unit that images an image by photoelectrically converting incident light, or an image by photoelectrically converting incident light. A program for causing a computer to execute an imaging process of an imaging apparatus including an imaging unit that captures an image, and generating an average image obtained by averaging pixel values of a plurality of images captured by the imaging unit with an exposure time of appropriate exposure, Each time a new average image obtained by averaging images captured by the imaging unit is generated, the new average image is displayed on the display unit that displays the image.

  In the second aspect of the present invention, an average image is generated by averaging pixel values of a plurality of images captured by the imaging unit with an exposure time of appropriate exposure, and the image captured by the imaging unit is Each time a new averaged average image is generated, the new average image is displayed on display means for displaying the image.

  According to the present invention, for example, a user can easily acquire an image of proper exposure from which a moving object is erased.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  The image pickup apparatus according to the first aspect of the present invention is an image pickup means (for example, FIG. 1) that picks up an image by photoelectrically converting incident light in an image pickup apparatus (for example, the digital camera 11 in FIG. 1). Imaging unit 41), operating means operated by a user (for example, the operating unit 21 in FIG. 1), and a plurality of images captured by the imaging unit with appropriate exposure time while the operating unit is being operated. An average image generating unit (for example, the average image generating unit 58 in FIG. 1) that generates an average image obtained by averaging pixel values of a single image, and when the operation of the operating unit is stopped, the average image is recorded on the recording medium. Recording control means (for example, the input / output control unit 62 in FIG. 1).

  The imaging device according to the first aspect includes a display unit (for example, the display unit 61 in FIG. 1) that displays an image and a new average image each time a new average image is generated by the average image generation unit. Can be further provided with display control means (for example, the display control unit 60 of FIG. 1).

  In the imaging device of the first aspect, while the operation unit is being operated, the imaging unit captures the position of the subject shown in each of a plurality of images captured by the imaging unit. A correction unit (for example, the correction unit 57 in FIG. 1) for correcting an image can be further provided, and the average image generation unit can average a plurality of images corrected by the correction unit.

  The average image generating means in the imaging device of the first aspect includes an addition image obtained by adding n−1 images when the n-th image is captured by the imaging means, and the n-th image. And adding means for generating a new added image obtained by adding n images (for example, the adding unit 81 in FIG. 2) and the pixel value of the new added image obtained by the adding means. 1 / n multiplication means (for example, 1 / n multiplication unit 83 in FIG. 2) for 1 / n multiplication can be provided.

  The average image generating means in the imaging device of the first aspect is 1 / n for increasing the pixel value of the n-th image by 1 / n when the n-th image is picked up by the imaging means. Multiplying means (for example, 1 / n multiplier 331 in FIG. 9) and the pixel value of the average image generated from n−1 images are multiplied by (n−1) / n times (n−1) / n multiplication means (for example, (n-1) / n multiplication unit 334 in FIG. 9), the n-th image with a pixel value multiplied by 1 / n, and a pixel value of (n-1) / n An addition unit (for example, an addition unit 332 in FIG. 9) that adds pixel values with the average image that has been doubled may be included.

  An imaging method or program according to the first aspect of the present invention is an imaging method of an imaging apparatus including an imaging unit that images an image by photoelectrically converting incident light, or an image by photoelectrically converting incident light. In a program for causing a computer to execute an imaging process of an imaging apparatus including an imaging unit for imaging, a plurality of images captured by the imaging unit with an appropriate exposure time while an operation unit operated by a user is operated An average image is generated by averaging the pixel values (for example, step S37 and step S38 in FIG. 8, or step S87 to step S89 in FIG. 10), and when the operation of the operation means is stopped, the average image is Recording on a recording medium (for example, step S41 in FIG. 8 or step S92 in FIG. 10) is included.

  The image pickup apparatus according to the second aspect of the present invention is an image pickup apparatus (for example, FIG. 1) that picks up an image by photoelectrically converting incident light in an image pickup apparatus (for example, the digital camera 11 in FIG. 1). Imaging unit 41) and an average image generating unit that generates an average image obtained by averaging pixel values of a plurality of images captured by the imaging unit with an exposure time of appropriate exposure (for example, the average image generating unit 58 in FIG. 1). ), Display means for displaying an image (for example, the display unit 61 in FIG. 1), and the average image generation means, a new average image is displayed on the display means each time a new average image is generated. Display control means (for example, the display control unit 60 in FIG. 1).

  The image pickup apparatus according to the second aspect can further include recording control means (for example, the input / output control unit 62 in FIG. 1) for recording the average image on a recording medium.

  In the imaging device of the second aspect, while the operation unit is being operated, the imaging unit captures the position of the subject shown in each of the plurality of images captured by the imaging unit. A correction unit (for example, the correction unit 57 in FIG. 1) for correcting an image can be further provided, and the average image generation unit can average a plurality of images corrected by the correction unit.

  The average image generating means in the imaging device of the second aspect includes an addition image obtained by adding n−1 images when the n-th image is captured by the imaging means, and the n-th image. And adding means for generating a new added image obtained by adding n images (for example, the adding unit 81 in FIG. 2) and the pixel value of the new added image obtained by the adding means. 1 / n multiplication means (for example, 1 / n multiplication unit 83 in FIG. 2) for 1 / n multiplication can be provided.

  The average image generating means in the imaging device of the second aspect is 1 / n for increasing the pixel value of the n-th image by 1 / n times when the n-th image is picked up by the imaging means. Multiplying means (for example, 1 / n multiplier 331 in FIG. 9) and the pixel value of the average image generated from n−1 images are multiplied by (n−1) / n times (n−1) / n multiplication means (for example, (n-1) / n multiplication unit 334 in FIG. 9), the n-th image with a pixel value multiplied by 1 / n, and a pixel value of (n-1) / n An addition unit (for example, an addition unit 332 in FIG. 9) that adds pixel values with the average image that has been doubled may be included.

  The imaging method or the program according to the second aspect of the present invention provides an imaging method of an imaging apparatus including an imaging unit that images an image by photoelectrically converting incident light, or an image by photoelectrically converting incident light. In a program for causing a computer to execute an imaging process of an imaging apparatus including an imaging unit that captures an image, an average image is generated by averaging pixel values of a plurality of images captured by the imaging unit with an exposure time of appropriate exposure (for example, Step S37 and Step S38 in FIG. 8 or Steps S87 to S89 in FIG. 10, each time a new average image is generated by averaging the images captured by the imaging means, a new average image is generated. Is displayed on the display means for displaying (for example, step S39 in FIG. 8 or step S90 in FIG. 10).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration example of an embodiment of a digital camera (digital still camera) 11 to which the present invention is applied.

  1 includes an operation unit 21, an imaging unit 41, an SDRAM (Synchronous Dynamic Random Access Memory) 54, a motion vector detection unit 55, a SAD (Sum of Absolute Differences) table 56, a correction unit 57, and an average image generation unit. 58, a display control unit 60, a display unit 61, an input / output control unit 62, a storage unit 63, a drive 64, and a memory card 65.

  The operation unit 21 includes, for example, a release switch 31 and a touch panel superimposed on a display unit 61 described later, and is operated by a user. The operation unit 21 supplies an operation signal corresponding to a user operation to a necessary block of the digital camera 11.

  The imaging unit 41 captures the subject by receiving and photoelectrically converting the light incident thereon, and supplies the captured image obtained as a result to the SDRAM 54 for storage (temporarily).

  Here, the imaging unit 41 includes an imaging lens 51, an imaging element 52, and a camera signal processing unit 53, and the imaging lens 51 forms an image of a subject on the light receiving surface of the imaging element 52. The image sensor 52 is composed of, for example, a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) sensor, and the like. The image (light) of a subject imaged on the light receiving surface is converted into an analog signal by photoelectric conversion. The image signal is supplied to the camera signal processing unit 53.

  The camera signal processing unit 53 performs, for example, gamma correction processing or white balance processing on the analog image signal supplied from the imaging device 52. Thereafter, the camera signal processing unit 53 performs A / D (Analog / Digital) conversion on the analog image signal, and supplies the digital image signal (captured image) obtained as a result to the SDRAM 54 for storage.

  The SDRAM 54 stores the captured image supplied from the camera signal processing unit 53 (imaging unit 41).

  Here, as a shooting mode of the digital camera 11, for example, in the imaging unit 41, normal shooting in which one captured image is captured with an exposure time that is appropriate exposure in response to a single pressing operation of the release switch 31. A moving object erasure shooting mode for obtaining an image in which a moving object is erased by averaging a plurality of captured images captured with an exposure time corresponding to a proper exposure according to a pressing operation of the release switch 31 There is. Hereinafter, a case where the shooting mode is the shooting mode for moving object elimination will be described. Note that when the shooting mode is the normal shooting mode, the motion vector detection unit 55 to the average image generation unit 58 described below do not function.

  The motion vector detection unit 55 reads the captured images captured by the imaging unit 41 from the SDRAM 54 in the order of capturing. The motion vector detection unit 55 averages the first captured image read from the SDRAM 54 (the captured image first captured by the imaging unit 41 after the release switch 31 is pressed) via the correction unit 57. While supplying to the image generation part 58, it supplies to the SAD table 56, and memorize | stores it as a reference image mentioned later.

  Here, the reference image refers to the position of the second and subsequent captured images (second captured image captured by the image capturing unit 41 after the release switch 31 is pressed) by the correction unit 57 described later. An image that serves as a reference for correction.

  Further, the n-th captured image refers to an n-th captured image captured among a plurality of captured images captured in the moving object erasure shooting mode. In other words, in the digital camera 11 in the moving object erasing shooting mode, N captured images captured from when the pressing operation of the release switch 31 is started to when the pressing operation is stopped (released) are: It is to be targeted for averaging. The n-th captured image refers to n (n = 1, 2,..., N−1) among N captured images captured while the release switch 31 is being pressed. N) This is a captured image that has been imaged first.

  For the second and subsequent captured images read from the SDRAM 54, the motion vector detection unit 55 detects a motion vector representing the motion of the captured image with respect to the reference image stored in the SAD table 56, and a correction unit 57 together with the captured image. To supply.

  The SAD table 56 stores the first captured image supplied from the motion vector detection unit 55 as a reference image.

  The correction unit 57 similarly corrects the captured image supplied from the motion vector detection unit 55 based on the motion vector of the captured image supplied from the motion vector detection unit 55, and generates an average image of the corrected captured image. Supplied to the unit 58.

  The average image generation unit 58 averages the pixel values of a plurality of corrected captured images supplied from the correction unit 57 to generate an average image. The average image generating unit 58 will be described in detail in FIG. 2 or FIG. 9 described later.

  The display control unit 60 reads the new average image from the average image generation unit 58 every time a new average image (including the first captured image) is generated by the average image generation unit 58, and displays the display unit 61 is displayed.

  The display unit 61 displays an average image supplied from the display control unit 60 according to the control of the display control unit 60. As the display unit 61, for example, an LCD (Liquid Crystal Display) or the like can be employed.

  An average image generation unit 58, a storage unit 63, and a drive 64 are connected to the input / output control unit 62. The input / output control unit 62 controls the exchange of images with the average image generation unit 58, the storage unit 63, or the drive 64.

  The storage unit 63 stores the image supplied from the input / output control unit 62.

  The drive 64 supplies the image supplied from the input / output control unit 62 to the memory card 65 and stores it. Further, the drive 64 reads an image from the memory card 65 and supplies it to the input / output control unit 62.

  The memory card 65 is a removable memory card that can be attached to and removed from the drive 64 of the digital camera 11, and stores an image supplied from the input / output control unit 62.

  FIG. 2 is a block diagram illustrating a detailed configuration example of the average image generation unit 58 of FIG.

  The average image generator 58 includes an adder 81, an SDRAM 82, a 1 / n multiplier 83, and an SDRAM 84.

  The addition unit 81 is supplied with the corrected n-th captured image from the correction unit 57. The adder 81 reads an added image obtained by adding n−1 (n = 2, 3,..., N−1, N) captured images from the SDRAM 82, and supplies the added image and the correction unit 57. The pixel values of the n-th captured image thus added are added, and the added image obtained as a result is supplied to the SDRAM 82 and stored.

  When the captured image supplied from the correction unit 57 is the first captured image, the adding unit 81 supplies the one captured image as it is to the SDRAM 82 as an added image and stores it.

  The SDRAM 82 stores the added image (including the first captured image) supplied from the adding unit 81 and including the n captured images (n = 1, 2,..., N−1, N ).

  The 1 / n multiplication unit 83 reads an added image obtained by adding n captured images from the SDRAM 82, and by multiplying the pixel value of the added image by 1 / n, an average image obtained by averaging the n captured images is obtained. Generate. This average image is supplied from the 1 / n multiplier 83 to the SDRAM 84 and stored in an overwritten form.

  The average image stored in the SDRAM 84 (average image generation unit 58) is read by the display control unit 60 and displayed on the display unit 61. The average image stored in the SDRAM 84 is read by the input / output control unit 62, supplied to the storage unit 63 or the memory card 65, and stored.

  In the average image generation unit 58 as described above, the pixel value of the added image obtained by adding n captured images is multiplied by 1 / n, so that an average image with less gradation loss due to a so-called rounding error can be generated.

  Next, processing of the motion vector detection unit 55 in FIG. 1 will be described with reference to FIGS. 3 and 4.

  FIG. 3 is a diagram illustrating a state in which the motion vector detection unit 55 detects a motion vector representing the motion of the second and subsequent captured images in the moving object erasure imaging mode.

  The upper side of FIG. 3 is a diagram showing a reference image 151 on which a stationary subject 161 is projected and a captured image 152 on which the same subject 162 as the subject 161 is projected (second and subsequent images). The lower side of FIG. 3 is a diagram illustrating a reference image 151 that is divided into m × n blocks in length × width. Also, arrows in the m × n blocks on the lower side of FIG. 3 represent motion vectors detected based on the respective blocks.

  As described above, the reference image 151 represents the first captured image, and the captured image 152 represents the second and subsequent captured images. The reason why the position of the subject 161 with respect to the reference image 151 is different from the position of the subject 162 with respect to the captured image 152 is due to, for example, camera shake.

  The motion vector detection unit 55 uses the reference image 151 and the captured image 152 to detect a motion vector that represents the motion of the captured image 152 with respect to the reference image 151.

  That is, the motion vector detection unit 55 divides the reference image 151 into m × n blocks in the vertical and horizontal directions, as shown in the lower side of FIG. 3, and the captured image 152 most similar to each block. Block matching for detecting a motion vector is performed by obtaining the upper region.

  FIG. 4 is a diagram for explaining a method of detecting a motion vector by such block matching.

  In FIG. 4, the reference image 151 and the captured image 152 are superimposed. In FIG. 4, an xy coordinate system is defined in which the lower left point of the reference image 151 (captured image 152) is the origin O, the right direction is the x direction, and the upper direction is the y direction.

As shown in FIG. 4, the motion vector detection unit 55 uses the divided blocks 151 a to 151 f on the reference image 151 as a template, and is an area on the captured image 152 that is most similar to each of the blocks 151 a to 151 f. 152a to 152f are obtained. Further, for example, the motion vector detection unit 55 uses the positions (C _x , C _y ) of the centers (centers of gravity) of the blocks 151a to 151f on the reference image 151 as the start points, and the positions (C _x ′) of the centers of the regions 152a to 152f. , C _y ′) is detected as a motion vector.

  In this way, the motion vector detection unit 55 calculates m × n motion vectors detected for m × n blocks on the reference image 151 shown in the lower side of FIG. ) The image is supplied to the correction unit 57 together with the captured image 152.

  Next, the processing of the correction unit 57 in FIG. 1 will be described with reference to FIG.

  The correcting unit 57 corrects the captured image 152 using the reference image 151 as a reference. In other words, the correction unit 57 aligns the position of the subject 161 on the reference image 151 with the position of the subject 162 on the captured image 152 that is the same as the subject 161. to correct.

  In the affine transformation, the positional relationship between the position (x, y) of the reference image 151 (pixels thereof) and the position (x ′, y ′) of the captured image 152 is expressed by the following equation (1).

  Here, according to the affine transformation of the equation (1), the position (x, y) is rotated by an angle θ around the origin O and (x, y) = (s, It is converted (corrected) into a position (x ′, y ′) translated by t).

  In the following, the parameters s, t, and θ that define the affine transformation of Expression (1) are referred to as affine parameters (s, t, and θ).

  In the affine transformation of Expression (1), the movement of the digital camera 11 in the direction from the digital camera 11 toward the subject is not taken into consideration, but the affine transformation can be performed in consideration of such a movement. . In this case, instead of the 2 × 2 matrix on the right side of Equation (1), a matrix obtained by multiplying the matrix by a parameter relating to enlargement / reduction is used.

  The correction unit 57 obtains the affine parameters (s, t, θ) by the method of least squares using the m × n motion vectors shown on the lower side of FIG. 3 and the above-described equation (1).

Specifically, the correction unit 57 performs affine transformation on the center position (C _x , C _y ) of m × n blocks on the reference image 151 by the above-described equation (1). Then, the correction unit 57 uses the position (C _x , C _y ) of the center of m × n blocks on the reference image 151 as the start point, and the position after the affine transformation of the position (C _x , C _y ) as the end point _Is calculated as a converted motion vector _VGM . The converted motion vector V _GM is expressed by the following equation (2).

Here, the converted motion vector V _GM in Expression (2) is represented by an expression using affine parameters (s, t, θ) as variables.

The correction unit 57 calculates the square error between the converted motion vector V _GM of the block on the reference image 151 and the motion vector of the block detected by the above-described block matching (hereinafter, also referred to as a matching motion vector V _{BM as} appropriate). The affine parameters (s, t, θ) that minimize the sum E of are obtained. The sum E of square errors is expressed by the following equation (3).

  Here, Σ in Equation (3) represents the sum of m × n blocks on the reference image 151 shown on the lower side of FIG. The affine parameter (s, t, θ) that minimizes the sum E of square errors in Equation (3) is obtained by partial differential of the sum E of square errors with the affine parameters (s, t, θ). It can be obtained by solving an equation with 0 as an equation.

  Next, the correction unit 57 corrects the position (x ′, y ′) of the captured image 152 to the position (x, y) of the reference image 151 (position) using the obtained affine parameters (s, t, θ). The inverse affine transformation is performed. As a result, the captured image 152 is corrected so that the positions of the subject 161 on the reference image 151 and the subject 162 on the captured image 152 are aligned as shown in FIG.

  Here, since the imaging unit 41 captures an image with a larger number of pixels than the effective number of pixels employed as the captured image 152, in reality, an image larger than the captured image 152 (an image with a large number of pixels) 182. Are being imaged.

  After correcting the image 182 including the captured image 152, the correction unit 57 extracts an image in a range that completely matches the range of the reference image 151 in the image 182 as the corrected captured image 152, and calculates the average. This is supplied to the image generation unit 58.

  Note that how large the image 182 is to be an image (an image having a larger number of pixels) than the captured image 152 is determined based on, for example, a statistic of the amount of camera shake when a person performs so-called handheld imaging. The

  Further, as described above, the correction unit 57 obtains the affine parameters (s, t, θ) and corrects the captured image using the affine parameters (s, t, θ). It is for aligning a subject whose position is shifted due to camera shake, and not for aligning a moving moving object. Therefore, the affine parameters (s, t, θ) that are not affected by the motion of the moving object projected on the captured image are used as the affine parameters (s, t, θ) used in the correction unit 57 for correcting the captured image. Need to ask. Therefore, in the correction unit 57, the sum E of the square errors in the above-described equation (3) is obtained only for the block on which no moving object is projected.

Whether or not the moving object is a non-projected block can be determined based on, for example, a histogram of the matching motion vector V _BM obtained for the block on the reference image 151. That is, in the histogram of the matching motion vector V _BM determined for blocks of the reference image 151, the block of the frequency with the highest matching motion vector V _BM distance between close matching motion vector V _BM is the moving object is projected The block of the matching motion vector V _BM that is far from the matching motion vector V _BM having the highest frequency can be determined to be a block on which a moving object is projected.

  Next, the average image generated by the average image generation unit 58 of FIG. 2 will be described with reference to FIGS. 6 and 7.

  FIG. 6 is a diagram illustrating a state in which a new average image is generated in the average image generation unit 58 of FIG.

The upper side of FIG. 6 represents the exposure amounts 211 _{1 to} 211 _N of the _{first to Nth} captured images captured with the exposure time of the appropriate exposure, and the left side of FIG. 6 shows the added image generated by the adding unit 81. The exposure amounts 232 _{1 to} 232 _N are shown. The right side of FIG. 6 represents exposure amounts 251 _{1 to} 251 _N of the average image generated by the 1 / n multiplication unit 83. Note that the added image generated by the adder 81 is stored in the SDRAM 82, and the average image generated by the 1 / n multiplier 83 is stored in the SDRAM 84.

The adding unit 81 uses the first captured image supplied from the correcting unit 57 as it is as an added image of the exposure amount 232 ₁ (211 ₁ ). Moreover, 1 / n multiplier section 83, an addition image exposure 232 _1, the average image of exposure 251 ₁ (232 _1). Here, the exposure amounts 211 ₁ , 232 ₁ and 251 ₁ all represent the same exposure amount, but are given different numbers for convenience of explanation.

The addition unit 81 adds the addition image of the exposure amount 232 ₁ and the second captured image supplied from the correction unit 57, and adds the exposure amount 232 ₂ (the exposure amount 232 ₁ and the exposure amount 211 ₂ ). (Exposure amount) addition image is generated. Next, the 1 / n multiplication unit 83 adds the _two captured images, halves the pixel value of the addition image with the exposure amount 232 ₂ , and generates an average image with the exposure amount 251 ₂ .

Thereafter, the addition unit 81 adds the addition image with the exposure amount 232 ₂ and the third captured image supplied from the correction unit 57 to generate an addition image with the exposure amount 232 ₃ . Then, the 1 / n multiplication unit 83 adds the three captured images, and multiplies the pixel value of the addition image with the exposure amount 232 ₃ by 1/3 to generate an average image with the exposure amount 251 ₃ .

Hereinafter, similarly, the addition unit 81 adds the addition image of the exposure amount 232 _{n−1 and} the _n-th captured image supplied from the correction unit 57 to generate an addition image of the exposure amount 232 _n . Then, the 1 / n multiplication unit 83 adds 1 / n times the pixel value of the added image of the exposure amount 232 _{n obtained} by adding the _n captured images, and generates an average image of the exposure amount 251 _n (n = 2). , 3,..., N-1, N).

As a result, the exposure amounts 251 _{1 to} 251 _N of the average image generated by the average image generation unit 58 have the exposure amount of appropriate exposure.

  FIG. 7 shows a display example of the average image of the display unit 61 of FIG.

In FIG. 7, an average image 261 _n is an average image obtained by averaging n captured images.

  As described above, the average image generation unit 58 generates an average image obtained by averaging a plurality of captured images.

  In a plurality of captured images, a stationary object (stationary object) does not change even when averaged, but the stationary object component in the pixel value does not change, so it appears in the average image as it is, but the moving object ( With respect to (moving object), as the number of captured images used for averaging increases, the moving object component occupying the pixel value decreases as a result of averaging, so that it does not appear in the average image.

For example, when a subject 271 that is a stationary body and subjects 272 and 273 that are moving bodies exist in the imaging range of the digital camera 11, when the first captured image is captured, the display unit 61 displays As shown in the figure, first, together with the subject 271, an average image 261 _{1 that} is the _first captured image on which the subjects 272 and 273 are projected is displayed, and when the second captured image is captured, the first image An average image 2612 obtained by averaging the _second captured image is displayed.

In the average image 261 ₂ , the components of the subjects 272 and 273 occupying the pixel value are reduced by averaging, and the subjects 272 and 273 are so thin.

When the third captured image is captured, an average image 261 ₃ obtained by averaging the first to _third captured images is displayed.

  Hereinafter, similarly, when a new captured image is captured, an average image is newly generated by the average image generation unit 58, and the average image in which the subjects 272 and 273 are thinner is displayed on the display unit 61. The

As described above, a new average image is generated while the release switch 31 is being pressed. Therefore, the user presses the release switch 31 when the desired average image 261 _N is displayed on the display unit 61, that is, for example, when the subjects 272 and 273 are thin enough to become invisible. By stopping, the desired average image 261 _N can be stored (captured) in the storage unit 63 or the memory card 65.

  Next, with reference to the flowchart of FIG. 8, processing in the shooting mode for moving object elimination of the digital camera 11 when the average image generation unit 58 of FIG. 1 has the configuration shown in FIG. 2 will be described.

  When the user starts pressing the release switch 31, in step S31, the imaging unit 41 captures an image of the subject with the exposure time of proper exposure, and supplies the captured image obtained as a result to the SDRAM 54 for storage. Proceed to S32.

  In step S <b> 32, the motion vector detection unit 55 reads out the captured images captured by the imaging unit 41 from the SDRAM 54 in the order of capturing. In step S32, the motion vector detection unit 55 determines whether or not the captured image read from the SDRAM 54 is the first captured image. If it is determined that the captured image is the first captured image, the process proceeds to step S33.

  In step S33, the motion vector detection unit 55 supplies the first captured image as an addition image to the SDRAM 82 via the correction unit 57 and the addition unit 81, and stores them in the SAD table 56 as a reference image. Supply and store, and proceed to step S38.

  On the other hand, if it is determined in step S32 that the motion vector detection unit 55 is not the first captured image (the second and subsequent captured images), the process proceeds to step S34, and the second and subsequent images read from the SDRAM 54 are processed. For the captured image, a motion vector representing the motion of the captured image with respect to the reference image stored in the SAD table 56 is detected, supplied together with the captured image to the correction unit 57, and the process proceeds to step S35.

  In step S35, the correction unit 57 obtains the affine parameters (s, t, θ) by the least square method using the motion vector supplied from the motion vector detection unit 55 and the above-described equation (1). Proceeding to S36, using the obtained affine parameters (s, t, θ), the captured image supplied from the motion vector detecting unit 55 is corrected, and the corrected captured image is added to the adding unit 81 (average image generating unit 58). To proceed to step S37.

  In step S37, the addition unit 81 reads the addition image from the SDRAM 82, adds the addition image and the pixel value of the n-th captured image after correction supplied from the correction unit 57, and an image obtained thereby. Are supplied to and stored in the SDRAM 82 as a new added image, and the process proceeds to step S38.

  In step S38, the 1 / n multiplication unit 83 reads an added image obtained by adding n captured images from the SDRAM 82, and multiplies the pixel value of the added image by 1 / n. In step S38, the 1 / n multiplication unit 83 supplies the 1 / n-fold added image to the SDRAM 84 as a new average image, stores it in an overwritten form, and proceeds to step S39.

  In step S39, the display control unit 60 reads the average image stored in the SDRAM 84 in step S38 performed immediately before, supplies the average image to the display unit 61 for display, and proceeds to step S40.

  In step S40, the average image generator 58 determines whether or not the pressing operation of the release switch 31 is continued. If it is determined in step S40 that the pressing operation of the release switch 31 is continued, the process returns to step S31, and the same processing is repeated thereafter. Thereby, in step S38, every time the average image generation unit 58 generates a new average image, the new average image is displayed on the display unit 61.

  On the other hand, when it is determined in step S40 that the pressing operation of the release switch 31 has been stopped, that is, the user who has continued pressing the release switch 31 looks at the average image displayed on the display unit 61. Assuming that a desired image (average image) from which the moving object has been erased is obtained, if the pressing operation of the release switch 31 is stopped, the process proceeds to step S41, and the input / output control unit 62 reads the average image (release / release) from the SDRAM 84. When the pressing operation of the switch 31 is stopped, the average image displayed on the display unit 61 is read and supplied to the storage unit 63 or the memory card 65 (via the drive 64) to be stored. Thereafter, the process ends.

  Next, another configuration example of the average image generation unit 58 in FIG. 1 will be described with reference to FIG.

  FIG. 9 is a block diagram showing another configuration example of the average image generation unit 58 of FIG.

  The average image generation unit 58 includes a 1 / n multiplication unit 331, an addition unit 332, an SDRAM 333, and an (n−1) / n multiplication unit 334.

  The 1 / n multiplication unit 331 is supplied with a corrected n-th captured image from the correction unit 57. The 1 / n multiplication unit 331 multiplies the pixel value of the n-th captured image supplied from the correction unit 57 by 1 / n, and supplies the 1 / n image obtained as a result to the addition unit 332.

  The adder 332 includes a 1 / n (n = 2, 3,..., N−1, N) image supplied from the 1 / n multiplier 331 and an (n−1) / n multiplier 334 described later. The pixel values of the (n−1) / n image supplied from the above are added, and the image obtained as a result is supplied to the SDRAM 333 as a new average image and stored in an overwritten form.

  In addition, when the 1 / n image supplied from the 1 / n multiplication unit 331 is a 1/1 image (first captured image), the addition unit 332 uses the 1/1 image as it is as an average image. Is supplied to the SDRAM 333 and stored therein.

  The SDRAM 333 stores the average image supplied from the adding unit 332.

  The (n−1) / n multiplication unit 334 reads the average image from the SDRAM 333 and adds the (n−1) / n image obtained by multiplying the pixel value of the average image by (n−1) / n. To the unit 332.

  The average image stored in the SDRAM 333 (average image generation unit 58) is read by the display control unit 60 and displayed on the display unit 61. The average image stored in the SDRAM 333 is read by the input / output control unit 62, supplied to the storage unit 63 or the memory card 65, and stored.

  By adopting the configuration as described above, the average image generation unit 58 in FIG. 9 can employ an SDRAM having a small storage capacity compared to the case where the configuration in FIG. 2 is employed.

  Further, in the average image generation unit 58 of FIG. 9, the 1 / n image obtained by multiplying the pixel value of the n-th captured image supplied from the correction unit 57 by 1 / n and the pixel value of the average image read from the SDRAM 333 are obtained. The pixel value of (n-1) / n times the (n-1) / n image is added to generate a new average image. For example, the pixel value of the n-th captured image, A new average image may be generated by adding the pixel value obtained by multiplying the pixel value of the average image read from the SDRAM 333 by (n−1) and multiplying the pixel value after the addition by 1 / n. . This can reduce gradation loss due to so-called rounding errors.

  Next, with reference to the flowchart of FIG. 10, processing in the shooting mode for moving object elimination of the digital camera 11 when the average image generation unit 58 of FIG. 1 has the configuration shown in FIG. 9 will be described.

  When the user starts the pressing operation of the release switch 31, the same processes as in steps S31 and S32 of FIG. 8 are performed in steps S81 and S82, respectively.

  In step S82, when the motion vector detection unit 55 determines that the captured image read from the SDRAM 54 is the first captured image, the process proceeds to step S83, and the correction unit uses the first captured image as an average image. 57, supplied to the SDRAM 333 via the 1 / n multiplier 331 and the adder 332 and stored therein. Furthermore, in step S83, the motion vector detection unit 55 supplies the first captured image as a reference image to the SAD table 56 for storage, and the process proceeds to step S90.

  On the other hand, if it is determined in step S82 that the motion vector detection unit 55 is not the first captured image (the second and subsequent captured images), the process proceeds to step S84, and in steps S84 to S86, FIG. The same processing as in steps S34 to S36 is performed.

  Thereafter, the process proceeds from step S86 to step S87, where the 1 / n multiplication unit 331 (average image generation unit 58) multiplies the pixel value of the nth captured image supplied from the correction unit 57 by 1 / n, and the result. The obtained 1 / n image is supplied to the adding unit 332, and the process proceeds to step S88.

  In step S88, the (n−1) / n multiplication unit 334 obtains the (n−1) / n image obtained by multiplying the pixel value of the average image read from the SDRAM 333 by (n−1) / n, The data is supplied to the adding unit 332, and the process proceeds to step S89.

  In step S89, the adder 332 includes pixels of the 1 / n image supplied from the 1 / n multiplier 331 and the (n-1) / n image supplied from the (n-1) / n multiplier 334. The values are added, and the image obtained as a result is supplied to the SDRAM 333 as a new average image, stored in an overwritten form, and proceeds to step S90. Hereinafter, in steps S90 to S92, the steps of FIG. The same processing as S39 to S41 is performed. Thereafter, the process ends.

  8 and FIG. 10, the correction unit 57 corrects the position of the captured image captured by the image capturing unit 41 to the position of the reference image. It is possible to correct image shake (positional deviation) such as camera shake occurring inside.

  In addition, since the average image is displayed every time the average image is generated by the average image generation unit 58 in the display unit 61, the user can confirm the degree of erasure of moving objects in the average image in real time. . Thereby, the user can obtain an image (average image) of proper exposure from which the moving object is erased. Here, the image of the appropriate exposure is, for example, a predetermined shutter speed set by the digital camera 11 when using the automatic exposure (AE) function of the digital camera 11 when capturing in the normal shooting mode. When the user sets a predetermined shutter speed and aperture by manual operation, it means an image of the exposure amount according to the setting.

  A series of processes performed by the motion vector detection unit 55, the correction unit 57, the average image generation unit 58, the display control unit 60, and the input / output control unit 62 described above can be executed by dedicated hardware, Can also be executed. When a series of processing is executed by software, a program constituting the software can execute various functions by installing a so-called embedded computer or various programs. For example, it is installed from a program storage medium in a general-purpose personal computer.

  FIG. 11 is a block diagram illustrating a configuration example of a computer that executes the above-described series of processing by a program.

  A CPU (Central Processing Unit) 361 executes various processes according to a program stored in a ROM (Read Only Memory) 362 or a storage unit 368. A RAM (Random Access Memory) 363 appropriately stores programs executed by the CPU 361, data, and the like. The CPU 361, ROM 362, and RAM 363 are connected to each other by a bus 364.

  An input / output interface 365 is also connected to the CPU 361 via the bus 364. Connected to the input / output interface 365 are an input unit 366 including a keyboard, a mouse, and a microphone, and an output unit 367 including a display and a speaker. The CPU 361 executes various processes in response to commands input from the input unit 366. Then, the CPU 361 outputs the processing result to the output unit 367.

  The storage unit 368 connected to the input / output interface 365 is composed of, for example, a hard disk, and stores programs executed by the CPU 361 and various data. The communication unit 369 communicates with an external device via a network such as the Internet or a local area network.

  Further, a program may be acquired via the communication unit 369 and stored in the storage unit 368.

  A drive 370 connected to the input / output interface 365 drives a removable medium 371 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 368 as necessary.

  As shown in FIG. 11, a program storage medium that stores a program that is installed in a computer and can be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read). Removable media 371, which is a package media consisting of only memory), DVD (digital versatile disc), magneto-optical disc (including MD (mini-disc)), or semiconductor memory, or a program that is temporary or permanent ROM 362 that is stored in the memory, and a hard disk that constitutes the storage unit 368. The program is stored in the program storage medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 369 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program storage medium is not limited to the processing performed in time series according to the described order, but is not necessarily performed in time series. Or the process performed separately is also included.

  In the moving object erasure shooting mode processing of FIGS. 8 and 10 as described above, the motion vector detection unit 55 of FIG. 1 detects the motion vector by the block matching described above, but the motion vector detection method is as follows. For example, the motion vector may be detected by a gradient method.

  The motion vector detection unit 55 in FIG. 1 can also detect a motion vector using a reference image reduced at an appropriate reduction ratio and a captured image (second and subsequent images).

  Further, the correction unit 57 in FIG. 1 performs correction for aligning the subject by affine transformation. For example, by using a sensor such as an angular velocity sensor or an acceleration sensor for the digital camera 11, image blurring is performed. It may be detected and optically corrected.

  The display control unit 60 displays the average image on the display unit 61 each time the average image generation unit 58 generates a new average image. For example, the display control unit 60 generates the average image using the average image generation unit 58. For example, the average image to be displayed may be displayed every m (<N). As a result, the display control unit 60 is burdened of displaying an average image each time a new average image is generated by the average image generation unit 58, compared to when the average image is displayed on the display unit 61. Can be reduced.

  Further, in the 1 / n multiplication unit 83 in FIG. 2, every time a new addition image is generated by the addition unit 81, the pixel value of the addition image is multiplied by 1 / n. For example, the pixel value of the generated addition image may be multiplied by 1 / n every m (<N). Thus, each time a new addition image is generated by the addition unit, the 1 / n multiplication unit 83 multiplies the pixel value by 1 / n times as compared with the case where the pixel value of the addition image is multiplied by 1 / n. Can be reduced.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram which shows the structural example of one Embodiment of the digital camera to which this invention is applied. It is a block diagram which shows the detailed structural example of the average image generation part of FIG. It is a figure explaining the method to detect the motion vector of the 2nd or more captured image by the motion vector detection part of FIG. It is another figure explaining the method of detecting the motion vector of the 2nd or more captured image by the motion vector detection part of FIG. It is a figure explaining correction | amendment of the captured image after the 2nd sheet | seat by the correction | amendment part of FIG. It is a figure which shows a mode that the average image production | generation part of FIG. 1 produces | generates a new average image. It is a figure which shows a mode that an average image is displayed on the display part of FIG. 3 is a flowchart for describing processing in an imaging mode for moving object elimination when the average image generation unit in FIG. 1 has the configuration shown in FIG. 2. It is another block diagram which shows the detailed structural example of the average image generation part of FIG. 10 is a flowchart for describing processing in an imaging mode for moving object elimination when the average image generation unit in FIG. 1 has the configuration shown in FIG. 9. It is a block diagram which shows the structural example of a computer.

Explanation of symbols

  21 operation unit, 31 release switch, 41 imaging unit, 54 SDRAM, 55 motion vector detection unit, 56 SAD table, 57 correction unit, 58 average image generation unit, 60 display control unit, 61 display unit, 62 input / output control unit , 63 Storage unit, 64 drives, 65 Memory card

Claims (14)

In an imaging device that captures an image,
Imaging means for imaging an image by photoelectrically converting incident light; and
Operation means operated by a user;
An average image generating means for generating an average image obtained by averaging pixel values of a plurality of images captured by the imaging means with an exposure time of appropriate exposure while the operating means is being operated;
An image pickup apparatus comprising: a recording control unit that records the average image on a recording medium when the operation of the operation unit is stopped. Display means for displaying an image;
The imaging apparatus according to claim 1, further comprising: a display control unit that causes the display unit to display a new average image each time a new average image is generated in the average image generation unit. A correction unit that corrects the image captured by the imaging unit so as to align the position of the subject shown in each of the plurality of images captured by the imaging unit while the operation unit is operated;
The imaging apparatus according to claim 1, wherein the average image generation unit averages a plurality of images corrected by the correction unit. The average image generating means includes
When the n-th image is picked up by the image pickup means, an addition image obtained by adding the n−1 images and the n-th image are added to add a new image obtained by adding the n-th image. Adding means for generating a summation image;
The imaging apparatus according to claim 1, further comprising: 1 / n multiplication means for multiplying a pixel value of a new addition image obtained by the addition means by 1 / n. The average image generating means includes
1 / n multiplication means for multiplying the pixel value of the nth image by 1 / n when the nth image is picked up by the imaging means;
(n-1) / n multiplication means for multiplying the pixel value of the average image generated from n-1 images by (n-1) / n times,
2. An adding unit that adds pixel values of the n-th image having a pixel value multiplied by 1 / n and the average image having a pixel value multiplied by (n−1) / n. The imaging device described in 1. In an imaging method of an imaging apparatus including an imaging unit that images an image by photoelectrically converting incident light,
While the operating means operated by the user is being operated, the imaging means generates an average image that averages the pixel values of a plurality of images captured with an exposure time of appropriate exposure,
An imaging method including a step of recording the average image on a recording medium when the operation of the operation means is stopped. In a program for causing a computer to execute an imaging process of an imaging apparatus including an imaging unit that captures an image by photoelectrically converting incident light,
While the operating means operated by the user is being operated, the imaging means generates an average image that averages the pixel values of a plurality of images captured with an exposure time of appropriate exposure,
A program for causing a computer to execute an imaging process including a step of recording the average image on a recording medium when the operation of the operation means is stopped. In an imaging device that captures an image,
Imaging means for imaging an image by photoelectrically converting incident light; and
An average image generating means for generating an average image obtained by averaging pixel values of a plurality of images captured by the imaging means with an exposure time of appropriate exposure; and
Display means for displaying an image;
An image pickup apparatus comprising: a display control unit that causes the display unit to display a new average image each time a new average image is generated in the average image generation unit. The imaging apparatus according to claim 8, further comprising recording control means for recording the average image on a recording medium. A correction unit that corrects the image captured by the imaging unit so as to align the position of the subject shown in each of the plurality of images captured by the imaging unit while the operation unit is operated;
The imaging apparatus according to claim 8, wherein the average image generation unit averages a plurality of images corrected by the correction unit. The average image generating means includes
When the n-th image is picked up by the image pickup means, an addition image obtained by adding the n−1 images and the n-th image are added to add a new image obtained by adding the n-th image. Adding means for generating a summation image;
The imaging apparatus according to claim 8, further comprising: 1 / n multiplication means for multiplying a pixel value of a new addition image obtained by the addition means by 1 / n. The average image generating means includes
1 / n multiplication means for multiplying the pixel value of the nth image by 1 / n when the nth image is picked up by the imaging means;
(n-1) / n multiplication means for multiplying the pixel value of the average image generated from n-1 images by (n-1) / n times,
9. An adding unit that adds pixel values of the n-th image having a pixel value multiplied by 1 / n and the average image having a pixel value multiplied by (n−1) / n. The imaging device described in 1. In an imaging method of an imaging apparatus including an imaging unit that images an image by photoelectrically converting incident light,
Generating an average image obtained by averaging pixel values of a plurality of images captured by the imaging unit with an exposure time of appropriate exposure;
An imaging method including a step of displaying a new average image on a display unit that displays an image every time a new average image obtained by averaging images captured by the imaging unit is generated. In a program for causing a computer to execute an imaging process of an imaging apparatus including an imaging unit that captures an image by photoelectrically converting incident light,
Generating an average image obtained by averaging pixel values of a plurality of images captured by the imaging unit with an exposure time of appropriate exposure;
A program that causes a computer to execute an imaging process including a step of displaying a new average image on a display unit that displays an image every time a new average image obtained by averaging images captured by the imaging unit is generated.

Images