JP5760654B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

The present invention relates to an image processing apparatus , an image processing method, and a program.

In recent digital cameras, blurring of an image caused by camera shake occurring within an imaging time is suppressed by various means such as adjustment of shooting conditions and image processing. Hereinafter, such correction is referred to as “camera shake correction”.
As a method for correcting camera shake, several continuous frames of a moving image or several still images with a short shutter time obtained by continuous shooting are processed. A method of combining processing objects by aligning data of each pixel to be processed and averaging each of them, that is, a so-called multiplane addition method is used.
Hereinafter, an image obtained as a result of combining processing targets (frame group or still image group) by the multi-plane addition method is referred to as a “composite image”.

Here, in the multi-plane addition method, when all the pixels (all regions) of the frame group or the still image group are subjected to addition averaging, the camera shake region is effectively corrected in the resultant composite image. However, in the area of the moving subject, blurring occurs due to simple composition.
In order to reduce such blurring of a moving subject, Japanese Patent Application Laid-Open No. 2004-133867 discloses a method in which only the area other than the moving subject is added and averaged while the moving subject area is not added and averaged.

JP 2009-290827 A

However, in the method described in Patent Document 1, as a process of moving subject areas, a predetermined one of a frame group or a still image group is used as a reference image, and the moving subject area in the reference image is directly adopted and pasted. Is adopted. In many cases, subject blurring occurs in such a moving subject region in the reference image.
Here, subject blur means that data (pixel value) of each pixel is obtained by integrating the amount of light accumulated in each pixel of the image sensor during the exposure time (shutter time) for capturing one still image. However, if the subject moves within the exposure time, the light before and after the movement is mixed and integrated, which means that the region of the subject is blurred.
For this reason, even if the method described in Patent Document 1 is applied, subject blurring cannot be avoided, and the image quality of the composite image after image processing according to the multiplane addition method is deteriorated.

  The present invention has been made in view of such a situation, and in a composite image after image processing according to the multiplane addition method, image quality is reduced by reducing both blur caused by camera shake and subject blur and subject blur. It aims at improving.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention is obtained by an image acquisition unit that acquires data of a plurality of captured images including the same subject under a plurality of different shooting conditions, and the image acquisition unit. A condition determining unit that determines whether or not the shooting condition satisfies a predetermined condition regarding the movement of the subject among the plurality of different shooting conditions for each of the plurality of captured image data; and the condition determining unit Based on the determination result, from among the plurality of captured image data acquired by the image acquisition means, a synthesis reference image selection means for selecting data of a synthesis reference image that serves as a reference when performing image synthesis; Of the plurality of captured image data acquired by the image acquisition means, two or more image capturing data including the composite reference image data selected by the composite reference image selection means. A synthesis unit configured to generate composite image data by executing image synthesis processing according to a multi-plane addition method using the image data as a processing target based on the synthesis reference image. .

  According to the present invention, in a composite image after image processing according to the multiplane addition method, it is possible to improve image quality by reducing both blur caused by camera shake and subject blur and subject blur.

1 is a block diagram illustrating a hardware configuration of an imaging apparatus as an embodiment of an image processing apparatus of the present invention. It is a functional block diagram which shows the functional structure for performing an imaging process among the functional structures of such an imaging device. 3 is a flowchart for explaining a flow of imaging processing executed by the imaging apparatus of FIG. 1 having the functional configuration of FIG. 2. It is a schematic diagram which shows an example of the selection method of the process target of the image process according to a multiplane addition system.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of an imaging apparatus 1 as an embodiment of an image processing apparatus of the present invention.
The imaging device 1 is configured as a digital camera, for example.

  The imaging device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an image processing unit 14, a bus 15, an input / output interface 16, and an imaging unit. 17, an input unit 18, an output unit 19, a storage unit 20, a communication unit 21, and a drive 22.

  The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 20 to the RAM 13.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

The image processing unit 14 is configured by a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like, and performs various image processing on image data in cooperation with the CPU 11.
For example, the image processing unit 14 performs image processing according to the multi-plane addition method on the data of a plurality of captured images continuously output from the imaging unit 17, and outputs the resultant composite image data. To do.

  The CPU 11, ROM 12, RAM 13, and image processing unit 14 are connected to each other via a bus 15. An input / output interface 16 is also connected to the bus 15. An imaging unit 17, an input unit 18, an output unit 19, a storage unit 20, a communication unit 21, and a drive 22 are connected to the input / output interface 16.

  Although not shown, the imaging unit 17 includes an optical lens unit including a focus lens and a zoom lens, and an image sensor.

The optical lens unit includes various lenses such as a focus lens and a zoom lens in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 17.
In the present specification, such an output signal of the imaging unit 17 is hereinafter referred to as “captured image data”. The captured image data is appropriately supplied to the CPU 11.

  The operation of the imaging unit 17 is controlled by the imaging control unit 41 described with reference to FIG. For example, the imaging control unit 41 sets various imaging conditions for the imaging unit 17. For example, the imaging control unit 41 sets a zoom magnification as one of the imaging conditions, and controls a lens driving unit (not shown) to drive the zoom lens so that the set zoom magnification is obtained.

The input unit 18 includes various buttons such as a power button and a shutter button, and inputs various types of information according to user instruction operations.
The output unit 19 includes a display, a speaker, and the like, and outputs images and sounds.
The storage unit 20 is configured by a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 21 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 31 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 22. The program read from the removable medium 31 by the drive 22 is installed in the storage unit 20 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 20 in the same manner as the storage unit 20.

FIG. 2 is a functional block diagram showing a functional configuration for executing the imaging process among the functional configurations of the imaging apparatus 1 as described above.
The “imaging processing” here refers to imaging a plurality of captured images repeatedly and continuously by so-called continuous shooting, and performing image processing according to the multi-plane addition method on the data of the plurality of captured images. This refers to a series of processing until recording of the resultant composite image data.

  When the execution of the imaging process is controlled, in the CPU 11, the imaging control unit 41, the image acquisition unit 42, the display control unit 43, and the composite reference image selection unit 44 function. In the image processing unit 14, a condition determination unit 51 and a synthesis unit 52 function.

  The imaging control unit 41 sets various imaging conditions according to various operations on the input unit 18 by the user and controls the imaging operation of the imaging unit 17 based on the various imaging conditions. The imaging unit 17 performs an imaging operation based on the control of the imaging control unit 41, outputs captured image data obtained as a result, and temporarily records the data in the storage unit 20.

The image acquisition unit 42 outputs the captured image output from the imaging unit 17 and temporarily stored in the storage unit 20 via the input / output interface 16 before the recording instruction operation by pressing the shutter button of the input unit 18. Data is acquired as live view image data.
That is, in addition to the image acquisition unit 42, the CPU 11 that functions as the imaging control unit 41 and the display control unit 43 performs a live view imaging process and a live view display process during the imaging process, thereby generating a live view image. The image is displayed on the output unit 19 (display of the imaging apparatus 1).
Specifically, the imaging control unit 41 continues the imaging operation by the imaging unit 17 when the imaging process starts. Then, while the image capturing operation by the image capturing unit 17 is continued, the image capturing control unit 41 temporarily stores the captured image data sequentially output from the image capturing unit 17 in the memory (the storage unit 20 in the present embodiment). Remember me. Such a series of control processes is the “live view imaging process” referred to herein.
In addition, the image acquisition unit 42 sequentially reads the data of each captured image temporarily recorded in the memory (the storage unit 20 in the present embodiment) during the live view imaging process, and supplies the data to the display control unit 43. . The display control unit 43 causes the output unit 19 to sequentially display the captured image. Such a series of control processes is the “live view display process” referred to herein. The captured image displayed on the output unit 19 by the live view display process is the “live view image” here.

  Thereafter, when a recording instruction operation is performed by pressing the shutter button of the input unit 18, the imaging unit 17 performs a so-called continuous shooting operation under the control of the imaging control unit 41, and outputs data of a plurality of captured images. And once stored in the storage unit 20. The image acquisition unit 42 acquires the data of the plurality of captured images as a processing target for executing image processing according to the multiplane addition method, and supplies the acquired processing target to the condition determination unit 51.

The condition determination unit 51 determines whether or not various conditions for selecting a composite reference image to be described later are satisfied for each data of a plurality of captured images acquired as processing targets by the image acquisition unit 42.
The type and number of such conditions are not particularly limited, but in this embodiment, the first condition that camera shake and large subject blur do not occur, the subject blur of the person's face (especially eyes) does not occur. A second condition, a third condition that the person's eyes are not collapsed, and a fourth condition that satisfies a predetermined imaging condition such as a shutter time being shorter than a predetermined reference are employed.
For this reason, the condition determination unit 51 includes a blur determination unit 61 that determines the first condition, a face determination unit 62 that determines the second condition, an eye determination unit 63 that determines the third condition, and a fourth condition. An imaging condition determination unit 64 for determination is provided.

  The composite reference image selection unit 44 comprehensively determines the determination results of various conditions by the condition determination unit 51, and from a plurality of captured image data acquired as processing targets by the image acquisition unit 42, a predetermined one sheet The data is used as image data (hereinafter referred to as “compositing reference image data”) used as a reference in the combining unit 52 described later.

The synthesizing unit 52 performs image processing according to the multi-plane addition method on the data of the plurality of captured images acquired as the processing target by the image acquiring unit 42, and the resultant synthesized image data is output to the drive 22. And stored as removable media 31.
In this case, only the area other than the moving subject is added and averaged, and the moving subject area is not added and averaged, and what is included in the synthesized reference image selected by the synthesized reference image selection unit 44 is directly adopted in the synthesized image. The
More precisely, the synthesis unit 52 calculates the correlation between the data of the synthesis reference image and the data of the other captured image for each pixel, and performs other imaging only for the highly correlated pixels (mainly pixels other than the moving subject). The pixel values of the image are added and averaged, and the pixel values of pixels with low correlation (mainly moving subject pixels) are used without being added and averaged.
Here, the synthesis reference image selection unit 44 selects, as a synthesis reference image, a captured image that has no subject blur or a minimum motion blur for a moving subject by comprehensively considering the determination results of various conditions by the condition determination unit 51. Can do. A moving subject included in such a composite reference image, that is, a subject having no subject blur or a minimum moving subject is adopted as it is in the composite image, and as a result, subject blur is suppressed to a minimum and subject blurring is not caused. In a region where no noise is generated, noise is reduced by averaging.

Next, an imaging process executed by the imaging apparatus 1 having the functional configuration of FIG. 2 described above will be described with reference to FIG.
FIG. 3 is a flowchart for explaining the flow of imaging processing executed by the imaging apparatus 1 of FIG. 1 having the functional configuration of FIG.

  In the imaging process, a predetermined operation of the input unit 18 by the user is performed, the operation mode of the imaging apparatus 1 is switched to, for example, the imaging mode, and the continuous-shot composite image mode for reducing subject blur is selected. The following processing is executed at the start.

  In step S11, the imaging control unit 41 sets a continuous-shot composite image mode.

In step S12, the display control unit 43 starts a live view image display process. Thereby, the live view image is displayed on the display of the output unit 19.
In the present embodiment, it is assumed that the live view image continues to be displayed on the display of the output unit 19 until the imaging process is completed.

In step S <b> 13, the imaging control unit 41 determines whether a recording instruction operation is performed by pressing the shutter button of the input unit 18.
When the recording instruction operation is not performed, it is determined as NO in Step S13, and the process returns to Step S13 again. That is, until the recording instruction operation is performed, the determination process in step S13 is repeatedly executed.

Here, in this embodiment, the following two types of operations are employed as the pressing operation of the shutter button of the input unit 18.
That is, the first operation is an operation in which the shutter button of the input unit 18 is pressed halfway (a predetermined position that does not reach the lower limit), and is called “half-press operation” or simply “half-press”. . When half-pressed, the imaging control unit 41 executes an AF (Auto Focus) process or the like.
The second operation is an operation in which the shutter button of the input unit 18 is pressed to the lower limit, and is called “full press operation” or simply “full press”. This full press is adopted as a recording instruction operation in the present embodiment.
Accordingly, when the button is fully pressed, it is determined as YES in Step S13, and the process proceeds to Step S14.

In step S14, the imaging control unit 41 determines that the exposure is determined to be appropriate exposure (when the shutter time obtained by lowering the two-stage shutter time with respect to the shutter time specified by the imaging condition is a predetermined reference). In order to obtain a total of eight captured images of four sets of images with a reduced (shortened) two-stage shutter time and images with a reduced (shortened) three-stage shutter time, continuous shooting is performed on the imaging unit 17. Make it work.
As a result, data of a total of eight captured images is output from the imaging unit 17 and temporarily stored in the storage unit 20. When the data of these eight captured images is acquired by the image acquisition unit 42 and supplied to the condition determination unit 51, the process proceeds to step S15.

  In step S15, the condition determination unit 51 performs demosaic processing on each data of a total of eight captured images, and converts the form from RAW data to YUV data.

In step S <b> 16, the condition determination unit 51 aligns each data of a total of eight captured images.
Specifically, the condition determining unit 51, the temporally sets the data of the center of the captured image as a registration datum, projective transformation matrix to the position data of the standards image to match its position, the position match was based It calculates | requires with respect to the data of each captured image other than a quasi image. Then, the condition determining unit 51 performs alignment of each data by applying the projective transformation matrix to each captured image data.

  In step S <b> 17, the condition determination unit 51 tries to detect a face from each data of a total of eight captured images and determines whether or not a face exists.

When the face does not exist, it is determined as NO in Step S17, and the process proceeds to Step S18.
In step S <b> 18, the condition determination unit 51 performs blur determination on a total of four pieces of captured image data with a three-stage shutter time lowered among a total of eight pieces of captured image data.
Here, among the data of a total of eight captured images, the three-stage shutter time was lowered with respect to a predetermined reference (the shutter time obtained by reducing the two-stage shutter time with respect to the shutter time specified by the shooting condition). The fact that each of the data of a total of four captured images is targeted is that the imaging condition determination unit 64 determines whether or not the fourth condition that the shutter time is short with respect to a predetermined reference is satisfied as an imaging condition. means.
The blur determination means determining whether or not the first condition that no camera shake or large subject blur occurs is satisfied. That is, the blur determination unit 61 performs blur determination by determining whether or not the first condition is satisfied with respect to each of four pieces of captured image data with the three-stage shutter time lowered. In blur determination, differential image data is generated, and determination using the pixel value is performed.
As described above, more accurate determination can be made by using the differential image of only the target image for identification instead of the motion vector between the images for identifying the subject blur. This is because if a motion vector between images is used, it is an average motion between two images, and if the motion is not linear, accurate determination cannot be made. The method of generating differential image data is not particularly limited, and for example, a method using primary differentiation such as Sobel, Robinson, Prewitt operator, or secondary differentiation such as Laplacian operator is applicable.
In step S19, the compositing reference image selection unit 44 determines subject blurring and hand movement from the image group (four captured images) in which the three-step shutter time is reduced based on the blur determination result in step S18. The image with the least blur is selected as the composite reference image.

In this way, if there is no face in each data of the eight captured images, it is determined NO in step S17, and the composite reference image is selected by the processing in steps S18 and S19.
On the other hand, when a face exists in each data of a total of eight captured images, it is determined as YES in step S17, the process proceeds to step S20, and the following process is executed.

In step S <b> 20, the condition determination unit 51 performs face determination and eye determination for each of the total of four captured image data with the three-stage shutter time being reduced among the total of eight captured image data.
Here, among the data of a total of eight captured images, targeting the total of four captured image data with the three-stage shutter time lowered is the same as in step S18, and the shutter time is short as the imaging condition. This means that the imaging condition determination unit 64 determines whether or not the fourth condition is satisfied.
Further, the face determination means determining whether or not the second condition that subject blurring of a human face (particularly eyes) does not occur is satisfied. That is, the face determination unit 62 performs face determination by determining whether or not the second condition is satisfied for a total of four pieces of captured image data with the three-step shutter time lowered. Note that when eye subject blur is determined, eye detection is performed. In the determination of subject blur, differential image data is generated and a determination is made using the pixel value.
The eye determination means determining whether or not a third condition that the person's eyes are not collapsed is satisfied. That is, the eye determination unit 63 performs eye determination by determining whether or not the third condition is satisfied with respect to each of four pieces of captured image data with the three-step shutter time lowered. Note that eye detection is performed as a prerequisite for eye determination.
In step S21, the composite reference image selection unit 44 selects an eye from an image group (four captured images) obtained by reducing the three-step shutter time based on the results of the face determination and the eye determination in step S20. The image with the smallest subject blurring of the face (eyes) is selected as the composite reference image.

  In this way, when the composite reference image is selected in the process of step S19 or S21, the process proceeds to step S22.

  In step S <b> 22, the combining unit 52 selects four captured image data including the composite reference image from among the total of eight captured image data as a processing target, and applies the multiplane addition method to the processing target. Follow the image processing.

Here, the selection method of the processing target of the image processing according to the multiplane addition method is not particularly limited as long as it is selected so as to include the synthesis reference image, but in this embodiment, the selection method shown in FIG. 4 is adopted. ing.
FIG. 4 shows an example of a method for selecting a processing target of image processing according to the multi-plane addition method.
The composite reference image 72 is selected from a total of four captured images in which the three-stage shutter time is shortened as described above. Therefore, in the example of FIG. 4, as the remaining three processing objects, the first picked-up image 71 and the third picked-up image out of a total of four picked-up images with the two-step shutter time shortened. A captured image 73 and a captured image 74 are selected.
In this case, the image processing according to the multi-plane addition method is executed by averaging the pixel values to be processed only in regions of the composite reference image 72 that are highly correlated with the captured images 71, 73, and 74. Is done. As a result, data of the composite image 81 in which the subject is not blurred (blurred) and the background noise is small is obtained.

Returning to FIG. 3, in step S <b> 23, the compositing unit 52 records the composite image data generated in the process of step S <b> 22 in this manner on the removable medium 31.
Thereby, the imaging process is completed.

As described above, the imaging apparatus 1 according to the present embodiment includes the image acquisition unit 42, the condition determination unit 51, the synthesis reference image selection unit 44, and the synthesis unit 52.
The image acquisition unit 42 acquires data of a plurality of captured images including the same subject.
The condition determination unit 51 determines whether or not a predetermined condition regarding the movement of the subject is satisfied for each of the plurality of captured image data acquired by the image acquisition unit 42.
Based on the determination result of the condition determination unit 51, the combination reference image selection unit 44 selects the data of the combination reference image from among the plurality of captured image data acquired by the image acquisition unit 42.
The synthesizing unit 52 processes two or more captured image data including the synthesized reference image data selected by the synthesized reference image selecting unit 44 among the plurality of captured image data acquired by the image acquiring unit 42. Then, composite image data is generated by executing image processing according to the multi-plane addition method.
Here, by adopting an appropriate one as the predetermined condition, it is possible to minimize the risk of selecting a blink image or a subject blur image as a composite reference image. Further, in an area where no subject blur occurs, noise due to a shortened shutter time for camera shake suppression can be achieved.
Moreover, the condition determination part 51 determines whether the conditions regarding the imaging condition when a some captured image is imaged are satisfy | filled. Then, the condition determination unit 51 determines whether or not the condition regarding the shutter time is satisfied as the condition regarding the imaging condition, thereby minimizing the risk of selecting the subject blurred image as the composite reference image. it can.
In other words, in a composite image after image processing according to the multi-plane addition method, noise caused by shortening the shutter time for camera shake suppression, etc., and subject blur are both reduced to improve image quality. Can do.
In particular, by adopting a condition that the shutter time is short with respect to a predetermined reference as a condition relating to the shutter time, an image that is less affected by subject blurring and camera shake as the shutter time is short as data of the composite reference image. It becomes possible to select the data.
Moreover, since the image data to be combined includes image data shot with a predetermined standard, the necessary brightness is ensured as compared with a case where only image data shot with a shutter time shorter than the predetermined reference is combined. Therefore, it is possible to reduce the number of times of image data synthesis required for this purpose, and it is possible to reduce processing time and noise associated with the synthesis processing.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

In the above-described embodiment, the conditions determined by the condition determination unit 51 are the first condition that no camera shake or large subject blur occurs, and the second condition that no subject blur occurs on the human face (particularly eyes). The third condition that the person's eyes are not collapsed and the fourth condition that satisfies a predetermined imaging condition such as a shutter time shorter than the predetermined reference are employed.
However, the condition determined by the condition determination unit 51 is sufficient if it is a condition related to the movement of the subject, and the type and number are not particularly limited.
For example, the image acquisition unit 42 acquires data of a plurality of captured images before and after the shutter full-press timing, and among the data of the plurality of captured images, those close to the shutter full-press timing, It is also possible to adopt a condition of selecting as a composite reference image.

Further, in the above-described embodiment, the synthesis reference image selection unit 44 treats each condition as the same importance and selects the synthesis reference image comprehensively.
However, it is not particularly necessary that the conditions have the same importance, and the composition reference image selection unit 44 weights each condition according to the importance and selects a composition reference image based on the weighted conditions. May be.

In the above-described embodiment, the target of image processing according to the multi-plane addition method is selected from the group of four captured images in which the three-stage shutter time is shorter than the predetermined reference, as shown in FIG. A composite reference image 72 and three captured images 71, 73, 74 with a two-step shutter time shorter than a predetermined reference are employed.
However, the target of image processing according to the multi-plane addition method is not particularly limited, and conversely, the composite reference image selected from the captured image group in which the two-step shutter time is shorter than the predetermined reference, and the predetermined reference One or more captured images in which the three-stage shutter time is shortened may be employed.
Alternatively, the change in the shutter time is not particularly limited to two or three stages, and may be any combination of arbitrary stages. In other words, shortening the shutter time by two steps or three steps means shortening the shutter time stepwise, for example, 1/2 or 1/4. However, the stage and degree of shortening the shutter time can be set as appropriate according to the mounting. In this case, any one composite reference image and any one or more picked-up images can be targeted for image processing according to the multiplane addition method.
Furthermore, as the predetermined reference for the shutter time, the shutter time obtained by lowering the two-stage shutter time is employed in the above-described embodiment, but the present invention is not limited to this. For example, only one stage, three stages, or other stages are used. A shutter time obtained by lowering the shutter time, a fixed value determined in advance, or a variable value calculated by a predetermined algorithm can be used.

In addition, the image processing apparatus of the present embodiment is applied to the imaging apparatus 1, but is not limited to this, and can be applied to any apparatus as long as it can perform image processing. .
For example, the present invention can be applied to electronic devices in general. Specifically, for example, the present invention can be applied to a smartphone, a notebook personal computer, a television receiver, a video camera, a portable navigation device, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 2 is merely an example and is not particularly limited. That is, it is sufficient that the imaging apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional blocks are used to realize this function is not particularly limited to the example of FIG.
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. The recording medium etc. provided in The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which a program is recorded, the hard disk included in the storage unit 20 in FIG.

In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.
Further, in the present specification, the term “system” means an overall apparatus configured by a plurality of devices, a plurality of means, and the like.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalents thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
Image acquisition means for acquiring data of a plurality of captured images including the same subject;
Condition determining means for determining whether or not a predetermined condition regarding the movement of the subject is satisfied for each data of the plurality of captured images acquired by the image acquiring means;
Based on the determination result of the condition determination unit, a combination reference image selection unit that selects data of a combination reference image for image combination from the data of the plurality of captured images acquired by the image acquisition unit;
Among the plurality of captured image data acquired by the image acquisition unit, multi-plane processing is performed on two or more captured image data including the composite reference image data selected by the composite reference image selection unit. Combining means for generating composite image data by executing image processing according to an addition method;
An image processing apparatus comprising:
[Appendix 2]
The condition determining means determines whether or not a condition regarding subject blur is satisfied as at least one of the predetermined conditions;
The image processing apparatus according to appendix 1, wherein:
[Appendix 3]
The condition determining means identifies the subject blur using data of differential images of the plurality of captured images, and determines whether or not a condition regarding the subject blur is satisfied based on a result of the identification.
The image processing apparatus according to Supplementary Note 2, wherein
[Appendix 4]
When the subject is a person, the condition determining unit determines whether or not a condition regarding the subject blur is satisfied with respect to subject blur in the eyes of the person.
The image processing apparatus according to appendix 2 or 3, characterized by the above.
[Appendix 5]
The condition determination means determines whether or not a condition that the subject's eyes are not closed when the subject is a person,
The image processing apparatus according to any one of supplementary notes 1 to 4, wherein
[Appendix 6]
The condition determining means determines whether or not a condition regarding an imaging condition when the plurality of captured images are captured is satisfied;
The image processing apparatus according to any one of supplementary notes 1 to 5, characterized in that:
[Appendix 7]
The condition determining means determines whether or not a condition relating to a shutter time is satisfied as a condition relating to the imaging condition;
The image processing apparatus according to appendix 6, wherein:
[Appendix 8]
The image acquisition means sets a shooting condition in which a shutter time when capturing each of the plurality of captured images is shortened by a predetermined ratio with respect to a shutter time corresponding to appropriate exposure,
The image processing apparatus according to appendix 7, wherein the condition determination unit determines whether or not a condition that a shutter time is short with respect to a predetermined reference is satisfied as a condition relating to the imaging condition.
[Appendix 9]
The image acquisition means captures the plurality of photographed images under a plurality of photographing conditions with different shutter times,
9. The image processing apparatus according to appendix 8, wherein the condition determination unit determines whether or not a shooting condition corresponding to the shortest shutter time among the plurality of shooting conditions.
[Appendix 10]
A computer that controls image processing performed on data of a plurality of captured images including the same subject;
Condition determining means for determining whether or not a predetermined condition regarding the movement of the subject is satisfied for each of the data of the plurality of captured images;
Based on the determination result of the condition determination unit, a synthesis reference image selection unit that selects data of a synthesis reference image for image synthesis from the data of the plurality of captured images;
Image processing according to a multi-plane addition method is performed on two or more captured image data including the composite reference image data selected by the composite reference image selection unit from among the plurality of captured image data. A synthesis means for generating composite image data by executing,
A program characterized by functioning as

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Image processing part, 15 ... Bus, 16 ... Input / output interface, 17 ... Image pick-up unit 18 ... Input unit 19 ... Output unit 20 ... Storage unit 21 ... Communication unit 22 ... Drive 31 ... Removable media 41 ... Imaging Control unit, 42 ... Image acquisition unit, 43 ... Display control unit, 44 ... Composition reference image selection unit, 51 ... Condition determination unit, 52 ... Composition unit, 61 ... Blur determination 62, face determination unit, 63 ... eye determination unit, 64 ... imaging condition determination unit

Claims (11)

Image acquisition means for acquiring data of a plurality of captured images including the same subject under a plurality of different shooting conditions ;
Each data of the acquired plurality of captured images by the image acquisition unit, in a plurality of different imaging conditions, determines the condition determining whether the predetermined condition is satisfied photographing condition regarding the movement of the subject Means,
Based on the determination result of the condition determination unit, a combined reference image that selects data of a combined reference image used as a reference when performing image combining from the data of the plurality of captured images acquired by the image acquisition unit A selection means;
Among the plurality of captured image data acquired by the image acquisition unit, two or more captured image data including the composite reference image data selected by the composite reference image selection unit is processed as the processing target. A synthesizing unit for generating synthesized image data by executing an image synthesizing process according to a multi-plane addition method based on a reference image;
An image processing apparatus comprising: The image acquisition means acquires data of a plurality of captured images including the same subject for each of a plurality of different shooting conditions,
Whether the condition determination unit is data of a captured image captured under one of the plurality of different imaging conditions for each of the plurality of captured image data acquired by the image acquisition unit. Determine
The image processing apparatus according to claim 1. The condition determining means determines whether or not a condition regarding subject blur is satisfied as at least one of the predetermined conditions;
The image processing apparatus according to claim 1 or 2, characterized in that. The condition determining means identifies the subject blur using data of differential images of the plurality of captured images, and determines whether or not a condition regarding the subject blur is satisfied based on a result of the identification.
The image processing apparatus according to claim 3 . When the subject is a person, the condition determining unit determines whether or not a condition regarding the subject blur is satisfied with respect to subject blur in the eyes of the person.
The image processing apparatus according to claim 3 or 4, characterized in that. The condition determination means determines whether or not a condition that the subject's eyes are not closed when the subject is a person,
The image processing apparatus according to any one of claims 1 to 5, characterized in that. An image obtained by setting a plurality of shooting conditions with different shutter times by shortening the shutter time for capturing each of a plurality of captured images including the same subject at a predetermined ratio with respect to the shutter time corresponding to the appropriate exposure. Acquisition means;
Condition determining means for determining, for each data of the plurality of captured images acquired by the image acquiring means, whether or not the shooting condition corresponds to the shortest shutter time when the plurality of captured images are captured. When,
Based on the determination result of the condition determination unit, a combined reference image that selects data of a combined reference image used as a reference when performing image combining from the data of the plurality of captured images acquired by the image acquisition unit A selection means;
Among the plurality of captured image data acquired by the image acquisition unit, two or more captured image data including the composite reference image data selected by the composite reference image selection unit is processed as the processing target. A synthesizing unit for generating synthesized image data by executing an image synthesizing process according to a multi-plane addition method based on a reference image;
An image processing apparatus comprising: An image processing method applied to data of a plurality of captured images including the same subject acquired under a plurality of different shooting conditions ,
A condition determination process for determining whether or not the imaging conditions satisfy a predetermined condition regarding the movement of the subject among the plurality of different imaging conditions for each of the plurality of captured image data;
Based on the determination result of the condition determination process , a combination reference image selection process that selects, from among the plurality of captured image data, data of a combination reference image that serves as a reference when performing image combination;
A multi-plane based on the composite reference image with two or more captured image data including the composite reference image data selected by the composite reference image selection process as a processing target among the plurality of captured image data A composition process for generating composite image data by executing an image composition process according to the addition method;
An image processing method comprising: A computer that controls image processing performed on data of a plurality of captured images including the same subject acquired under a plurality of different shooting conditions ,
A condition determination function for determining whether or not the image capturing condition satisfies a predetermined condition regarding the movement of the subject among the plurality of different image capturing conditions for each of the plurality of captured image data;
Based on the determination result of the condition determination function, the image synthesis is performed from the data of the plurality of captured images and the data of the plurality of captured images captured under one of the plurality of shooting conditions. A composite reference image selection function for selecting the composite reference image data used as a reference when
A multi-plane based on the composite reference image with two or more captured image data including the composite reference image data selected by the composite reference image selection function as the processing target among the plurality of captured image data A composition function for generating composite image data by executing image composition processing according to the addition method;
A program characterized by realizing. For data of a plurality of captured images including the same subject acquired by setting a plurality of shooting conditions at different shutter times with the shutter time at the time of imaging shortened at a predetermined ratio with respect to the shutter time corresponding to the appropriate exposure An image processing method applied to
A condition determination process for determining whether or not a shooting condition corresponding to the shortest shutter time when the plurality of captured images are captured for each of the plurality of captured image data;
Based on the determination result of the condition determination process, a combination reference image selection process that selects, from among the plurality of captured image data, data of a combination reference image that serves as a reference when performing image combination;
A multi-plane based on the composite reference image with two or more captured image data including the composite reference image data selected by the composite reference image selection process as a processing target among the plurality of captured image data A composition process for generating composite image data by executing an image composition process according to the addition method;
An image processing method comprising: For data of a plurality of captured images including the same subject acquired by setting a plurality of shooting conditions at different shutter times with the shutter time at the time of imaging shortened at a predetermined ratio with respect to the shutter time corresponding to the appropriate exposure To the computer that controls the image processing
A condition determination function for determining whether or not the shooting conditions corresponding to the shortest shutter time when the plurality of captured images are captured for each of the plurality of captured image data;
Based on the determination result of the condition determination function, a combination reference image selection function for selecting, from among the plurality of captured image data, data of a combination reference image serving as a reference when performing image combination;
A multi-plane based on the composite reference image with two or more captured image data including the composite reference image data selected by the composite reference image selection function as the processing target among the plurality of captured image data A composition function for generating composite image data by executing image composition processing according to the addition method;
A program characterized by realizing.

Images