JP2010067125A - Image processing apparatus, method for processing image, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus, a method for processing an image, and a program capable of selecting a background image having less camera shakes. <P>SOLUTION: The processing apparatus generates consecutive multiple photographic frames, obtains a pixel value of the generated photographic frames; generates a pseudo-background image based on the obtained pixel value; extracts only a subject region based on the difference between the generated pseudo-background image and the photographic frames; and selects a background frame used for a background region, excluding the subject region extracted from the photographic frames. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  2. Description of the Related Art Conventionally, a strobe image is known in which a moving subject is photographed with a fixed camera and the movement of the subject is shown in one image (photo). In the era of film cameras, strobe images were generated by performing multiple strobes on a moving object during long exposures. Today, when digital cameras have become popular, image processing by computers in cameras It is possible to generate a strobe image.

For example, in Patent Document 1, an image where subjects do not overlap is selected from continuous shot images, and the difference between the images is used to determine the subject area. Techniques for extraction are disclosed.
Further, for example, in Patent Document 2, a moving object is identified by specifying a time section in which a background is reflected from a correlation pattern of a specific investigation area on a moving image, or a subject before and after an image in which the subject is reflected. There is disclosed a technique that assumes a moving speed of a moving subject by predicting a subject that does not fluctuate using an image in which no image is reflected.
Japanese Patent No. 3793258 Japanese Patent No. 35699992

  However, in the above prior art, the position of the subject is determined by extracting the subject from images where the subject does not overlap, or by identifying a time interval in which a background-like image is captured from a specific area on the moving image. However, there is no room for selection for the background portion that occupies the largest area in the actually generated strobe image. Therefore, when the strobe image is generated, if the background is blurred compared to the subject, there is a problem that the impression of the entire image is deteriorated.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of selecting a background image with small blur.

  The invention described in claim 1 is made to achieve the above object. In the image processing apparatus, an extraction unit that extracts a subject area of a moving subject from a plurality of frames of images, and the plurality of frames of images. Selection means for selecting an image with the smallest blur, and synthesis means for synthesizing a strobe image based on a subject area extracted by the extraction means and a background area of the image selected by the selection means. It is characterized by that.

  According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the selection unit calculates a high-frequency component of the background region for each of the images of the plurality of frames, and the calculated high-frequency component is The largest image is selected.

  According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, the high-frequency component is a sum of square sums of differences between pixel values at obliquely adjacent positions.

  According to a fourth aspect of the present invention, in the image processing apparatus according to the second aspect, the high-frequency component is a sum of square sums of differences in pixel values at adjacent positions in the vertical and horizontal directions.

  According to a fifth aspect of the present invention, in the image processing apparatus according to the first aspect, the selection unit calculates a positional variation amount between the images based on a result of alignment of the images of the plurality of frames. An image having the smallest calculated amount of position variation is selected.

  According to a sixth aspect of the present invention, in the image processing apparatus according to the first aspect of the present invention, there is provided photographing means for continuously photographing images of a plurality of frames and acquiring an evaluation value of focus accuracy by autofocus of each image. In addition, the selection unit may select an image with the smallest blur based on the evaluation value of the focus accuracy acquired by the photographing unit.

  According to a seventh aspect of the present invention, in the image processing device according to the first aspect of the present invention, the image processing device further includes a photographing unit that continuously captures images of a plurality of frames, and a detection unit that detects a blur amount generated in the device. The selection unit calculates a position variation amount between the images of the plurality of frames based on the blur amount detected by the detection unit, and selects an image having the smallest calculated position variation amount. And

  The invention according to claim 8 includes an extraction step of extracting a subject area of a moving object from a plurality of frames of images, a selection step of selecting an image with the smallest blur among the plurality of frames of images, and the extraction step. And a synthesizing step of synthesizing a stroboscopic image based on the extracted subject region and the background region of the image selected in the selecting step.

  The invention according to claim 9 is an extraction function for extracting a subject area of a moving object from a plurality of frames of images, a selection function for selecting an image with the smallest blur among the plurality of frames, and It is a program for realizing a composition function for synthesizing a strobe image based on a subject area extracted by an extraction function and a background area of an image selected by the selection function.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a program capable of selecting a background image with small blur.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a schematic configuration of a digital camera 1 according to the present embodiment.
As shown in FIG. 1, the digital camera 1 includes an image data generation unit 21, a data processing unit 22, a user interface unit 23, and an acceleration sensor 24.

The image data generation unit 21 includes an optical lens unit 101 and an image sensor 102, and has a function of photographing a subject.
The optical lens unit 101 is configured by a lens that collects light for photographing a subject, and includes a peripheral circuit for adjusting camera setting parameters such as focus, exposure, and white balance.
The image sensor 102 is configured by, for example, a CCD (Charge Coupled Device) or the like, and captures an image formed by the optical lens unit 101 condensing light as digitized image data (image frame). The captured image data is temporarily stored in the memory 201 of the data processing unit 22.

  Note that the image data generation unit 21 can perform low-resolution image shooting (preview shooting) and high-resolution image shooting. Low-resolution image shooting, for example, has a low image resolution of about XGA (Extended Graphics Array: 1024 × 768 dots), but can capture moving images and read images at a speed of 30 fps (frames / second). On the other hand, in high-resolution image capturing, for example, image capturing can be performed with the maximum number of pixels that can be captured (for example, 4 million pixels for a camera with 4 million pixels), but the image reading speed is slow. .

The data processing unit 22 includes a memory 201, a video output unit 202, an image processing unit 203, a control unit 204, and a program memory 207.
The memory 201 temporarily stores the image data captured by the image sensor 102 every time shooting processing is performed. The memory 201 also stores image data necessary for image processing, various flag values, threshold values, and the like. Further, the memory 201 includes a display memory area for storing and reading display image data for displaying an image.
The video output unit 202 reads the display image data stored in the display memory area of the memory 201, generates an RGB signal based on the read display image data, and uses the generated RGB signal as the user interface unit 23. Is output to the liquid crystal display unit 301. In addition, the RGB signal can be externally output via the external interface 303 of the user interface unit 23 to display an image on an external device such as a television, a PC, or a projector.
The image processing unit 203 performs predetermined image processing for image display on the image data temporarily stored in the memory 201. The image data subjected to the image processing is stored in the display memory area of the memory 201 as display image data.

The control unit 204 includes a CPU and a storage device (not shown), an AF processing unit 205, and a shake correction processing unit 206, and performs various control operations according to the program for the digital camera 1 stored in the program memory 207.
Here, the AF processing unit 205 determines the focus accuracy evaluation value of the image formed by the optical lens unit 101, for example, the contrast value and the edge amount described later of the focus lens (not shown) of the optical lens unit 101. Control the position. Further, the blur correction processing unit 206 drives a blur correction lens (not shown) of the optical lens unit 101 in a direction orthogonal to the optical axis in accordance with the blur amount of the digital camera 1 input from the acceleration sensor 24, and the image sensor 102. Control is performed so as to correct blurring of the image formed on the image.

  The program memory 207 is composed of a storage device such as a ROM (Read Only Memory) or a flash memory, for example, and stores various programs and data necessary for the operation of the control unit 204. Specifically, a shooting program for shooting a subject and generating a strobe image of the subject, a composition program for extracting a subject from a specified composite frame and combining the strobe image, and generating a pseudo background image A pseudo background image generation program, a subject determination parameter calculation program for determining a subject, a subject extraction program for extracting only a subject, a background selection program for selecting a preferable background, and the like.

The user interface unit 23 includes a liquid crystal display unit 301, an operation unit 302, an external interface 303, and an external memory 304.
The liquid crystal display unit 301 displays a subject image based on the RGB signals output from the video output unit 202. Specifically, a live view image based on a plurality of image data (image frames) generated by the image data generation unit 21, a moving image recorded in the external memory 304 during recording, or the external memory 304 is displayed. Play back and display moving images recorded in the.
Note that the liquid crystal display unit 301 may include a video memory (not shown) that temporarily stores display image data output from the video output unit 202 as appropriate.
The operation unit 302 is a function for the user to perform a predetermined operation on the digital camera 1, and outputs an operation signal corresponding to the user operation to the control unit 204. The operation unit 302 includes, for example, a shutter button, a selection determination button, a playback button, a shooting button, and the like.
The external interface 303 is a terminal for connection with an external device such as a television, a PC, or a projector, and transmits and receives data via a predetermined communication cable.
The external memory 304 is configured by, for example, a card-type nonvolatile memory (flash memory), a hard disk, and the like, and stores a plurality of image data of subject images taken by the image data generation unit 21.

  The acceleration sensor 24 physically and directly detects the movement of the digital camera 1 and inputs a measurement value indicating the angular velocity to the control unit 204.

FIG. 2 is a diagram showing a state in which flash photography is actually performed. FIG. 3 is a diagram showing a group of frames continuously photographed in FIG. 2 and an image (strobe image) obtained by synthesizing the group of frames. That is, when the subject is continuously photographed with the digital camera 1 as shown in FIG. 2, a strobe image as shown in FIG. 3B is produced.
When a moving subject is continuously photographed by the fixed digital camera 1, only the subject moves within the frame. As a result of continuous shooting, a “frame” as shown in FIG. 3A is generated. An area of the moving subject (hereinafter referred to as a moving area) in this frame is extracted by a process (photographing process shown in FIG. 5) described later, and the extracted moving areas are overlapped to obtain FIG. A composite image having the motion as shown is generated.

In order to calculate the moving region, first, the positional deviations of all the frames are corrected such that the positions of the feature points in the region without fluctuation are the same.
Then, the pixel values (pixel values) of the coordinates in all the frames that have been aligned are rearranged in order independently for each coordinate. Here, the pixel value is, for example, a total value of RGB (red (R), green (G), blue (B)) added values, YUV (luminance signal (Y), difference between luminance signal and blue component ( U) and the difference between the luminance signal and the red component (V)) are the sum of Y and UV multiplied by coefficients. If the moving image is taken continuously, the background image (the image of the part excluding the moving object) will be reflected in the largest amount, and the center value of the rearranged pixel values will be regarded as the background. be able to. Then, for example, as shown in FIG. 4, coordinates with a large difference amount obtained by subtracting the generated background image from each frame can be regarded as a subject.

However, for example, when an image used as the background area has a camera shake, the impression of the background image deteriorates the impression of the entire image. That is, when writing a background image, it is necessary to select an image with less positional deviation.
Therefore, processing performed to solve these problems will be described with reference to the flowcharts shown in FIGS.

(First embodiment)
FIG. 5 is a flowchart illustrating an example of a photographing process performed in the digital camera 1. This shooting process is realized when the control unit 204 executes a shooting program stored in the program memory 207 when the user presses the shutter button and starts shooting an image.
Specifically, when the user performs a shooting operation, that is, a shutter button is pressed, an input signal corresponding to the operation is input to the control unit 204. When receiving the input signal, the control unit 204 controls the image data generation unit 21 to perform continuous shooting of the moving subject. This continuous shooting operation is performed while the user continues to press the shutter button.

  First, as shown in FIG. 5, the captured image is temporarily stored in the memory 201 (step S101), and an evaluation value representing the focus accuracy of autofocus (AF) is stored (step S102). Steps S101 and S102 are repeated until a stop operation is performed (“N0” in step S103). On the other hand, when a stop operation is performed during continuous shooting (“Yes” in step S103), the positional deviation is corrected so that the feature points of all frames match (step S104). After that, a composition process is performed for compositing a strobe image from the captured image with the misalignment corrected (step S105). When the synthesis process is performed, the synthesized image (strobe image) is displayed (step S106), and then the synthesized image is stored in the external memory 304 (step S107), and the process ends.

FIG. 6 is a flowchart illustrating an example of the synthesis process.
As shown in FIG. 6, when an image to be combined is acquired, a pseudo background image generation process (step S121) and a subject determination parameter calculation process (step S122) described later are performed. Thereafter, subject extraction processing (step S123) is performed on all frames. Then, the image of the subject area of the last effective frame is acquired (step S124), and only the unregistered portion of the acquired image of the subject area is overwritten and registered (steps S125 and S126). Until then, only the unregistered portion of the image of the subject area is overwritten (“No” in step S127, step S128). When overwriting is completed (“Yes” in step S127), background selection processing described later is performed on unregistered coordinates that have not yet been written (step S129), and data of the frame selected for the background is written (step S129). S130). Thereby, for example, as shown in FIG. 3B, data is written in all coordinates. The composite image generated in this way is displayed as a preview on the liquid crystal display unit 301 (step S106 in FIG. 5) and stored in the external memory 304 (step S107 in FIG. 5).

そして、ステップＳ１４２で取得されたピクセル値を値に従って並び替えたときに中央に位置するピクセル値を取得し、擬似背景画像（データ）の座標（ｘ，ｙ）に登録する（ステップＳ１４４）。当該登録処理は、例えば、数２によって表すことができる。

全座標について上記処理を繰り返す（ステップＳ１４５、ステップＳ１４６）ことで、擬似背景画像を生成することができる。生成された擬似背景画像はメモリ２０１に格納される。なお、ここでは座標ごとに取得されたピクセル値の中央値を利用したが、例えば、フレーム枚数や変動対象によっては平均値を取ることで高速化が可能な場合もある。また、処理対象の画像及び生成される擬似背景画像の座標数を実際の画像サイズより縮小する事で高速化することもできる。 FIG. 7 is a flowchart illustrating an example of the pseudo background image generation process.
First, as shown in FIG. 7, the coordinates (x, y) are initialized (step S141). Next, the value (pixel value) of the same coordinate (x, y) of each frame is acquired (step S142, step S143). The pixel value can be calculated, for example, by calculating Equation 1.

Then, the pixel value located in the center when the pixel values acquired in step S142 are rearranged according to the values is acquired and registered in the coordinates (x, y) of the pseudo background image (data) (step S144). The registration process can be expressed by Equation 2, for example.

By repeating the above process for all coordinates (steps S145 and S146), a pseudo background image can be generated. The generated pseudo background image is stored in the memory 201. Although the median value of the pixel values acquired for each coordinate is used here, for example, depending on the number of frames and the object to be changed, it may be possible to increase the speed by taking an average value. In addition, the speed can be increased by reducing the number of coordinates of the image to be processed and the generated pseudo background image from the actual image size.

すべてのフレームについて座標（ｘ，ｙ）のピクセル差分値ｆｄ（ｎ，ｘ，ｙ）を算出するステップＳ１５２の処理が終了した後（ステップＳ１５３で“Ｙｅｓ”）、この同一座標（ｘ，ｙ）におけるピクセル差分値ｆｄ（ｎ，ｘ，ｙ）の標準偏差（以下、座標標準偏差ｆｓ（ｘ，ｙ））を算出して登録する（ステップＳ１５４）。当該座標標準偏差ｆｓ（ｘ，ｙ）は、例えば、数４を演算することで算出することができる。

全ての座標について上記処理を終了すると（ステップＳ１５５で“Ｙｅｓ”）、ステップＳ１５４で算出された座標標準偏差ｆｓ（ｘ，ｙ）を超えるピクセル値を抽出する（ステップＳ１５７、ステップＳ１５８）。そして、抽出された座標標準偏差ｆｓ（ｘ，ｙ）を超えるピクセル値の標準偏差（以下、変動閾値ｍｏｖｅ）を算出する（ステップＳ１５９）。当該変動閾値ｍｏｖｅは、例えば、数５を演算することで算出することができる。

こうして算出されたパラメータを利用して、例えば図４に示すように、各フレームから被写体領域を抽出する。 FIG. 8 is a flowchart illustrating an example of subject determination parameter calculation processing.
First, as shown in FIG. 8, the coordinates (x, y) are initialized (step S151). Next, the pixel value f (n, x, y) of the coordinate (x, y) of the frame n and the pixel value fb (x, y) of the coordinate (x, y) of the pseudo background image generated by the pseudo background image generation. ) And fd (n, x, y) are calculated and registered as pixel difference values (step S152). The pixel difference value can be calculated by, for example, calculating Equation 3.

After the process of step S152 for calculating the pixel difference value fd (n, x, y) of the coordinates (x, y) for all the frames is completed (“Yes” in step S153), the same coordinates (x, y) The standard deviation (hereinafter, coordinate standard deviation fs (x, y)) of the pixel difference value fd (n, x, y) is calculated and registered (step S154). The coordinate standard deviation fs (x, y) can be calculated by, for example, calculating Equation 4.

When the above processing is completed for all coordinates (“Yes” in step S155), pixel values exceeding the coordinate standard deviation fs (x, y) calculated in step S154 are extracted (steps S157 and S158). Then, a standard deviation of pixel values exceeding the extracted coordinate standard deviation fs (x, y) (hereinafter, fluctuation threshold move) is calculated (step S159). The variation threshold move can be calculated, for example, by calculating Equation 5.

Using the parameters thus calculated, for example, a subject area is extracted from each frame as shown in FIG.

その後、生成された０と１のデータに対して、連続している領域には同じ番号を振り、離れている領域には別の番号を振るラベリングを行う。そして、ラベリングした領域のうち最大の領域のみを残し（ステップＳ１７５）、さらに、膨張（ステップＳ１７６）、穴埋め（ステップＳ１７７）、収縮（ステップＳ１７８）といった欠損を補完するための処理を行う。また、この際に、生成した領域情報の周辺部に透過パラメータを付加させることで、より自然な上書きが可能となる。当該被写体抽出を全フレームに対して行うことで、全フレームの領域マスクデータが生成されることとなる（ステップＳ１７９、ステップＳ１８０で“Ｎｏ”）。 FIG. 9 is a flowchart illustrating an example of subject extraction processing.
First, as shown in FIG. 9, the pixel difference value fd (n, x, y), which is the difference between the pixel values of the image of each frame and the pseudo background image, is a coordinate (x , Y) is 1 and data fp (n, x, y) is generated with coordinates (x, y) smaller than the value of the fluctuation threshold move being 0 (steps S171 to S174). The generated data fp (n, x, y) can be calculated, for example, by calculating Equation 6.

Thereafter, the generated 0 and 1 data are labeled by assigning the same number to consecutive regions and assigning another number to distant regions. Then, only the largest area among the labeled areas is left (step S175), and further, a process for complementing a defect such as expansion (step S176), hole filling (step S177), and contraction (step S178) is performed. At this time, by adding a transmission parameter to the periphery of the generated region information, more natural overwriting can be performed. By performing the subject extraction for all the frames, the area mask data for all the frames is generated (“No” in Step S179 and Step S180).

  In this embodiment, binary (0 and 1) area mask data is generated. However, there is a case where the difference from the pseudo background image cannot be expressed clearly or not (binarization cannot be performed). Alternatively, multi-value region mask data having steps may be generated based on a difference from a predetermined threshold value.

全座標について上記処理を終了すると（ステップＳ１９５で“Ｙｅｓ”）、ステップＳ１９４で加算した背景領域の高周波成分を、ステップＳ１９３で求めた背景領域の座標数で除することで平均化する（ステップＳ１９６）。このように、フレームごとの平均化された高周波成分を算出する。平均化された高周波成分が最大である場合（ステップＳ１９７で“Ｙｅｓ”）は、平均化された高周波成分が最大であるフレームを背景フレームとして更新する（ステップＳ１９８）。
すべてのフレームについて上記処理を繰り返す（ステップＳ１９９で“Ｎｏ”）ことで、背景領域の高周波成分が最も大きいフレームを選択することが可能となる。そして、選択されたフレームデータを利用して、未登録座標の書き込みを行う（図６のステップＳ１３０）。
なお、本実施形態では数７にあるようなロバーツフィルタを用いたが、ソーベル等の微分フィルタを用いてもよい。 FIG. 10 is a flowchart illustrating an example of the background selection process.
First, as shown in FIG. 10, the number of background coordinates and the high frequency component are cleared (step S191). Next, it is determined whether the coordinate position in the frame is the background (background area) based on the area mask data of each frame generated in step S179 of FIG. 9 (step S192), and the coordinate position is the background area. If it is determined (“Yes” in step S192), the number of background coordinates is updated (+1) (step S193), and a high frequency component at the coordinate position is calculated and added (step S194). The high frequency component can be calculated, for example, by calculating Equation 7.

When the above processing is completed for all coordinates (“Yes” in step S195), the high frequency components of the background area added in step S194 are averaged by dividing by the number of coordinates of the background area obtained in step S193 (step S196). ). In this way, the averaged high frequency component for each frame is calculated. If the averaged high-frequency component is the maximum (“Yes” in step S197), the frame having the maximum averaged high-frequency component is updated as the background frame (step S198).
By repeating the above process for all frames (“No” in step S199), it becomes possible to select a frame having the largest high-frequency component in the background region. Then, unregistered coordinates are written using the selected frame data (step S130 in FIG. 6).
In this embodiment, the Roberts filter as shown in Equation 7 is used, but a differential filter such as Sobel may be used.

  As described above, in the first embodiment, the high-frequency components of the region regarded as the background in the continuously shot image are compared, and the frame having the maximum high-frequency component is adopted as the background frame, so that the background region is not blurred. A small background image can be selected. In addition, it is possible to reduce background blur that occupies most of the combined strobe image as much as possible, and as a result, it is possible to make the composite image an image with less blur.

In the first embodiment, the frame having the largest high-frequency component in the background region is used as the background frame. However, for example, when an edge histogram is used for camera autofocus, the same can be performed.
In other words, the camera's autofocus is considered to be in focus at the lens position where the evaluation value (high-frequency component value) is large. If the evaluation value can be obtained continuously in each frame, the high-frequency component is increased using that value. It is possible to select a background image with less blur. Hereinafter, background selection processing using autofocus will be described with reference to the flowchart of FIG.

(Second Embodiment)
FIG. 11 is a flowchart illustrating an example of background selection processing (background selection processing 2).
First, as shown in FIG. 11, ef [n] representing an autofocus (AF) edge amount that is a high-frequency component in the target frame n is acquired (step S211). Here, the edge amount is the sum of differences between adjacent or neighboring pixel values in the image. If the edge amount of the target frame acquired this time in step S211 is larger than the edge amount of the already acquired frame (“Yes” in step S212), the target frame is updated as a background frame (step S213).
By repeating the above process for all frames (“No” in step S214), it is possible to select a frame having the largest high-frequency component in the background region.

As described above, in the second embodiment, the high-frequency component (edge amount) of the region regarded as the background in the continuously shot image is compared, and the frame having the maximum edge amount is employed as the background frame, thereby allowing the background It is possible to select a background image with less blur in the area. In addition, it is possible to reduce background blur that occupies most of the combined strobe image as much as possible, and as a result, it is possible to make the composite image an image with less blur.
In addition, since the focus accuracy is determined at the time of shooting, the optimum amount of background can be obtained by comparing the edge amount ef [n] representing the focus accuracy in the target frame n by the number of frames and selecting the frame having the maximum edge amount. Selection can be made.

  When the focus accuracy is evaluated by the contrast value, the contrast value may be acquired in step S211 and it may be determined whether the contrast value is large in step S212.

当該ずれ量はカメラの変動量であるため、すべてのフレームの中で値が最小となっているものを手振れの発生していないフレームとみなすことができる。すなわち、平行移動成分が小さい画像を背景として選択することで、ぶれの小さい画像を選択することができる。 (Third embodiment)
FIG. 12 is a flowchart illustrating an example of background selection processing (background selection processing 3). In the background selection process 3, an image with the smallest shift between frames is selected as the background. Note that, at the stage of the background selection process 3 in the composition process, since the alignment is completed as shown in steps S104 and S105 of FIG. 5, the moving component of the deformation matrix as shown in Equation 8 used for the alignment is used. From the (positioning parameter), it is possible to calculate the amount of shift (translation component) between frames.

Since the amount of shift is a variation amount of the camera, a frame having the smallest value among all the frames can be regarded as a frame in which no camera shake occurs. That is, by selecting an image having a small parallel movement component as the background, an image with small blur can be selected.

ステップＳ２２１で今回算出した対象フレームの平均移動量が、既に算出したフレームの平行移動量と比較して小さい場合（ステップＳ２２２で“Ｙｅｓ”）は、当該対象フレームを背景フレームとして更新する（ステップＳ２２３）。すべてのフレームについて上記処理を繰り返す（ステップＳ２２４で“Ｎｏ”）ことで、平均移動量が最も小さいフレームを選択することができる。 First, as shown in FIG. 12, the parallel movement amount d [n] in the target frame n is calculated, for example, by calculating Equation 9 (step S221). Here, it is assumed that the position correction calculation parameters shown in Equation 8 have already been calculated.

If the average movement amount of the target frame calculated this time in step S221 is smaller than the already calculated parallel movement amount of the frame (“Yes” in step S222), the target frame is updated as a background frame (step S223). ). By repeating the above process for all frames (“No” in step S224), it is possible to select a frame having the smallest average moving amount.

  For example, when the background is discriminated by the high frequency component as in the first embodiment and the second embodiment, the one having the highest high frequency component e [n] or ef [n] may be selected. When the background is determined by the amount of parallel movement as in the embodiment, the one having the smallest value d [n] is selected. That is, by selecting an image with the smallest amount of parallel movement as the background, it is possible to select an image with less blur.

  As mentioned above, although concretely demonstrated based on embodiment which concerns on this invention, this invention is not limited to the said embodiment, It can change in the range which does not deviate from the summary.

For example, in order to specify the amount of shake for each frame, it is also effective to use the measurement value of an acceleration sensor mounted on a camera in recent years.
When using the measurement value by the acceleration sensor, the movement amount (position fluctuation amount) of each frame is calculated immediately after shooting, as in the case of using autofocus. Further, the background selection process using the measurement value by the acceleration sensor can be performed using the movement amount calculated by Equation 9, as in FIG. That is, by selecting an image with the smallest amount of movement as the background, it is possible to select an image with a small amount of blur.

The high-frequency component including the amount of edge for autofocus may be the sum of the square sums of the differences between the pixel values at the diagonally adjacent positions as shown in Equation 7, or the pixels at the adjacent positions in the vertical and horizontal directions. It is good also as the sum total of the square sum of the difference of a value. In this way, if it is calculated by the difference between the pixel values at the adjacent or neighboring positions, it becomes more sensitive to fine changes, and the large edge amount means that the difference between the pixel values at the neighboring or neighboring positions is large, so that there is a contrast. It can be said that it is a clear image with little blur. In addition, if it is calculated by the difference between pixel values at distant positions, it will not respond to small changes.
Furthermore, although the difference in pixel values between adjacent or neighboring positions is used, the sum of luminance differences between neighboring or neighboring pixels may be used. Further, it may be the sum of chroma differences between adjacent or neighboring pixels, or the sum of brightness differences between neighboring or neighboring pixels. Furthermore, the sum of differences of RGB (red (R), green (G), blue (B)) between adjacent or neighboring pixels, or YUV (luminance signal (Y), luminance signal between neighboring or neighboring pixels). And the sum of the difference between the luminance signal and the red component (V)), and CMYK (cyan (C), magenta (M), yellow (Y ) And black (Bk)).
In other words, the sum of differences between adjacent or neighboring elements may be used as long as they represent an image.

  Note that the image processing apparatus according to the present invention can be provided as an image processing apparatus provided with the configuration and functions according to the present invention in advance, like the digital camera 1 exemplified in the above embodiment, and each function of the control unit 204. By applying a program that realizes the same function, an existing image processing apparatus can be made to function as the image processing apparatus according to the present invention.

  The application method of such a program is arbitrary. For example, the program can be applied by being stored in a storage medium such as a CD-ROM or a memory card, or can be applied via a communication medium such as the Internet.

1 is a block diagram illustrating a schematic configuration of a digital camera 1 according to an embodiment. It is a figure showing a situation where flash photography is actually performed. FIG. 3 is a diagram illustrating a group of frames continuously photographed in FIG. 2 and an image (strobe image) obtained by synthesizing the group of frames. It is a figure showing signs that a subject image is extracted by subtracting a background image from a frame to be synthesized. 3 is a flowchart illustrating an example of a photographing process performed in the digital camera 1; It is the flowchart shown about an example of the synthetic | combination process. It is the flowchart shown about an example of the pseudo | simulation background image generation process. 5 is a flowchart illustrating an example of subject determination parameter calculation processing. 5 is a flowchart illustrating an example of subject extraction processing. It is the flowchart shown about an example of the background selection process. It is the flowchart shown about an example (background selection process 2) of a background selection process. It is the flowchart shown about an example (background selection process 3) of the background selection process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 21 ... Image data generation part, 101 ... Optical lens part, 102 ... Image sensor, 22 ... Data processing part, 201 ... Memory, 202 ... Video Output unit 203... Image processing unit 204... Control unit 205 205 AF processing unit 206 blur correction processing unit 207 program memory 23 user interface unit 301 ... Liquid crystal display unit 302 ... Operation unit 303 ... External interface 304 ... External memory 24 ... Acceleration sensor

Claims (9)

Extraction means for extracting a subject area of a moving subject from a plurality of frames of images;
Selecting means for selecting an image with the smallest blur among the images of the plurality of frames;
Synthesizing means for synthesizing a strobe image based on the subject area extracted by the extracting means and the background area of the image selected by the selecting means;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the selection unit calculates a high frequency component of the background region for each of the images of the plurality of frames and selects an image having the largest calculated high frequency component. .   The image processing apparatus according to claim 2, wherein the high-frequency component is a total sum of squares of differences between pixel values at obliquely adjacent positions.   The image processing apparatus according to claim 2, wherein the high-frequency component is a sum of square sums of differences between pixel values at positions adjacent to each other in the vertical and horizontal directions.   The selection unit calculates a position variation amount between the images based on a result of alignment of the images of the plurality of frames, and selects an image having the smallest calculated position variation amount. Item 8. The image processing apparatus according to Item 1. In addition to continuously shooting images of a plurality of frames, the camera further includes shooting means for acquiring an evaluation value of focus accuracy by autofocus of each image,
The image processing apparatus according to claim 1, wherein the selection unit selects an image with the smallest blur based on the evaluation value of the focus accuracy acquired by the photographing unit. Photographing means for continuously photographing images of a plurality of frames;
Detecting means for detecting the amount of shake generated in the device,
The selection unit calculates a position variation amount between the images of the plurality of frames based on a blur amount detected by the detection unit, and selects an image having the smallest calculated position variation amount. The image processing apparatus according to claim 1. An extraction step of extracting a subject area of a moving subject from a plurality of frames of images;
A selection step of selecting an image with the smallest blur among the images of the plurality of frames;
A synthesizing step of synthesizing a stroboscopic image based on the subject region extracted by the extracting step and the background region of the image selected by the selecting step;
An image processing method comprising: On the computer,
An extraction function for extracting a subject area of a moving subject from a multi-frame image;
A selection function for selecting an image with the smallest blur among the images of the plurality of frames;
A synthesis function for synthesizing a strobe image based on a subject area extracted by the extraction function and a background area of the image selected by the selection function;
A program to realize

Images