JP2009017223A - Imaging device, image processing device, and their image processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To appropriately correct blurring included in an image. <P>SOLUTION: In an imaging device, an object position determination unit 320 decides a position of an object included in a taken image which is an image taken by an imaging unit 110 using a selection input of the object in an operation reception unit 250. An object movement amount detection unit 330 detects a movement amount of the object included in the taken image. An object region extraction unit 360 extracts the object region. An object movement prediction amount calculation unit 350 predicts a movement amount of the object based on the detected movement amount of the object. A blurring correction unit 381 corrects the taken image and makes a corrected image based on the movement amount of the object. An image composition unit 383 composes the corrected image in the object region extracted by an object image extraction unit 382 on the object region of the taken image. A region boundary correction unit 384 changes and corrects pixel values of the composite image according to a distance from a boundary in a perimeter region of the boundary of a background and the object in the composite image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus that corrects blurring of a captured image, an image processing apparatus, an image processing method therefor, and a program that causes a computer to execute the method.

  In recent years, an imaging apparatus such as a digital still camera that captures an image of a subject such as a person and records it as image data has rapidly spread, and the performance of these imaging apparatuses is increasing. In addition, downsizing of these imaging apparatuses is also progressing, and various imaging apparatuses that can be carried have been proposed. Accordingly, the user can take the image pickup device for traveling or the like and take pictures at various places.

  When shooting with these imaging devices held by hand, two types of blurring, namely blurring of the imaging device itself and blurring of the subject, mainly occur.

  As a method for detecting these shakes, a detection method such as a shake detection method of an imaging apparatus itself using a vibration sensor, a speed sensor, an acceleration sensor, a gravity sensor, or a motion vector detection method using image processing is mainly known. Yes.

  Using the blur component detected by these detection methods, the blur in the shot image can be corrected by moving the position of the shot image in a direction to cancel the blur component.

For example, the image obtained by the image sensor is sampled based on the sampled camera shake detection information by sampling the camera shake detection information supplied from the camera shake sensor at the sample timing corresponding to the shutter speed in the sampling signal supplied from the timing generator. An imaging device that corrects camera shake by controlling a signal readout position has been proposed (see, for example, Patent Document 1).
Japanese Patent Laid-Open No. 05-316404 (FIG. 1)

  In the above-described conventional technology, the blur correction process is uniformly performed regardless of the movement of the subject included in the captured image. As a result, it is possible to correct blur such as camera shake as the entire captured image.

  In general, as a situation where an imaging device is used, for example, it is conceivable to photograph a person playing on a moving vehicle in an amusement park, or to photograph a turtle swimming in an aquarium tank. As described above, when a moving object is photographed by the imaging apparatus, a so-called subject blur is often generated in which the object included in the photographed image is blurred.

  However, in these correction processes, since the background and the person included in the photographed image are treated uniformly, the blur component remains in the person, the face, eyes, nose, mouth, etc. that the photographer aimed at as the subject. It is possible to become.

  Therefore, when photographing a moving object, it is conceivable to perform so-called panning, in which the lens of the imaging apparatus is moved in accordance with the moving object. However, since panning is said to be a difficult technique, there is a problem that it is difficult for beginners and the like to shoot. Moreover, although the blurring of the subject can be reduced by performing the panning, the background is often blurred, and it is difficult to meet the request of a person who wants to make the entire photograph an image with high sharpness.

  For this reason, for example, in the case of shooting a moving turtle, if the blur included in the image is corrected appropriately and the turtle's background blur can be reduced along with the moving turtle's blur, It is considered that an appropriate image according to the user's preference can be provided.

  Accordingly, an object of the present invention is to appropriately correct blur included in an image.

  The present invention has been made in order to solve the above-mentioned problems. The first aspect of the present invention is to detect an amount of motion of a predetermined area included in the imaging data and imaging means for converting incident light into imaging data. Motion amount detecting means for performing correction, correcting means for correcting the predetermined area of the imaged data based on the amount of motion to obtain a corrected image, and combining means for combining the corrected image with the predetermined area of the imaged data And an image processing method thereof, and a program for causing a computer to execute the method. As a result, the amount of motion of a predetermined area included in the imaging data is detected, the predetermined area of the imaging data is corrected based on the amount of movement to obtain a corrected image, and the corrected image is combined with the predetermined area of the imaging data. The effect of doing.

  Further, in the first aspect, the image processing device further includes an image storage unit that stores the imaging data, and a storage control unit that stores the detected motion amount in the image storage unit in association with the image of the predetermined area. The correction unit corrects the predetermined area of the imaging data stored in the image storage unit based on the amount of motion stored in the image storage unit corresponding to the imaging data, and corrects the corrected image. And the said synthetic | combination means can synthesize | combine the said correction | amendment image to the said predetermined area | region of the imaging data memorize | stored in the said image storage means. Thus, the detected amount of motion is stored in the image storage unit in association with the image of the predetermined region, and the predetermined region of the imaging data stored in the image storage unit is converted into the amount of motion corresponding to the imaging data. Thus, the corrected image is corrected to produce a corrected image, and the corrected image is combined with a predetermined area of the imaged data. In this case, the storage control means can store the area information related to the predetermined area in the image storage means in association with the image of the predetermined area. This brings about the effect that the area information relating to the predetermined area is stored in the image storage means in association with the image of this area. In this case, the area information can include a reduced image of an image included in the predetermined area. As a result, the reduced image of the image included in the predetermined region is stored in the image storage unit in association with the image in this region.

  Further, in the first aspect, the image processing apparatus further includes a recording operation receiving unit that receives a recording operation of an image captured by the imaging unit, and the motion amount detecting unit is configured to perform frame-by-frame processing on the image captured by the imaging unit. The amount of movement of the predetermined area is detected, and the storage control means records the image picked up by the image pickup means when the recording operation is accepted and is detected when the recording operation is accepted. The amount of motion of the predetermined area can be recorded in the image storage means in association with the image. As a result, the motion amount of a predetermined area is detected for each frame of the captured image, and the image captured when the recording operation is accepted and the predetermined amount detected when the recording operation is accepted. This brings about the effect that the movement amount of the area is associated and recorded in the image storage means. In this case, the apparatus further includes a motion amount predicting unit that predicts a new motion amount based on the motion amount of the predetermined area detected when the recording operation is received, and the storage control unit includes: When the recording operation is accepted, the image picked up by the image pickup means can be stored, and the new motion amount can be associated with the image and stored in the image storage means. Accordingly, a new motion amount is predicted based on the motion amount of the predetermined area detected when the recording operation is accepted, and the image captured when the recording operation is accepted and the new motion amount Are associated and stored in the image storage means.

  In the first aspect, a display unit that displays the captured image, a selection reception unit that receives a selection input of a specific object included in the displayed image, and a region of the selected object. The image processing apparatus may further include predetermined area extraction means for extracting as the predetermined area. As a result, the captured image is displayed, and when a selection input of a specific object included in the displayed image is received, an area of the selected object is extracted as a predetermined area.

  Further, in the first aspect, boundary correction for changing a pixel value of the synthesized image according to a distance from the boundary in a peripheral region of the boundary of the image included in the predetermined area in the synthesized image. Means may further be provided. Thereby, in the peripheral region of the boundary of the image included in the predetermined region in the combined image, there is an effect that the pixel value of the combined image is changed according to the distance from the boundary.

  According to a second aspect of the present invention, there is provided input means for inputting imaging data obtained by converting incident light and a motion amount of a predetermined area included in the imaging data, and the imaging data based on the motion amount. An image processing apparatus and an image processing method therefor, comprising: a correcting unit that corrects the predetermined region to obtain a corrected image; and a combining unit that combines the corrected image with the predetermined region of the imaging data. And a program for causing a computer to execute the method. As a result, the imaging data and the amount of motion of a predetermined area included therein are input, the predetermined area of the imaging data is corrected based on the amount of motion to obtain a corrected image, and this corrected image is used as the predetermined amount of the imaging data. It brings about the effect of synthesizing in the region of.

  According to the present invention, it is possible to achieve an excellent effect that blur included in an image can be corrected appropriately.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating a functional configuration example of an imaging apparatus 100 according to an embodiment of the present invention.

  The imaging apparatus 100 includes an imaging unit 110, a CPU 120, a RAM 130, a camera signal processing engine 140, an object detection engine 150, an object region extraction engine 160, a motion amount detection engine 180, and a shake correction engine 190. The image display engine 200, the display unit 210, the media controller 220, the captured image storage unit 230, the operation input controller 240, the operation reception unit 250, and the data bus 260.

  The imaging unit 110 includes an optical system and an imaging device, and photoelectrically converts an image captured through the optical system into an image signal, and outputs the photoelectrically converted image signal to the camera signal processing engine 140.

  A CPU (Central Processing Unit) 120 controls the entire imaging apparatus 100 based on various control programs.

  A RAM (Random Access Memory) 130 is a memory that stores various data, and includes various image storage areas for image processing.

  The camera signal processing engine 140 performs appropriate signal processing on the image signal output from the imaging unit 110 and outputs it to the data bus 260.

  The target object detection engine 150 detects a target object that is a specific object that is a target of the subject from the input image. For example, the object detection engine 150 can detect a human face included in the input image as an object.

  The object area extraction engine 160 extracts an area in the input image for the object determined by the user or the object detected by the object detection engine 150.

  The motion amount detection engine 180 detects a motion amount in an input image with respect to an object determined by a user or an object detected by the object detection engine 150.

  The blur correction engine 190 includes an area extracted by the object area extraction engine 160 and a motion detected by the motion amount detection engine 180 for the object determined by the user or the object detected by the object detection engine 150. The blur component is corrected based on the amount.

  The image display engine 200 outputs various images to the display unit 210 based on control from the CPU 120.

  The display unit 210 is an image display device that displays images corresponding to various image data output from the image display engine 200. For example, a liquid crystal display (LCD) or an electronic viewfinder (EVF) is used as the image display device.

  The media controller 220 outputs the image data input from the captured image storage unit 230 to the RAM 130 or the like via the data bus 260 and outputs various image data to the captured image storage unit 230 for recording.

  The captured image storage unit 230 is an information recording medium for recording various data such as JPEG (Joint Photographic Experts Group) format image data.

  The operation input controller 240 outputs the operation input received by the operation receiving unit 250 to the CPU 120 via the data bus 260.

  The operation accepting unit 250 includes various operation members such as up / down / left / right keys for designating an object in the image displayed on the display unit 210 and a shutter button to be pressed when taking an image. Is received, the content of the accepted operation input is output to the operation input controller 240. In addition, about the display part 210 and the operation reception part 250, you may make it comprise these one part or all as a touch panel integrally.

  FIG. 2 is a block diagram illustrating a functional configuration example of the imaging apparatus 100 according to the embodiment of the present invention. Here, an example of a functional configuration in the case of detecting the amount of movement and area information of the target object from the captured image, and storing the captured image and the amount of movement of the target object and area information in association with each other in the captured image storage unit 230. An example of a functional configuration in the case of correcting blurring of a subject using the amount of movement of a target object and region information for a captured image stored in a captured image storage unit 230 will be described.

  The imaging apparatus 100 includes an imaging unit 110, a display unit 210, a captured image storage unit 230, an operation reception unit 250, an image input unit 310, an object position determination unit 320, and an object motion amount detection unit 330. , An object position update unit 340, an object motion prediction amount calculation unit 350, an object region extraction unit 360, a storage control unit 370, an object shake correction unit 380, and a display control unit 390.

  The imaging unit 110 corresponds to the imaging unit 110 illustrated in FIG. 1, and outputs a photoelectrically converted image signal to the image input unit 310.

  The operation reception unit 250 corresponds to the operation reception unit 250 illustrated in FIG. 1, and an operation input for selecting an object included in the image displayed on the display unit 210 is input to the object position determination unit 320. Output. In addition, when the user presses the shutter button, the operation reception unit 250 outputs an operation input to the storage control unit 370 to store the image captured at the time of the pressing. Further, the operation reception unit 250 performs an operation input for displaying the image captured by the imaging unit 110 on the display unit 210 or an operation for displaying the image corrected by the object blur correction unit 380 on the display unit 210. The input is output to the display control unit 390.

  The image input unit 310 inputs the image signal output from the imaging unit 110, and an image corresponding to the input image signal is input to the object position determination unit 320 and the object region extraction unit 360 for each frame. The data is output to the storage control unit 370 and the display control unit 390.

  The object position determination unit 320 determines at least one object as an object in accordance with an operation input from the operation reception unit 250 among subjects included in the image output from the image input unit 310. The determined position of the object is output to the object movement amount detection unit 330 and the object region extraction unit 360. In addition, the object position determination unit 320 determines one frame based on the update information of the position of the object output from the object position update unit 340 after the object is determined according to the operation input from the operation reception unit 250. The position of the object is updated every time, and the updated position of the object is output to the object movement amount detection unit 330 and the object region extraction unit 360. The determination of the object by the object position determination unit 320 will be described in detail with reference to FIG. Further, the object position determination unit 320 may determine the object detected by the object detection engine 150.

  The object movement amount detection unit 330 detects the amount of movement of the object for each frame based on the position of the object output from the object position determination unit 320. This is output to the object position update unit 340 and the target object motion prediction amount calculation unit 350. The detection of the amount of movement will be described in detail with reference to FIGS. 5 (a) and 5 (b).

  The object position update unit 340 creates update information of the position of the object based on the amount of movement of the object output from the object movement amount detection unit 330, and updates the position information of the created object. This is output to the object position determination unit 320.

  The target object motion prediction amount calculation unit 350 calculates a motion amount predicted for the target object as a motion prediction amount based on the target motion amount output from the target object motion amount detection unit 330. The calculated motion prediction amount is output to the storage control unit 370. Note that the calculation of the motion prediction amount will be described in detail with reference to FIGS. 5B and 5C.

  The object area extraction unit 360 sets the area surrounding the object determined by the object position determination unit 320 in the image output from the image input unit 310 to the position of the object output from the object position determination unit 320. Based on this, information relating to the area surrounding the extracted object is output to the storage control unit 370. The information regarding the area surrounding the object is, for example, the coordinates, shape, reduced image, and the like of the area surrounding the object. The area surrounding the object will be described in detail with reference to FIG.

  The storage control unit 370 extracts the image output from the image input unit 310, the motion prediction amount output from the target motion prediction amount calculation unit 350 corresponding to the image, and the target region extraction corresponding to the image. The information related to the region surrounding the object output from the unit 360 is associated with the information and stored in the captured image storage unit 230.

  The captured image storage unit 230 is a storage unit that associates and stores a captured image, a motion prediction amount corresponding to the image, and information regarding a region surrounding the object corresponding to the image, and stores the stored information. The image is output to the object blur correction unit 380. Note that captured images and the like recorded in the captured image storage unit 230 will be described in detail with reference to FIG.

  The object blur correction unit 380 includes a blur correction unit 381, an object image extraction unit 382, an image composition unit 383, and a region boundary correction unit 384.

  The shake correction unit 381 corrects the shake of the captured image stored in the captured image storage unit 230 based on the amount of movement of the object stored in association with the image. The blur correction image is output to the object image extraction unit 382. This blur correction will be described in detail with reference to FIG.

  The object image extraction unit 382 is configured to detect, from the shake correction image, the object in the area based on the information about the area surrounding the object stored in association with the image of the shake correction image output from the shake correction unit 381. An object image is extracted, and the extracted object image is output to the image composition unit 383. This object image extraction will be described in detail with reference to FIG.

  The image composition unit 383 selects the object image output from the object image extraction unit 382 based on the information related to the area surrounding the object stored in association with the captured image stored in the captured image storage unit 230. Then, the image is synthesized with the area surrounding the object in the captured image, and the synthesized image is output to the area boundary correction unit 384. This object image composition will be described in detail with reference to FIG.

  The region boundary correction unit 384 performs boundary correction processing at the region boundary of the target image combined with the captured image, and outputs the target blur correction image subjected to the boundary correction processing to the display control unit 390. This boundary correction process will be described in detail with reference to FIGS.

  The display control unit 390 causes the display unit 210 to display the image output from the image input unit 310 or the object blur correction image output from the region boundary correction unit 384 in response to an operation input from the operation reception unit 250. Control is executed.

  The display unit 210 displays the image output from the image input unit 310 or the object blur correction image output from the region boundary correction unit 384 based on the control of the display control unit 390.

  Next, an object determination method for determining an object included in an image captured by the imaging unit 110 will be described in detail with reference to the drawings.

  FIG. 3 is a diagram illustrating a display example when an image captured by the imaging unit 110 is displayed on the display unit 210. FIG. 3A is a diagram showing an image 400 including an AF area mark 402 when a turtle 401 swimming in an aquarium tank is displayed on the display unit 210. FIG. FIG. 3C is a diagram showing an image 405 including a cursor 406 when a turtle 401 swimming in an aquarium tank is displayed on the display unit 210. FIG. 4 is a diagram illustrating an image 410 when 411 is displayed on the display unit 210. FIG.

  When photographing an object using the imaging apparatus 100, the user points the lens of the imaging unit 110 toward the object and looks at a finder (not shown) or the display unit 210 while the object has an angle of view. Try to be within range. When the object is within the range of the angle of view, the object is displayed on the display unit 210. For example, when the object is a turtle 401, as shown in FIGS. 3A and 3B, the turtle 401 is displayed on the display unit 210, and when the object is a person 411, FIG. As shown in (c), the person 411 is displayed on the display unit 210. Thus, when the target object is included in the range of the angle of view, the position of the target object can be determined.

  Here, an image 400 shown in FIG. 3A includes an AF area mark 402 indicating an AF (Auto-Focus) area used at the time of autofocusing together with the turtle 401 as the object. Note that the AF area mark 402 corresponds to the position of the focus area during autofocus. In addition, the image 405 illustrated in FIG. 3B includes a cursor 406 that moves in response to an operation input from the operation reception unit 250 together with the turtle 401 that is the object. Further, the image shown in FIG. 3C includes a face area mark 413 indicating the area of the face 412 of the person 411 detected by the object detection engine 150 together with the person 411 that is the object.

  As an object determination method, for example, an AF area used during autofocus can be determined as the position of the object. For example, as shown in FIG. 3A, the position of the turtle 401 to which the AF area mark 402 is attached is determined as the position of the object.

  Further, as a method for determining an object, for example, the position of the object can be determined by an operation input from the user in the operation reception unit 250. For example, as shown in FIG. 3B, the cursor 406 is moved so as to overlap the turtle 401 by an operation input from the operation receiving unit 250, and a determination button (not shown) Z)). Thereby, the position of the turtle 401 which is a user's target object can be determined (designated). When the display unit 210 and the operation receiving unit 250 are integrally configured as a touch panel, the target can be determined by the user pressing the position of the target displayed on the patch panel. .

  Furthermore, as a method for determining an object, for example, the position of a human face detected by the object detection engine 150 can be determined as the position of the object. For example, as shown in FIG. 3C, the position of the face 412 of the person 411 to which the face area mark 413 is attached is determined as the position of the object. In addition to the human face, a detection engine that detects the face of an animal such as a pet may be used.

  In addition, you may make it display the target object position mark which shows the position of the determined target object on the display part 210 so that it may attach | subject to the determined target object.

  Next, a region extraction method for extracting the determined region of the object will be described in detail with reference to the drawings.

  FIG. 4 is a diagram illustrating an outline of a region of an object determined from images captured in the imaging unit 110. 4A is a diagram showing subject area information 420 in the images 400 and 405 shown in FIGS. 3A and 3B, and FIG. 4B is a diagram in the image 410 shown in FIG. 3C. 5 is a diagram showing subject area information 430. FIG.

  When an object is determined from images captured by the imaging unit 110, the image captured by the imaging unit 110 is divided into an area of the object and an area other than the object. , The region of the object is extracted. For example, when the turtle 401 shown in FIGS. 3 (a) and 3 (b) is determined as an object, as shown in FIG. 4 (a), an image 400 shown in FIGS. 3 (a) and 3 (b) and In the subject area information 420 corresponding to 405, an object area 421 indicating the area of the turtle 401 is extracted. When the face 412 of the person 411 shown in FIG. 3C is determined as the target object, as shown in FIG. 4B, subject area information corresponding to the image 410 shown in FIG. Among the objects 430, the object area 431 indicating the area of the person 411 is extracted.

  Here, the region extraction of the object will be described. As an object region extraction method, for example, the size of an AF area used during autofocus can be extracted as an object region as it is. For example, as shown in FIG. 3A, the range of the AF area mark 402 attached to the turtle 401 is extracted as the object area.

  In addition, as an object region extraction method, for example, a human face region detected by the object detection engine 150 can be extracted as a target region. For example, as shown in FIG. 3C, the range of the face area mark 413 attached to the face 412 of the person 411 is extracted as the area of the object.

  Further, as an object region extraction method, for example, an image currently captured by the imaging unit 110 can be divided into regions that include the determined position of the object.

  In this area division, the determined shape of the target object is detected from an image one frame before the image currently captured by the imaging unit 110, and the target area is determined based on this shape. As a method for determining such a region, for example, a method of performing region division using a decision tree is known (Richard Nock: “Fast and Reliable Color Region Merging inspired by Decision Tree Pruning”, IEEE International Conference on Computer Vision and Pattern Recognition, pp.271-276, IEEE CS press, 2001).

  The subject area information 420 shown in FIG. 4A is obtained by dividing an area for each related area in the images 400 and 405 shown in FIGS. 3A and 3B. This is the area of the turtle 401 that is the determined object. Here, each divided area is shown by color.

  The subject area information 430 shown in FIG. 4B is obtained by dividing an area for each related area in the image 410 shown in FIG. 3C, and the object area 431 is a determined object. This is an image area of a person 411. Here, the object region 431 is indicated by oblique lines.

  When the object region determined for the image captured in the imaging unit 110 is extracted, as shown in FIG. 4A or 4B, the shape of the extracted object region is displayed on the display unit. You may make it display on the one part area | region of 210. FIG. Further, the image may be displayed so as to be superimposed on the image captured in the imaging unit 110.

  Next, a motion amount detection method for detecting the determined motion amount of the object will be described in detail with reference to the drawings.

  FIG. 5 is a diagram illustrating an outline of the transition of the movement of the object determined from the images captured in the imaging unit 110. FIGS. 5A to 5C are diagrams showing simplified images 440, 450, and 460 of the image 400 or 405 shown in FIG. 3A or 3B, and the turtle included in the image 400 or 405 is shown. It is a figure which shows the outline of the transition of the turtle 441 corresponding to 401. FIG.

  When an object is determined from images captured in the imaging unit 110, the object region of the determined object is updated until an operation input for pressing the shutter is performed in the operation reception unit 250. And the detection of the movement amount of the object is continuously performed.

  As a motion amount detection method for detecting a motion amount of an object, for example, the motion amount of the object region is detected based on a current image acquired from the imaging unit 110 and an image one frame before this image. The method can be used.

For example, by using a pattern detection method using SAD (Sum of Absolute Differences) as a similarity, which part in the current image is most similar to the image in the previous frame and the image pattern in the object area. A method of detecting the amount of movement of the object region can be used. For example, using the following equation 1 _to calculate the _R SAD (u, v).

Here, A indicates a set of pixels included in the object area, T (x, y) indicates the pixel value of the image one frame before, and I (x, y) indicates the pixel value of the current image. . Then, as shown in the following Expression 2, (u, v) that minimizes R _SAD (u, v) calculated using Expression 1 is set as a motion vector V (u, v) of the object area. be able to. Here, the motion vector is an example of a motion amount.
Motion vector of object region V (u, v) = min {R _SAD (u, v)} (Expression 2)

  For example, an image 450 illustrated in FIG. 5B is an image currently captured by the imaging unit 110, and an image 440 illustrated in FIG. 5A is an image one frame before the image 450. In this case, a predetermined position in the turtle 451 included in the image 450 is set as a target position 452, and a position corresponding to the target position 452 in the turtle 441 included in the image 440 is set as a target position 442. In addition, in the image 450 shown in FIG.5 (b), the position of the turtle 441 shown to Fig.5 (a) is shown with a broken line.

  As shown in FIGS. 5A and 5B, the turtle moves from the position of the turtle 441 to the position of the turtle 451 between the imaging time of the image 440 and the imaging time of the image 450. Corresponding to the movement of the turtle, the object position 452 has moved to the object position 452. A motion vector 453 is detected as the amount of motion in this movement. This detection of the movement amount is continued until the shutter is pressed.

  When the operation reception unit 250 presses the shutter, the turtle movement amount is predicted based on the motion vector detected when the shutter is pressed, and the predicted movement amount and the turtle target area are The image is stored in the captured image storage unit 230 together with the image captured by the imaging unit 110 at the time of pressing.

  FIG. 5C is a diagram showing an outline of the transition of the turtle 451 predicted based on the motion vector detected when the operation receiving unit 250 presses the shutter. For example, an image 450 illustrated in FIG. 5B is an image captured by the imaging unit 110 when the shutter is pressed, and an image 460 illustrated in FIG. An image showing the position of Here, in the image 460 shown in FIG. 5C, the position of the turtle 461 expected as the movement destination from the position of the turtle 451 shown in FIG. Further, the predicted motion vector is set as a motion vector 463.

  As the predicted motion vector, for example, the same value as the motion vector detected when the shutter is pressed can be used. That is, the motion vector 463 can be set to the same value as the motion vector 453. Further, the average of the motion vectors detected for each frame image from the frame image when the shutter is pressed to a predetermined number of frames before can be used. Further, a predicted motion vector value is calculated based on an increase or decrease of each motion vector detected for each frame image from the frame image when the shutter is pressed to a predetermined number of frames before, and this calculated A value can be used.

  Next, an example in which the amount of movement of the object and the object area are stored in the captured image storage unit 230 in association with the captured image will be described. Here, an example in which the amount of movement of the object and the object area are recorded in the image file of the captured image will be described. Note that the amount of movement of the object and the object area may be recorded in a meta information database or the like in association with the captured image.

  FIG. 6 is a diagram showing an outline of a file structure of a still image file recorded according to the DCF (Design Rule for Camera File system) standard. DCF is a file system standard for realizing mutual use of images via a recording medium between devices such as a digital still camera and a printer, and records on a recording medium based on the Exif (Exchangeable image file format). In this case, file naming and folder structure are specified. This Exif is a standard for adding image data and camera information to an image file, and defines a format (file format) for recording the image file.

  The still image file 470 is a still image file recorded according to the DCF standard, and is composed of attached information 471 and image information 472 as shown in FIG. The image information 472 is, for example, image data captured by the imaging unit 110.

  As shown in FIG. 6B, the attached information 471 includes attribute information 473 and a maker note 474. The attribute information 473 is attribute information related to the still image file 470 and includes, for example, shooting update date / time, image size, color space information, manufacturer name, and the like.

  The manufacturer note 474 is generally an area where user-specific data is recorded, and is an extended area where each manufacturer can freely record information (TAGID = 37500, MakerNote). In this example, the amount of movement of the object and the object area are recorded in the maker note 474. As the object area, position information and size are recorded, but a reduced image of the determined object may also be recorded.

  Thus, the captured image in which the amount of movement of the object and the object region are associated is recorded in the captured image storage unit 230. When an image stored in the captured image storage unit 230 is output (displayed, printed, etc.), for example, the blur component of the subject can be corrected and output. Next, this correction method will be described in detail with reference to the drawings.

  FIG. 7 is a diagram showing an image before correction and an image after correction based on the amount of movement of the object with respect to the image recorded in the photographed image storage unit 230.

  FIG. 7A is a diagram showing an image 500 recorded in the captured image storage unit 230. For example, when a turtle swimming in an aquarium is photographed with the imaging device 100 held by hand, the background of the turtle 501 is stationary as shown in FIG. In many cases, the turtle 501 moving while swimming is blurred. For this reason, in the embodiment of the present invention, only the portion of the object to be photographed with blur is corrected.

  FIG. 7B is a diagram illustrating an image 510 obtained by performing blur correction on the image 500 illustrated in FIG. 7A based on the amount of movement of the object. For example, when blur correction is performed on the image 500 based on the turtle 501 swimming and moving in an aquarium, the blur of the turtle 511 is corrected as shown in FIG. Since the background of the turtle 511 that has been corrected is also corrected, the background of the turtle 511 often blurs. For this reason, in the embodiment of the present invention, a corrected image is created by synthesizing an image of an object part in which blurring is corrected and an image of a part other than the object included in the image before correction. The creation of a corrected image by this synthesis will be described in more detail with reference to the drawings.

  FIG. 8 is a diagram showing an outline of a correction method in a case where an image is corrected by synthesizing an image of an object part whose blurring has been corrected and an image of a part other than the object included in the image before correction. .

  FIG. 8A is a diagram illustrating an image 520 in which an object image is cut out from an image recorded in the captured image storage unit 230 based on the object region of the object. Note that the image 520 corresponds to the image 500 shown in FIG. In the image 520, the object region 521 from which the object image is cut out is shown in black. In this way, the object image is cut out from the photographed image based on the turtle object region information stored together with the photographed image.

  FIG. 8B is a diagram showing an object image 530 cut out from an image subjected to blur correction based on the object region of the object. The object image 530 corresponds to the turtle 511 included in the image 510 shown in FIG. Further, in the area corresponding to the image 510, the area other than the object image 530 is shown in black. The cut object image 530 is an image that has been subjected to blur correction processing, as shown in FIG. This blur correction process will be described in detail using Equations 3 to 14.

  FIG. 8C is a diagram illustrating an image 540 obtained by combining the object image 530 subjected to the blur correction and the image 520 obtained by cutting the object image. In this way, by synthesizing the object image 530 subjected to the blur correction and the image 520 in which no blur has occurred, it is possible to generate a beautiful image in which both the object and the background are not blurred.

  Next, a description will be given in detail of a blur correction process for correcting blur for a shot image recorded in the shot image storage unit 230.

  First, with respect to the captured image recorded in the captured image storage unit 230, the luminance component and the color component are separated, and a luminance image g (x, y) in which only the luminance component is extracted is generated. Here, when the photographed image is an RGB (G (Green: Green), R (Red: Red), B (Blue: Blue)) color image, YUV (Y (luminance signal), U (difference between luminance signal and blue component)), V (difference between luminance signal and red component)) data is calculated from RGB data for all pixels. Then, the luminance signal Y of all the pixels in the captured image is set as a luminance image g (x, y).

For example, the luminance image g (x, y) may be one in which the luminance at each pixel is expressed by an 8-bit unsigned integer.
Y = 0.299R + 0.587G + 0.114B (Formula 3)
U = −0.169R−0.332G + 0.500B + 127.5 (Formula 4)
V = 0.500R−0.419G−0.081B + 127.5 (Formula 5)

Subsequently, based on the motion vector corresponding to the photographed image, a point spread function (PSF) h (x, y) representing the blur of the object is obtained using the following Expression 6.
h (x, y) = tn · V (x, y) (Formula 6)

  Here, V (x, y) indicates a motion vector recorded in the captured image, and n indicates that the resolution of the captured image to be corrected is divided by the resolution of the image when the motion vector is detected. T is a value obtained by dividing the exposure time when the motion vector is detected by one frame time.

  Further, the point spread function h (x, y) is a so-called spatial filter shape, and when an input of 1 is input to the origin of the image coordinates, an expression in which the impulse response is expressed in the image coordinate system is used. it can.

Subsequently, as the number of taps, the width of the captured image is M, the height of the captured image is N, and the luminance image g (x, y) is converted into a two-dimensional discrete Fourier transform (two-dimensional DFT) using the following Expression 7. )

Subsequently, as the number of taps, the width of the captured image is M, the height of the captured image is N, and the point spread function h (x, y) is converted into a two-dimensional discrete Fourier transform (two-dimensional) using the following Expression 8. DFT).

Subsequently, the restoration filter H _w (x, y) is obtained using the following equation 9 from H (x, y) obtained by equation 8.

  Here, Γ is a constant. For example, when the luminance image g (x, y) is expressed by an 8-bit unsigned integer, the value of Γ is preferably about 0.01 to 0.03.

Subsequently, based on G (x, y) obtained by Expression 7 and the restoration filter H _w (x, y), a function F obtained by performing a two-dimensional discrete Fourier transform on the shake correction image using the following Expression 10: (X, y) is determined.

Subsequently, F (x, y) obtained by Expression 10 is subjected to two-dimensional discrete inverse Fourier transform (inverse two-dimensional DFT) using the following Expression 11 to correct subject blur in the object region. A luminance image f (x, y) is calculated.

  Subsequently, the color image separated using Equations 3 to 5 is combined with the luminance image f (x, y) in which subject blur is corrected, and an image in which subject blur is corrected is generated.

For example, the value of f (x, y) at each pixel position is Y, and the RGB values of the color-corrected image from the values of U and V at each pixel position calculated using Expressions 3 to 5 are given by It is calculated using 12 to 14.
R = Y + 1.402V (Formula 12)
G = Y−0.344U−0.714V (Formula 13)
B = Y + 1.772U (Formula 14)

  From the RGB data thus calculated, for example, as shown in FIG. 7B, an object blur correction image in which the blur of the object area is corrected is generated.

  Subsequently, as illustrated in FIG. 8A, the object image in the object area 521 is cut out from the image recorded in the captured image storage unit 230 based on the object area of the object. Further, as shown in FIG. 8B, an object image 530 is extracted from the object shake correction image based on the object area of the object. Then, as illustrated in FIG. 8C, the target image 530 is combined with the target region of the image 520 to create a composite image 540.

  Next, boundary correction processing in the case where the object image included in the object blur correction image and the captured image before correction are combined will be described in detail with reference to the drawings.

  FIG. 9 is a diagram schematically illustrating a case where the original image 600 and the object image 602 are combined. In addition, in FIG. 9, each image is simplified and shown as a square. FIG. 9A is a diagram illustrating an original image 600 that is an image obtained by cutting out an object image from a captured image. Here, a part from which the object image is extracted is shown as an object area 601. Further, in the original image 600 shown in FIG. 9A, it is assumed that the area other than the object area 601 is an image without blur.

  FIG. 9B is a diagram illustrating an object image 602 extracted from an object blur correction image that is an image obtained by performing blur correction processing on the original image 600. A broken line 603 around the object image 602 is a broken line indicating the size of the object shake correction image corresponding to the original image 600.

  FIG. 9C is a diagram showing a composite image 610 that is an image obtained by combining the object image 602 with the object region 601 of the original image 600. In this way, the composite image 610 is created by compositing the target object image 602 that has been subjected to the blur correction process with the target object region 601 of the original image 600. As a result, the entire composite image 610 becomes a blur-free image.

  Here, when the object region portion of the current image and the corrected image is combined, the boundary portion of the combined region may be noticeable and unnatural as shown in FIG. 9C. Therefore, as shown in FIG. 10, image processing is performed on the boundary portion.

  FIG. 10 is a diagram illustrating an example of image processing performed on a boundary portion when an original image and an object image are combined. In FIG. 10, the region boundary between the original image 620 and the object image 630 when combining the original image 620 and the object image 630 is defined as the region boundary 640, and the original image 620 side and the object image are centered on the region boundary 640. An example of correcting the region within the range of 2n pixels on both sides of the 630 side will be described.

In this example, the pixel value C (x, y) of the composite image is obtained using the following Expression 15.
C (x, y) = w (x, y) .A (x, y) + (1.0-w (x, y)). B (x, y) (Equation 15)

  Here, A (x, y) is a pixel value of the original image 620, and B (x, y) is a pixel value of the object image 630. Further, w (x, y) is a weighting coefficient that takes a value in the range of 0.0 to 1.0. For example, the value on the region boundary 640 is set to 0.5, and the original image 620 is converted from the region boundary 640. The value of w (x, y) is set to increase by k every n pixels away from the side. Then, the correction is made to the range where k is 1 on the original image 620 side, and the correction is made to the range where k is 0 on the object image 630 side. For example, when the resolution of the original image 620 is about VGA (640 × 480), the boundary portion can be made inconspicuous with values of n = 1 and k = 0.05.

  As described above, the pixel values according to changes in the pixel values of both the original image 620 and the object image 630 as the n pixel regions 621, 622, 632, and 632 progress from the original image 620 toward the object image 630. Therefore, it is possible to make the boundary portion look natural.

  FIG. 11 is a diagram schematically illustrating a case where boundary correction processing is performed at the region boundary between the original image 600 and the object image 602. In addition, since FIG. 11 (a) and (b) are the same as FIG. 9 (a) and (b), description here is abbreviate | omitted.

  FIG. 11C is a diagram showing a composite image 650 that is an image obtained by combining the object image 602 shown in FIG. 11B with the object region 601 of the original image 600 shown in FIG. . Here, the composite image 650 shown in FIG. 11C is the same as the composite image 610 shown in FIG. 9C in that boundary correction processing is performed at the region boundary between the original image 600 and the object image 602. Different. In FIG. 11C, for the sake of explanation, the correction portion at the region boundary is shown relatively large.

  In this way, by performing correction processing on the region boundary between the original image 600 and the object image 602, the region boundary between the original image 600 and the object image 602 can be made inconspicuous and appear natural.

  Next, the operation of the imaging apparatus 100 according to the embodiment of the present invention will be described with reference to the drawings.

  FIG. 12 is a flowchart illustrating a processing procedure of captured image storage processing by the imaging device 100. In this example, a case where an image is input from the image input unit 310 for each frame will be described. In addition, a case will be described in which an object included in a captured image displayed on the display unit 210 is determined in response to an operation input from the operation reception unit 250 by the user.

  First, it is determined whether or not an operation input for determining an object is received by the operation receiving unit 250 in the captured image displayed on the display unit 210 (step S901). For example, the object can be determined by selecting the turtle 401 using the cursor 406 in the image 405 shown in FIG. If the operation input for determining the object is not received (step S901), the monitoring is continued until the operation input for determining the object is received.

  When an operation input for determining an object is received (step S901), the amount of movement of the object corresponding to the operation input is detected (step S902). For example, as shown in FIG. 5B, a motion vector 453 is detected as the amount of movement of the turtle 451. Subsequently, a region of the object corresponding to the operation input is extracted (step S903). For example, as shown in FIG. 4A, the object region 421 is extracted as a turtle region.

  Subsequently, it is determined whether or not an operation input for pressing the shutter has been received by the operation receiving unit 250 (step S904). If an operation input for pressing the shutter is not accepted (step S904), it is determined whether or not the object determined in step S901 is present in the captured image displayed on the display unit 210 (step S905). If the target object determined in step S901 does not exist in the captured image displayed on the display unit 210 (step S905), there is no target object for detecting the amount of motion, so the process returns to step S901. Monitoring is performed until an operation input for determining is accepted.

  On the other hand, when the target object determined in step S901 is present in the captured image displayed on the display unit 210 (step S905), based on the amount of motion of the target object detected in step S902, the target object is detected. The position is updated (step S906). Until the operation input for pressing the shutter is accepted (step S904), the detection of the amount of movement of the object whose position has been updated (step S902) and the extraction of the area of the object (step S903) are continued. Is done.

  When an operation input for pressing the shutter is received by the operation receiving unit 250 (step S904), the amount of movement of the target is predicted based on the amount of movement of the target detected when the shutter is pressed. (Step S907). For example, as illustrated in FIG. 5C, the motion vector 463 is calculated as the motion prediction amount of the turtle 451.

  Subsequently, the calculated motion estimation amount of the object and the object area extracted when the shutter is pressed are stored in the captured image storage unit 230 in association with the image captured when the shutter is pressed (Step S110). S908).

  FIG. 13 is a flowchart illustrating a processing procedure of object blur correction processing by the imaging apparatus 100. In this example, a case will be described in which blurring of an object included in the captured image stored in the captured image storage unit 230 is corrected.

  First, a captured image stored in the captured image storage unit 230 is input (step S911). Subsequently, it is determined whether or not the amount of movement of the object and the object area are recorded in the input captured image (step S912). If the amount of movement of the object and the object area are not recorded in the input photographed image (step S912), the input photographed image is displayed on the display unit 210 without performing the object blur correction process ( Step S924).

  When the amount of movement of the object and the object area are recorded in the input captured image (step S912), the luminance component and the color component are separated from the input captured image, and only the luminance component is present. The extracted luminance image g (x, y) is generated (step S913). For example, the luminance component and the color component are separated using the above Equations 3 to 5, and the luminance image g (x, y) is generated.

  Subsequently, a point spread function h (x, y) is calculated based on the amount of movement of the object recorded in the input captured image (step S914). For example, the point spread function h (x, y) is obtained using the above-described Expression 6.

  Subsequently, the luminance image g (x, y) generated in step S913 is subjected to two-dimensional discrete Fourier transform (two-dimensional DFT), and G (x, y) is calculated (step S915). For example, the number of taps is M, the width of the captured image is M, the height of the captured image is N, and the luminance image g (x, y) is converted into a two-dimensional discrete Fourier transform (two-dimensional DFT) using the above-described equation 7. Is done.

  Subsequently, the point spread function h (x, y) calculated in step S914 is subjected to two-dimensional discrete Fourier transform (two-dimensional DFT), and H (x, y) is calculated (step S916). For example, as the number of taps, the width of the captured image is M, the height of the captured image is N, and the point spread function h (x, y) is converted into a two-dimensional discrete Fourier transform (two-dimensional DFT) using Equation 8 above. )

Subsequently, the restoration filter H _w (x, y) is calculated from H (x, y) obtained in step S916 (step S917). For example, the restoration filter H _w (x, y) is obtained by using the above-described Equation 9 from H (x, y) obtained by Equation 8.

Subsequently, based on G (x, y) calculated in step S915 and the restoration filter H _w (x, y) calculated in step S917, a function F (2) obtained by performing a two-dimensional discrete Fourier transform on the shake correction image. x, y) is calculated (step S918). For example, based on G (x, y) obtained by the above equation 7 and the restoration filter H _w (x, y) obtained by the above equation 9, blur correction is performed using the above equation 10. A function F (x, y) obtained by two-dimensional discrete Fourier transform of the image is obtained.

  Subsequently, F (x, y) calculated in step S918 is subjected to two-dimensional discrete inverse Fourier transform (inverse two-dimensional DFT), and the luminance image f (x, y) obtained by correcting the subject blur in the object region. Is calculated (step S919). For example, F (x, y) obtained by the above-described equation 10 is subjected to a two-dimensional discrete inverse Fourier transform (inverse two-dimensional DFT) using the above-described equation 11 to correct subject blur in the object area. The obtained luminance image f (x, y) is calculated.

  Subsequently, the color component separated in step S913 is combined with the luminance image f (x, y) in which the subject blur is corrected to generate an image in which the subject blur is corrected (step S920). For example, the value of f (x, y) at each pixel position is Y, and the RGB value of the color correction image is calculated from the U and V values at each pixel position calculated using the above-described equations 3 to 5. This is calculated using Equations 12 to 14. Thereby, for example, as shown in FIG. 7B, a shake correction image in which the shake of the object area is corrected is generated.

  Subsequently, on the basis of the object area recorded in the input captured image, the object image is extracted from the shake correction image generated in step S920, and is extracted into the object area in the input captured image. The object images thus synthesized are synthesized (step S921). For example, as shown in FIG. 8C, the object image is combined with the input captured image.

  Subsequently, correction processing is performed on the region boundary of the target image in the combined image obtained by combining the target images (step S922). For example, as illustrated in FIG. 11C, correction processing is performed on the region boundary of the target object image 651 in the composite image 650.

  Subsequently, an object shake correction image obtained by performing correction processing on the region boundary of the object image in the composite image is displayed on the display unit 210 (step S923).

  As described above, for example, when the user photographs a turtle swimming in an aquarium tank with the imaging device 100 fixed, the turtle 501 is displayed as shown in an image 500 in FIG. Although there is no blur in the background, the subject is blurred and photographed in the swimming turtle 501. Therefore, in the embodiment of the present invention, detection of the amount of movement of the turtle and extraction of the region are executed at the time of shooting, and the image 510 (see FIG. 7 (b)) is generated. Then, the image of the turtle 511 included in the image 510 subjected to the blur correction is extracted, and the extracted image of the turtle 511 is combined with the region of the turtle 501 of the image 500 that is not subjected to the blur correction. As a result, it is possible to correct the blurring of only the moving object and create a beautiful image without blurring of the object and the background. In addition, it is possible to easily obtain an image with reduced object blur and background blur without requiring a special technique related to shooting.

  In the embodiment of the present invention, detection of the amount of movement of the object and extraction of the region are executed at the time of shooting, but only detection of the amount of movement of the target is executed at the time of shooting, and the object shake correction processing is performed. When performing, the region of the object may be extracted.

  In the embodiment of the present invention, the amount of movement and the region of the object are associated with the photographed image and recorded in the photographed image storage unit 230 at the time of photographing. An object shake correction may be performed on the captured image, and the image subjected to the object shake correction may be recorded in the captured image storage unit 230.

  In the embodiment of the present invention, the motion prediction amount calculated by the object motion prediction amount calculation unit 350 is used as the object motion amount for the object blur correction, but is detected by the object motion amount detection unit 330. You may make it use the made motion amount for object blurring correction.

  In the embodiment of the present invention, the imaging apparatus has been described as an example. However, the embodiment of the present invention can be applied to an image processing apparatus that can input a captured image and perform blur correction processing. it can.

  The embodiment of the present invention is an example for embodying the present invention and has a corresponding relationship with the invention-specific matters in the claims as shown below, but is not limited thereto. However, various modifications can be made without departing from the scope of the present invention.

  That is, in claims 1 to 8, the imaging device corresponds to the imaging device 100, for example. Further, in claim 9, an image processing apparatus corresponds to the imaging apparatus 100, for example.

  Further, in claim 1, the imaging unit corresponds to, for example, the imaging unit 110. Also, the motion amount detection means corresponds to the object motion amount detection unit 330, for example.

  Further, in claim 1 or claim 9, the correction means corresponds to, for example, the shake correction unit 381. The synthesizing unit corresponds to the image synthesizing unit 383.

  Further, in claim 2, the image storage means corresponds to, for example, the photographed image storage unit 230. A storage control unit corresponds to the storage control unit 370, for example.

  Further, in claim 5, the recording operation reception means corresponds to, for example, the operation reception unit 250.

  Further, in claim 6, the motion amount prediction means corresponds to, for example, the target object motion prediction amount calculation unit 350.

  Further, in claim 7, the display means corresponds to the display unit 210, for example. The selection receiving unit corresponds to the operation receiving unit 250, for example.

  Further, in claim 8, the boundary correction means corresponds to the region boundary correction unit 384.

  Further, in claim 9, input means corresponds to, for example, the image input unit 310.

  Further, in claim 10 or claim 11, the imaging procedure corresponds to, for example, steps S901 to S906. The motion amount detection procedure corresponds to step S903, for example. The correction procedure corresponds to, for example, steps S913 to S920. Also, the synthesis procedure corresponds to, for example, step S922.

  The processing procedure described in the embodiment of the present invention may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program May be taken as

2 is a block diagram illustrating a functional configuration example of an imaging apparatus 100. FIG. 2 is a block diagram illustrating a functional configuration example of an imaging apparatus 100. FIG. 6 is a diagram illustrating a display example when an image captured by an imaging unit 110 is displayed on a display unit 210. FIG. 3 is a diagram illustrating an outline of a region of an object determined from images captured by an imaging unit 110. FIG. It is a figure which shows the outline of the transition of the motion of the target object determined from the images taken into the imaging part. It is a figure which shows the outline of the file structure of the still image file recorded by DCF standard. It is a figure which shows the image before correct | amending about the image currently recorded on the picked-up image memory | storage part 230, and the image after correct | amending based on the motion amount of a target object. It is a figure which shows the outline of the correction | amendment method in the case of synthesize | combining the image of the object part in which blurring was correct | amended, and the image of parts other than the object contained in the image before correction | amendment. FIG. 6 is a diagram schematically illustrating a case where an original image 600 and an object image 602 are combined. It is a figure which shows an example of the image process performed to a boundary part when synthesize | combining an original image and a target object image. FIG. 10 is a diagram schematically illustrating a case where boundary correction processing is performed at a region boundary between an original image and an object image. 5 is a flowchart illustrating a processing procedure of a captured image storage process performed by the imaging apparatus 100. 6 is a flowchart illustrating a processing procedure of object blur correction processing by the imaging apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Imaging device 110 Imaging part 120 CPU
130 RAM
DESCRIPTION OF SYMBOLS 140 Camera signal processing engine 150 Object detection engine 160 Object area extraction engine 180 Motion amount detection engine 190 Shake correction engine 200 Image display engine 210 Display unit 220 Media controller 230 Photographed image storage unit 240 Operation input controller 250 Operation reception unit 260 Data Bus 310 Image input unit 320 Object position determination unit 330 Object motion amount detection unit 340 Object position update unit 350 Object motion prediction amount calculation unit 360 Object region extraction unit 370 Storage control unit 380 Object blur correction unit 381 Correction unit 382 Object image extraction unit 383 Image composition unit 384 Region boundary correction unit 390 Display control unit

Claims (11)

Imaging means for converting incident light into imaging data;
A motion amount detecting means for detecting a motion amount of a predetermined area included in the imaging data;
Correction means for correcting the predetermined area of the imaging data based on the amount of motion to obtain a corrected image;
An image pickup apparatus comprising: combining means for combining the corrected image with the predetermined area of the image pickup data. Image storage means for storing the imaging data;
Storage control means for storing the detected amount of motion in the image storage means in association with the image of the predetermined area;
The correction unit corrects the predetermined area of the imaging data stored in the image storage unit based on the amount of motion stored in the image storage unit corresponding to the imaging data to obtain a corrected image;
2. The imaging apparatus according to claim 1, wherein the synthesizing unit synthesizes the corrected image with the predetermined area of the imaging data stored in the image storage unit. The imaging apparatus according to claim 2, wherein the storage control unit stores area information relating to the predetermined area in the image storage unit in association with an image of the predetermined area. The imaging apparatus according to claim 3, wherein the area information includes a reduced image of an image included in the predetermined area. A recording operation receiving unit that receives a recording operation of an image captured by the imaging unit;
The motion amount detection means detects the motion amount of the predetermined area for each frame of the image captured by the imaging means,
The storage control unit records an image captured by the imaging unit when the recording operation is accepted, and associates the amount of motion of the predetermined area detected when the recording operation is accepted with the image. The image pickup apparatus according to claim 2, wherein the image storage unit records the image in the image storage unit. Further comprising a motion amount prediction means for predicting a new motion amount based on the motion amount of the predetermined area detected when the recording operation is received,
The storage control unit stores an image captured by the imaging unit when the recording operation is received, and stores the new amount of movement in the image storage unit in association with the image. The imaging device according to claim 5. Display means for displaying the captured image;
Selection accepting means for accepting selection input of a specific object included in the displayed image;
The imaging apparatus according to claim 1, further comprising a predetermined area extracting unit that extracts the area of the selected object as the predetermined area. The image processing apparatus further comprises boundary correction means for changing a pixel value of the synthesized image in accordance with a distance from the boundary in a peripheral region of the boundary of the image included in the predetermined area in the synthesized image. The imaging apparatus according to claim 1. Input means for inputting imaging data in which incident light is converted and a motion amount of a predetermined area included in the imaging data;
Correction means for correcting the predetermined area of the imaging data based on the amount of motion to obtain a corrected image;
An image processing apparatus comprising: a combining unit that combines the corrected image with the predetermined region of the imaging data. An imaging procedure for converting incident light into imaging data;
A motion amount detection procedure for detecting a motion amount of a predetermined area included in the imaging data;
A correction procedure for correcting the predetermined area of the imaging data based on the amount of motion to obtain a corrected image;
An image processing method comprising: a combining procedure for combining the corrected image with the predetermined region of the imaging data. An imaging procedure for converting incident light into imaging data;
A motion amount detection procedure for detecting a motion amount of a predetermined area included in the imaging data;
A correction procedure for correcting the predetermined area of the imaging data based on the amount of motion to obtain a corrected image;
A program for causing a computer to execute a synthesis procedure for synthesizing the corrected image with the predetermined region of the imaging data.