JP2012085205A - Image processing apparatus, imaging device, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce not only an image blur generated between a plurality of frames but also a blur by image deletion generated in one frame.SOLUTION: An image processing apparatus 109 comprises first processing means 1094 performing first processing for reducing an image blur, which is due to motion of an imaging device, between a plurality of continuous frames in a moving image generated by imaging by the imaging device, second processing means 1092 and 1093 performing second processing for reducing a blur by image deletion, which is due to the motion of the imaging device, for each frame in the moving image, and motion information acquisition means 1091 acquiring motion information of the imaging device. When performing the first processing, the second processing means determines whether the blur by image deletion is present or absent for each frame from the motion information and a frame rate and focal distance at the time of imaging. The second processing means does not perform the second processing on a frame in which the blur by image deletion is determined to be absent, and performs the second processing on a frame in which the blur by image deletion is determined to be present.

Description

  The present invention relates to an image processing apparatus that reduces image blur and image blur of a moving image (video) acquired by an imaging apparatus.

  Many imaging devices perform optical or electronic image blur reduction processing for reducing image blur in a video caused by small movement such as so-called camera shake. In addition, in order to reduce image blur caused by a motion larger than a camera shake that occurs when a user who has an imaging device performs shooting while walking, for example, Patent Document 1 discloses a technique between a plurality of frames in a video. An image blur reduction process for detecting and correcting the image change amount is disclosed.

  However, the movement applied to the imaging apparatus when taking an image while walking includes a sudden movement caused by an impact or the like when the user puts his / her foot on the ground, and an image within one frame is caused by this sudden movement. Blur due to flow (hereinafter referred to as image flow blur) occurs. When the image blur reduction process disclosed in Patent Document 1 is applied to an image including such an image blur, the image blur is noticeable because a large image blur is reduced. For this reason, in a series of videos, an unsightly video may be generated, such as appearing in a period with a blurred image as the user walks.

  In order to generate an image with less image blur, Patent Document 2 discloses a method of generating a single frame by synthesizing a plurality of frames generated at a higher frame rate than usual when a sudden movement is detected. Has been. Patent Document 3 discloses a method for selectively outputting only frames other than frames that may contain image blur.

JP 2008-259076 A JP 2006-033123 A JP 2007-150494 A

  However, the method disclosed in Patent Document 2 reduces image blur caused by motion faster than the frame rate at the time of imaging (that is, motion generated within the generation time of one frame). I can't. Further, in the method disclosed in Patent Document 3, if frames that may include image blur are consecutive and are excluded, there is a risk that the image skips due to a sudden change in the subject in the image. There is.

  The present invention provides an image processing apparatus, an imaging apparatus, an image processing method, and an image processing program capable of reducing image blur that occurs between a plurality of frames and also reducing image blurring that occurs within one frame. I will provide a.

  An image processing apparatus according to an aspect of the present invention performs a first process for reducing image blur caused by movement of an imaging device between a plurality of consecutive frames in a moving image generated by imaging by the imaging device. Processing means, second processing means for performing second processing for reducing image blur caused by movement of the imaging device for each frame in the moving image, and motion information acquisition means for acquiring motion information of the imaging device And have. When the first processing means performs the first processing, the second processing means uses the motion information acquired by the motion information acquisition means and the frame rate and focal length at the time of imaging of the imaging device. The image processing blur is determined for each frame, the second process is not performed on the frame determined to have no image blur, and the second processing is performed on the frame determined to have the image blur. It is characterized by performing processing.

  Note that an imaging apparatus having an imaging system that generates a moving image by imaging and the image processing apparatus also constitutes another aspect of the present invention.

  According to another aspect of the present invention, there is provided an image processing method that reduces image blur caused by movement of an imaging device between a plurality of consecutive frames in a moving image generated by imaging by the imaging device. A first processing step for performing processing; a second processing step for performing second processing for reducing image blur caused by motion of the imaging device for each frame in the moving image; and acquiring motion information of the imaging device A motion information acquisition step. Then, when performing the first process in the first process step, from the motion information acquired in the motion information acquisition step in the second process step, and the frame rate and focal length at the time of imaging of the imaging device The image processing blur is determined for each frame, the second process is not performed on the frame determined to have no image blur, and the second processing is performed on the frame determined to have the image blur. It is characterized by performing processing.

  According to another aspect of the present invention, there is provided an image processing program that first causes a computer to reduce image blur caused by movement of an imaging apparatus between a plurality of consecutive frames in a moving image generated by the imaging apparatus. A first processing step for performing the above-described processing, a second processing step for performing a second processing for reducing image blur caused by the motion of the imaging device for each frame in the moving image, and information on the motion of the imaging device. A process including a motion information acquisition step to be acquired is executed. When the first processing step causes the computer to perform the first processing, in the second processing step, the computer causes the computer to acquire the motion information acquired by the motion information acquisition step and the frame at the time of imaging by the imaging device. Based on the rate and focal length, the presence / absence of image blur is determined for each frame, and the second process is not performed on frames determined to have no image blur, and it is determined that there is image blur. The second process is performed on the received frame.

  According to the present invention, the image blur can be reduced by the first processing, and the image processing blur can be reduced while reducing the processing load by performing the second processing only on the frame including the image flow blur. Can be reduced.

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. 6 is a flowchart illustrating the operation of the imaging apparatus according to the first embodiment. FIG. 6 is a diagram for explaining calculation of image flow blur and image flow blur kernel in the first embodiment. 9 is a flowchart illustrating image shake correction processing of an imaging apparatus that is Embodiment 2 of the present invention. 9 is a flowchart illustrating image shake correction processing of an imaging apparatus that is Embodiment 3 of the present invention. FIG. 10 is a diagram illustrating an overview of image flow blur correction processing of an imaging apparatus that is Embodiment 3 of the present invention. Schematic which shows the structure of the image processing apparatus which is Example 4 of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration of an image pickup apparatus such as a video camera including an image processing apparatus according to an embodiment of the present invention or a digital still camera with a moving image shooting function.

  In FIG. 1, reference numeral 100 denotes a photographic lens that forms an image of light from a subject on an image sensor 102. Reference numeral 101 denotes an exposure control member including a diaphragm, a shutter, and the like, and controls exposure of the image sensor 102 by light passing through the photographing lens 100. The image sensor 102 is an image sensor such as a CCD sensor or a CMOS sensor that photoelectrically converts a subject image and outputs an electrical signal.

  An image forming circuit 103 converts an analog output signal from the image sensor 102 into a digital signal, and further performs gain adjustment, auto white balance, pixel interpolation processing, color conversion processing, and the like on the digital signal. A signal (moving image) is generated. An exposure control unit 104 controls the operation of the exposure control member 101 (aperture aperture diameter and exposure time (frame rate)). Reference numeral 105 denotes an AF control unit that controls focusing of the photographing lens 100. An imaging system that captures an image of a subject and generates a video signal (hereinafter also simply referred to as an image) by the taking lens 100, the exposure control member 101, the image sensor 102, the image forming circuit 103, the exposure control unit 104, and the AF control unit 105. Composed.

  Reference numeral 106 denotes a system control circuit that controls the operation of the entire imaging apparatus 1. A memory 107 stores various data and computer programs used in the system control circuit 106. A non-volatile memory 108 stores information such as various adjustment values.

  Reference numeral 109 denotes an image blur / image blur correction circuit as an image processing apparatus, which reduces image blur that appears between a plurality of consecutive frames in a video due to a gentle movement such as a camera shake of the imaging apparatus 1. Processing (first processing) is performed. Further, the image blur / image blur correction circuit 109 reduces image blur (image blur) caused by an image flow appearing in one frame of the image due to a rapid movement of the image pickup apparatus 1 (first image blur correction process). 2).

  Reference numeral 110 denotes a first frame memory that temporarily stores a plurality of frames of the video generated by the video forming circuit 103. Reference numeral 111 denotes a second frame memory that temporarily stores a plurality of frames of video that have undergone at least one of image blur correction processing and image flow blur correction processing by the image blur / image blur correction circuit 109. Reference numeral 112 denotes a memory control circuit that controls frame input / output with respect to the first and second frame memories 110 and 111. A video output unit 113 displays the video output from the image blur / image blur correction circuit 109 on a display monitor (not shown) or records it on a recording medium (magnetic tape, optical disk, semiconductor memory, etc.) (not shown). is there.

  Next, the configuration of the image blur / image blur correction circuit 109 will be described. 1091 is a motion information acquisition unit that detects a motion of the imaging device 1 (that is, acquires motion information) by calculating a motion vector indicating a motion direction and a motion amount in a video generated by the imaging system. This is a detection circuit. Reference numeral 1092 denotes an image flow blur evaluation value calculation circuit for calculating an image flow blur evaluation value for determining the presence or absence of image flow blur for each frame.

  Reference numeral 1093 denotes an image flow blur correction processing circuit that performs image flow blur correction processing for each frame based on the image flow blur evaluation value calculated by the image flow blur evaluation value calculation circuit 1092. Reference numeral 1094 denotes an image blur correction processing circuit that performs an image blur correction process based on the motion of the imaging apparatus 1 detected by the motion detection circuit 1091. The image blur correction processing circuit 1094 and an image blur / image blur control circuit 1097 described later constitute a first processing means. The image blur evaluation value calculation circuit 1092, the image blur correction processing circuit 1093, and the image blur / image blur control circuit 1097 constitute a second processing unit.

  Reference numeral 1095 denotes a temporary storage memory, and reference numeral 1096 denotes a nonvolatile memory. Reference numeral 1097 denotes an image blur / image blur control circuit that controls the operation of each circuit in the image blur / image blur correction circuit 109, and is constituted by a computer such as an MPU.

  Next, the configuration of the motion detection circuit 1091 will be described. Reference numeral 10911 denotes a motion vector calculation circuit that calculates (detects) a motion vector that is motion information of the imaging apparatus 1 between a plurality of consecutive frames (for example, between two frames generated before and after in time). Reference numeral 10912 denotes a motion estimation circuit that estimates the motion of the imaging apparatus 1 using the motion vector calculated by the motion vector calculation circuit 10911.

  Next, the operation of the imaging apparatus 1 will be described using the flowchart shown in FIG. FIG. 2A illustrates processing until image blur and image blur in a video obtained by imaging a subject using the imaging device 1 are reduced and displayed and recorded by the video output unit 113 described above. Indicates. This process is executed by an image blur / image blur control circuit (hereinafter simply referred to as a control circuit) 1097 according to an image processing program as a computer program.

  In step a201, when the control circuit 1097 detects that an imaging start switch (not shown) provided in the imaging device 1 has been operated, the exposure control unit 104 determines based on the luminance component of the output signal from the imaging device 102. The exposure amount of one frame to be controlled is set. Also, the AF control unit 105 performs focusing control based on the contrast information of the video. Thereafter, the control circuit 1097 causes the image forming circuit 103 to read an output signal (imaging signal) from the imaging element 102.

  In step a <b> 202, the control circuit 1097 causes the image forming circuit 103 to generate an image from the image signal read from the image sensor 102 in step a <b> 201, and stores the image in the first frame memory 110.

  In step a203, the control circuit 1097 determines whether or not an image blur / image blur correction mode for reducing image blur and image blur in the video is set by the user's selection operation in the imaging apparatus 1. . If the image blur / image blur correction mode is not set, the process proceeds to step a205, and the control circuit 1097 displays the video from the video forming circuit 103 that is not subjected to the image blur correction process and the image blur correction process. The video is output to the video output unit 113. On the other hand, if the image blur / image blur correction mode is set, the process proceeds to step a204.

  In step a204 (first processing step and second processing step), the control circuit 1097 causes the image blur / image blur correction circuit 109 to perform image blur correction processing and image flow on the video from the video forming circuit 103. Perform blur correction processing. The process at step a204 will be described with reference to the flowchart of FIG.

  In step b201 (motion information acquisition step), the control circuit 1097 causes the motion detection circuit 1091 to detect the motion of the imaging device 1. The motion vector calculation circuit 10911 of the motion detection circuit 1091 first extracts the feature points of the previous frame stored in the first frame memory 110. This feature point is a characteristic part such as a line segment or an intersection of lines in the image, and is extracted using an image processing method such as a Harris operator. Next, the motion vector calculation circuit 10911 obtains corresponding points corresponding to the feature points of the previous frame in the current frame. Corresponding points are obtained by a technique such as template matching using a template including a feature point and its surrounding area. Then, a motion vector between the previous frame and the current frame is calculated from the coordinates of the feature points of the previous frame and the coordinates of the corresponding points of the current frame. The motion vector calculation circuit 10911 performs the same process on a plurality of feature points in each frame.

  Next, in step b202, the control circuit 1097 causes the image blur evaluation value calculation circuit 1092 to calculate an image blur evaluation value serving as an index for determining the presence or absence of image blur. The image blur evaluation value calculation circuit 1092 calculates an image blur evaluation value using the motion vector calculated for each feature point in step b201. The reason why the motion vector is used for calculating the image blur evaluation value is that the magnitude of the motion vector reflects the presence or absence of the image blur.

  Assume that the number of feature points is N, the motion vector calculated at each feature point is mv (i) (i = 1 to N), and the image blur evaluation value E_mv is defined as in Expression 1.

E_mv = mv_s = func_s (mv) (Formula 1)
In Equation 1, mv_s is a representative motion vector (motion information) as one motion vector representing the entire frame obtained from motion vectors at a plurality of feature points. Func_s () represents a function for calculating a representative motion vector, and is realized by a function for calculating a median, for example. By using the representative motion vector, the motion of the imaging device 1 can be simplified and expressed.

  Although the case where the median is used as an image blur evaluation value will be described here, other values such as an average value may be used. Here, a case will be described in which a motion vector is used as an image blur evaluation value as it is. However, when determining the presence or absence of image blurring in more detail, a high-frequency component is calculated by performing filtering processing or orthogonal transformation processing on a frame (image), and an image indicating whether the high-frequency component is large or small A flow blur evaluation value may be used. Further, an edge component in the frame may be detected, and an image blur evaluation value indicating whether the change in the pixel value of the edge component is steep or gentle may be used. Both the magnitude of the high frequency component and the change in the pixel value of the edge component are included in the movement information of the imaging apparatus 1.

  Step b202 to step b204 described later correspond to the second processing step.

  Next, in step b203, the control circuit 1097 determines whether or not the current frame includes image blur (based on the presence or absence of image blur) based on the image blur evaluation value calculated in step b202. . Specifically, as shown in Expression 2, the determination is made by comparing the image blur evaluation value and the image blur evaluation threshold E_th.

E_th (s, f) <E_mv: No image blurring E_th (s, f) ≧ E_mv: Image blurring (Formula 2)
In Expression 2, s and f respectively represent a frame rate at the time of image generation by imaging (at the time of imaging) and a focal length of the imaging device 1 (the imaging lens 100). The image blur evaluation threshold value E_th is a threshold value determined according to the frame rate s and the focal length f at the time of imaging as described above. In other words, the control circuit 1097 determines the presence / absence of image blur from the representative motion vector, which is information on the movement of the imaging apparatus 1, and the frame rate and focal length when the imaging apparatus 1 is imaging.

  When the frame rate at the time of imaging is high or when the focal length is short, the image drift blur evaluation threshold value is high. In this case, it is determined that there is an occurrence of image blurring in a case where an extremely rapid movement occurs in the imaging apparatus 1. In addition, when the frame rate is low or the focal length is long, the image blur evaluation threshold is small. In this case, even if the imaging apparatus 1 has a gentle movement (however, a movement faster than a movement that causes image blur) as compared with a case where the frame rate is high or a focal length is short, image blurring may occur. Is easily determined. If it is determined that there is no image blur, the process proceeds to step b205. On the other hand, if it is determined that there is image blur, the process proceeds to step b204, and the control circuit 1097 causes the image blur / image blur correction circuit 109 to perform image blur correction processing.

  Here, the image blur correction process performed in step b204 will be described. If g is an observed image (degraded image) including image flow blur, and m is an ideal image not including image flow blur, the observed image g can be expressed as Equation 3.

g = h * m + n (Formula 3)
* Represents convolution, and h is an operator that degrades the image, and represents an image flow blur kernel that causes image flow blur. n represents an additional noise component. Image degradation (image blurring) can be reduced (corrected) by estimating m from g and h.

  When Expression 3 is Fourier-transformed and expressed in the frequency space, Expression 4 is obtained.

G = H · M + N (Formula 4)
Here, G, H, M, and N represent Fourier transforms of g, h, m, and n, respectively. At this time, the Wiener filter W is expressed by Equation 5.

W = H ^* · Pm / (| H | ² · Pm + Pn) (Formula 5)
Here, ^* represents a complex conjugate, and Pm and Pn represent the ideal image and the noise power spectrum, respectively. Here, since Pm is unknown, it is assumed that the signal and the noise are uncorrelated. That is, Pm / Pn = c (constant). At this time, Expression 5 can be described as Expression 6.

W = H ^* / (| H ² | + 1 / c) (Formula 6)
If the image-blurring kernel h is known from the equations 4 and 6, it is possible to calculate the image M in which the deterioration has been recovered.

  A method for setting the image flow blur kernel h that causes image flow blur will be described. The image blur kernel is set based on the motion vector calculated in step b201. This is shown in FIG. FIG. 3A shows an image in which image blur is generated and a motion vector obtained from the image. At time t, image blur is not noticeable. However, as a result of a sudden movement in the imaging apparatus when the time t shifts to the time t + 1, an image blur is generated that blurs as if the subject image flowed at the time t + 1. At this time, with reference to the image at time t, the motion vector is represented as indicated by an arrow 301 in the drawing. Using this motion vector 301, an image blur blur kernel is calculated as shown in Equation 7.

h = α (s, f) · mv_s (Formula 7)
As described above, mv_s is a representative motion vector of the entire frame, and is calculated by Expression 1. s and f are a frame rate and a focal length at the time of image pickup by the image pickup apparatus 1, respectively. α is a coefficient set as a value within a range of 0 to 1 set according to the frame rate s and the focal length f. The coefficient α is set small when the frame rate is high, that is, when the exposure time of the image sensor 102 per frame is short, and large when the frame rate is low. The coefficient α is set to be small when the focal length is short and large when the focal length is long.

  The image blur kernel calculated in this way can be expressed as an arrow 303 in FIG.

  From the image flow blur kernel h and the observed image g calculated as described above, an image with reduced image flow blur can be generated using Equation 4 and Equation 6. Then, after the image blur correction process is performed on the current frame determined to have image blur as described above, the process proceeds to step b205. As described above, the process proceeds to step b205 without performing the image blur correction process for the current frame determined to have no image blur.

  In step b205, the control circuit 1097 causes the motion estimation circuit 10912 to estimate the motion of the imaging device 1. The motion estimation circuit 10912 estimates the motion of the entire image corresponding to the motion of the imaging device 1 using the representative motion vector calculated in step b201. Here, the movement of the entire image is expressed as an image deformation amount. Specifically, the affine transformation shown in Expression 8 is used.

The motion estimation circuit 10912 estimates the conversion matrix A from the motion vector as an image deformation amount.

  Next, in step b206, the control circuit 1097 performs image blur correction processing using the image deformation amount (conversion matrix A) calculated in step b205. If the point x on the image is moved to the point x ′ by the affine transformation represented by the transformation matrix A,

By doing so, it is possible to perform image blur correction processing. Although the case where the inverse matrix A- ¹ of the transformation matrix A is directly used has been described here, the correction may be performed using a temporally smoothed transformation matrix. Step b205 and step b206 correspond to the first processing step.

  By performing the above processing, a good image with reduced image blur and reduced image blur is generated. Above, the process in step a204 is complete | finished.

  Next, in step a205, the control circuit 1097 displays or records the video processed in step a204 on the video output unit 113.

  According to the present embodiment, when image blur correction processing between a plurality of frames is performed, the image flow for each frame is based on the motion information of the imaging device 1 and the frame rate and focal length at the time of imaging of the imaging device 1. Determine if there is blur. Then, the image blur correction process is performed only for the frames determined to have image blur. As a result, it is possible to output a good video with reduced image blur and image blur while reducing the processing load.

  Further, the image blur correction process can be performed while the image blur correction process is being performed or after the image blur correction process is performed. Blur can be reduced.

  In the present embodiment, the case where the presence / absence of image blur is determined using a motion vector is described, but it is not always necessary to use a motion vector. For example, the image deformation amount calculated in step b205 of the flowchart shown in FIG. 2B, or the output value of a shake sensor that detects angular velocity and acceleration may be used. Furthermore, in this embodiment, the image blur kernel is also set using the motion vector, but may be set using the output value of the shake sensor.

  In this embodiment, the case where the image blur correction process is performed by electronic image processing has been described. However, the optical element such as the lens included in the photographing lens 100 is moved in a direction orthogonal to the optical axis. A typical image blur correction process may be performed.

  Next, as an embodiment 2 of the present invention, an embodiment in which the image blur correction effect is improved by performing the motion vector detection process again on the image after the image blur correction process has been performed will be described. .

  Since the configuration of the image pickup apparatus in the present embodiment is the same as that of the image pickup apparatus 1 shown in FIG. 1 in the first embodiment, common constituent elements are denoted by the same reference numerals as those in FIG. Further, the imaging apparatus according to the present exemplary embodiment performs the same process as that illustrated in the flowchart of FIG. However, the imaging apparatus according to the present exemplary embodiment performs the process illustrated in the flowchart of FIG. 4 instead of the process illustrated in the flowchart of FIG.

  In the flowchart of FIG. 4, steps b401 to b406 are the same as steps b201 to b206 of the flowchart shown in FIG.

  After the image blur correction process in step b404 is performed, in step b404 ′, the control circuit 1097 causes the motion detection circuit 1091 to detect the motion of the imaging apparatus 1 again. That is, the motion vector calculation circuit 10911 is caused to calculate a motion vector, and all the motion vector data calculated in step b401 is updated with the motion vector data newly calculated in step b404.

  In step b405, the control circuit 1097 causes the motion estimation circuit 10912 to estimate the motion of the imaging device 1 using the updated motion vector.

  As described above, by detecting the motion of the image pickup apparatus 1 (calculation of motion vector) using the image with reduced image blur, it is possible to calculate a more accurate motion vector and improve the motion detection accuracy. To do. As a result, the image blur correction effect can be improved.

  Next, Embodiment 3 of the present invention will be described. The present embodiment is an embodiment in which image blur correction processing is performed using a plurality of frames on which image blur correction processing has been performed in order to provide a user with higher definition video.

  Since the configuration of the image pickup apparatus in the present embodiment is the same as that of the image pickup apparatus 1 shown in FIG. 1 in the first embodiment, common constituent elements are denoted by the same reference numerals as those in FIG. Further, the imaging apparatus according to the present exemplary embodiment performs the same process as that illustrated in the flowchart of FIG. However, the imaging apparatus according to the present exemplary embodiment performs the process illustrated in the flowchart of FIG. 5 instead of the process illustrated in the flowchart of FIG.

  In the flowchart of FIG. 5, steps b501 to b506 are the same as steps b201 to b206 of the flowchart shown in FIG.

  If it is determined in step b503 that the current frame has image blur, the control circuit 1097 sets the image blur flag of the current frame to ON in step b507.

  In addition, after the image blur correction process is performed in step b506, in step b508, the control circuit 1097 determines whether or not the motion blur flag is ON. If ON, the process proceeds to step b509, and the control circuit 1097 causes the image blur correction processing circuit 1093 to perform the image blur correction process. On the other hand, if the image blur flag is OFF, the control circuit 1097 ends the image blur / image blur correction process.

  Here, the image blur correction process in step b509 will be described with reference to FIG. The image blur correction process in step b509 is performed using the video (plural frames) in which the image blur is reduced in step b506.

  In FIG. 6, reference numeral 601 denotes a plurality of frames that have already undergone image blur correction processing in step b506. In the image blur correction process, as indicated by a thick line in the drawing, a cutout region 602 is cut out from each of a plurality of frames so as to include the subject at the same position and sequentially output. Thereby, the position of the subject is always maintained at the same position on the video.

  In step b503, the frames 603 and 604 at time t (present) and time t-2 (previous) are frames with motion blur with the motion blur flag set to ON, and the image flow blur flag is turned off for other frames. It is assumed that the frame is a non-blurred frame. Then, the motion blur image correction process is performed using the image blur-free frame from the frame 603 with the image blur (current frame) 603 to be the target of the image blur correction process. Here, k = 3. Then, there are three image flow blur-free frames m (i) from time t-1 to t-4 (i = t−1, t−3, t−4, and image shift blur frames 604 (time t− 2) is used to perform an image blur correction process for newly generating a frame m ′ at time t according to Equation (10).

In Equation 10, each frame is calculated with the cut image coordinates. Further, i = t−1 to tk (however, a frame with image blur is excluded), and j = 1 to k.

  In this way, by performing image blur correction processing using a plurality of frames without image blur, it is high definition and less noise than when image blur correction processing using a single frame is performed. It is possible to generate a frame after the image blur correction process.

  Although the image blur is corrected here as a simple weighted average, a correction method using a plurality of frames (unequal-interval interpolation, image reconstruction type super-resolution, etc.) can also be applied.

  In addition, here, the case where the image blur correction process is performed using only the image blur-free frame generated before the target frame of the image blur correction process has been described. However, when image blur correction processing is performed on an image that has already been generated and recorded by imaging, the image blur-free frame generated after the target frame of the image blur correction processing is used. You may make it do. That is, it is possible to use at least one image blur-free frame before and after the target frame for the image blur correction process.

  Furthermore, here, the case where the image blur correction process is performed on the image that has been subjected to the image blur correction process has been described. However, only the amount of motion between frames is used without performing the image blur correction process. By performing the blur correction process, it is also possible to avoid image degradation due to geometric transformation.

  In each of the above embodiments, the case where the image processing apparatus is built in the imaging apparatus has been described. However, for example, as shown in FIG. 7, the image generated by the imaging device 1401 is transmitted to the personal computer 1402, and the personal computer 1402 performs the image blur correction process and the image blur correction process described in the first to third embodiments. May be. In this case, the personal computer 1402 functions as an image processing apparatus that executes the processing described with reference to FIGS.

  The video transmission method may be either a cable method or a wireless method, and may be transmitted via the Internet or a LAN.

  In this case, the movement of the imaging apparatus (acquisition of movement information) may be detected by the personal computer 1402, and the personal computer 1402 outputs the output from the shake sensor or the motion vector detection circuit mounted on the imaging apparatus. You may do it by taking in.

  Furthermore, the image processing apparatus that performs the image blur correction process and the image blur correction process described in each of the above embodiments may be mounted on a video display apparatus such as a monitor or a projector that reproduces and displays a video generated by imaging. .

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

  It is possible to provide an image processing apparatus and an imaging apparatus capable of satisfactorily reducing image blur and image blur.

DESCRIPTION OF SYMBOLS 1 Imaging device 109 Image blur / image blur correction circuit 1091 Motion detection circuit 1092 Image blur evaluation value calculation circuit 1093 Image blur correction processing circuit 1094 Image blur correction processing circuit

Claims (6)

First processing means for performing a first process for reducing image blur caused by movement of the imaging device between a plurality of consecutive frames in a moving image generated by imaging by the imaging device;
Second processing means for performing second processing for reducing image blur caused by movement of the imaging device for each frame in the moving image;
Movement information acquisition means for acquiring movement information of the imaging device;
When the first processing means performs the first processing, the second processing means
From the motion information acquired by the motion information acquisition means and the frame rate and focal length at the time of imaging of the imaging device, determine the presence or absence of the image blur for each frame,
The second process is not performed on the frame determined to have no image blur, and the second process is performed on the frame determined to have the image blur. An image processing apparatus.   The image processing apparatus according to claim 1, wherein the motion information is acquired from the moving image on which the second processing has been performed, and the first processing is performed using the motion information.   The second processing means uses the plurality of frames, which are generated at least one before and after the frame determined to have image flow blur, to be determined to have no image flow blur as the second processing unit. The image processing apparatus according to claim 1, wherein:   An imaging apparatus comprising: an imaging system that generates a moving image by imaging; and the image processing apparatus according to claim 1. A first processing step of performing a first process for reducing image blur due to movement of the imaging device between a plurality of consecutive frames in a moving image generated by imaging by the imaging device;
A second processing step of performing a second process of reducing image blur caused by movement of the imaging device for each frame in the moving image;
A movement information acquisition step of acquiring movement information of the imaging device;
When performing the first processing in the first processing step, in the second processing step,
From the motion information acquired by the motion information acquisition step and the frame rate and focal length at the time of imaging of the imaging device, determine the presence or absence of the image blur for each frame,
The second process is not performed on the frame determined to have no image blur, and the second process is performed on the frame determined to have the image blur. Image processing method. On the computer,
A first processing step of performing a first process for reducing image blur due to movement of the imaging device between a plurality of consecutive frames in a moving image generated by imaging by the imaging device;
A second processing step of performing a second process of reducing image blur caused by movement of the imaging device for each frame in the moving image;
An image processing program for executing a process including a movement information acquisition step of acquiring movement information of the imaging device,
In the case of causing the computer to perform the first processing in the first processing step, in the second processing step, in the computer,
From the motion information acquired by the motion information acquisition step and the frame rate and focal length at the time of imaging of the imaging device, the presence or absence of the image blur is determined for each frame,
The second process is not performed on the frame determined to have no image blur, and the second process is performed on the frame determined to have the image blur. An image processing program characterized by that.