JP2007028251A - Image processing apparatus, image processing method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that a frame memory for storing one frame and more is necessary in order to correct camera shake. <P>SOLUTION: A frame image receiving means successively receives frame images in time-series order and a moving vector detection means detects the moving vector of the n-th frame image. A moving vector prediction means predicts the moving vector of the (n+1)th frame image and a segmentation position calculation means calculates a segmentation position indicating a position for segmenting a prescribed image in the (n+1)th frame image by using the predicted moving vector of the (n+1)th frame image. The detection of the moving vector of the n-th frame image, the prediction of the moving vector of the (n+1)th frame image and the calculation of the segmentation position of the (n+1)th frame image are performed during the acquisition of the n-th frame image by the frame image receiving means. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention acquires a frame image from a moving image or a time-series continuous still image including image shake such as camera shake, and a frame image (corrected frame image) obtained by cutting out a predetermined image from the acquired frame image. The present invention relates to an image processing device, an image processing method, a program, and a recording medium on which the program is recorded.

  An image processing technique has been proposed that corrects image shake caused by camera shake when shooting a desired image as a moving image using an image pickup device such as a video camera, digital camera, or mobile phone having an image shooting function. (For example, Patent Document 1).

  Japanese Patent Laid-Open No. 2004-228867 stores a number of frames of images taken with a tube in a frame memory, obtains a motion vector indicating the motion of an image between frames by a motion detection circuit, and positions to extract a corrected frame image from the frame image There is disclosed a television signal processing apparatus that corrects image shake by changing the frequency according to a motion vector.

  Image shake correction is performed by cutting out a partial image (corrected frame image) smaller than the frame image from the frame image and outputting the corrected frame image. When the correction frame image is cut out, the image is not shaken from the output correction frame image by cutting out the correction frame so as to eliminate the image shake.

  FIG. 8 is an explanatory diagram for explaining a motion vector of a frame image. FIG. 8 (upper part) shows frame images F (1), F (2), and F (3) in a moving image, respectively (numbers in parentheses indicate chronological order). The frame images FK (1), FK (2), and FK (3) indicated by broken lines inside the frame images F (1), F (2), and F (3) are the same size as the corrected frame image. This is the image. FIG. 8 (lower side) is a diagram in which the objects shown in the corrected frame images FK (1), FK (2), and FK (3) are arranged at the same position. At this time, the motion vector indicating the motion of the frame image FK (2) relative to the frame image FK (1) is V0 (2), and the motion of the frame image FK (3) relative to the frame image FK (2). Is represented by V0 (3). The motion vectors V0 (2) and V0 (3) indicate the movement of the imaging device caused by camera shake or the like.

  Therefore, in FIG. 8 (upper part), if the positions of the frame images FK (2) and FK (3) are moved in the direction of canceling the motion vectors V0 (2) and V0 (3), the frame image FK (1 ), FK (2), FK (3), the object shown in FIG. That is, a frame image in which image blur is eliminated is obtained. Thus, the frame image in which the motion vectors V0 (2) and V0 (3) are moved in the canceling direction is a corrected frame image.

  Next, image blur correction will be described with reference to FIG. First, the frame image receiving means 110 receives the frame image F (2) obtained from the moving image in chronological order. Next, using the received frame image F (2), the motion vector detecting unit 120 uses the motion vector V0 () indicating the motion of the frame image with reference to the previous frame image F (1) in time series order. 2) is detected. Then, the cutout position calculation unit 130 cancels the motion vector V0 (2) and calculates the cutout position of the corrected frame image that eliminates the image blur. Next, the partial image cutout unit 140 cuts out the correction frame image from the frame image F (2), and the image display unit 150 outputs the correction frame image.

Japanese Patent Publication No. 8-17454

  However, in the process of FIG. 19, the frame image F (2) is used in two places, the motion vector detection means 120 and the partial image cutout means 140. Therefore, after using the frame image F (2) in the motion vector detecting unit 120, the frame memory 113 for storing the frame image F (2) until the partial image cutting unit 140 uses it again is necessary.

  The present invention has been made to solve such a problem, and is an image processing device, an image display device, and an image display device that can suppress the amount of memory for holding a frame image, which is necessary when performing camera shake correction processing, An object is to provide an image pickup apparatus, an image processing method, a program, and a recording medium.

  In order to solve the above-described problem, in the image processing apparatus according to the present invention, (M + 1) (M is an integer of 2 or more) frame images F (1) to F (M + 1) to be displayed, which are continuous in time series. A frame image receiving means for sequentially receiving the frame images F (1) to F (M + 1) that are likely to cause image fluctuations, and a frame image F (p -1) By detecting a motion vector V0 (p) indicating the motion of the frame image F (p) with reference to (1 <p ≦ (M + 1)), the (M + 1) frame images F (1) are detected. Motion vector detection means for obtaining M motion vectors V0 (2) to V0 (M + 1) indicating the motion of M frame images F (2) to F (M + 1) out of F (M + 1); Motion vectors V0 (2) to V0 (M + 1) The overall motion of M frame images F (1) to F (M) indicated by (M-1) motion vectors V0 (2) to V0 (M) is specified by the image shaking. The (M + 1) th frame image F (M + 1) should be displayed on the time axis with X sine waves defined by the frequency divided into X base frequencies (X is an integer). By extending to the time, the predicted motion vector YV0 (M + 1), which is the motion vector of the frame image F (M + 1), indicating the motion of the frame image F (M + 1) with respect to the frame image F (M) is predicted. And a cutout position S () when a partial image HFG (k) that is a part of the frame image F (k) is cut out from the frame image F (k) (k is an integer indicating the time series order). k) cutting position calculation to calculate Means for calculating a cutout position S (M + 1) for attenuating the image fluctuation using the predicted motion vector YV0 (M + 1), and the cutout position S (M + 1). The partial image HFG (M + 1) from the frame image F (M + 1), and prior to cutting out the partial image HFG (M + 1) by the partial image cutout means. Receiving the frame image F (M) of the Mth frame in time-series order by the image receiving means, detecting the motion vector V0 (M) by the motion vector detecting means, and the predicted motion vector by the motion vector predicting means YV0 (M + 1) is predicted, and the cut-out position calculation unit calculates the cut-out position S (M + 1), and the partial image cut-out unit performs the calculation. The gist is that a part of the partial image HFG (M + 1) is cut out in parallel with a part of the detection of the motion vector V0 (M + 1) by the motion vector detecting means.

  According to this, when the partial image HFG (M + 1) starts to be cut out from the (M + 1) th frame image F (M + 1) in the time series order by the partial image cutting means, the Mth in the time series order by the image receiving means. Frame image F (M) of the current frame, detection of motion vector V0 (M) by the motion vector detection means, prediction of predicted motion vector YV0 (M + 1) by the motion vector prediction means, and extraction by the extraction position calculation means Since the calculation of the position S (M + 1) has been completed, the cutting position S (M + 1) required by the partial image cutting means has been calculated.

  For this reason, the partial image cutout means can immediately use the data of the frame image F (M + 1) received by the frame image receiving means. In addition, since the detection of the motion vector V0 (M + 1) by the motion vector detection means operates in parallel with the partial image extraction means, the data of the frame image F (M + 1) received by the frame image reception means is the motion vector detection. It can be used immediately by means. Therefore, it is not necessary to store the frame image F (M + 1) in a frame memory or the like, and the memory usage can be reduced.

  Further, the motion vector prediction unit uses the motion vector detected by the motion vector detection means to determine the motions of the M frame images F (1) to F (M) based on the X bases specified by the image shaking. Decompose into frequencies. Since the X sine waves defined by the X base frequencies are extended to the time for displaying the (M + 1) th frame on the time axis, the frame image F with reference to the frame image F (M) is used. A motion vector indicating (M + 1) motion is predicted with high accuracy.

  In the above-described image processing apparatus according to the present invention, the M frames having an arbitrary frame image as a reference position using the (M-1) motion vectors V0 (2) to V0 (M). The motion vector prediction means further includes motion trajectory calculation means for calculating M motion trajectories X0 (1) to X0 (M) that are trajectories indicating the overall motion of the images F (1) to F (M). The means decomposes the M motion trajectories X0 (1) to X0 (M) into the X base frequencies, and converts the X sine waves defined by the frequencies on the time axis, The predicted motion vector YV0 (M + 1) is predicted by extending the frame image F (M + 1) to the time to be displayed.

  According to this, M motion trajectories X0 (1) to X0 (M), which are trajectories indicating motion on the time axis, are calculated from (M-1) motion vectors V0 (2) to V0 (M). Is done. By representing the motion of the frame image as a trajectory, that is, a coordinate sequence, the calculation for the decomposition into the frequency and the synthesis of the sine wave is simplified, and the processing time can be shortened.

  In the above-described image processing apparatus according to the present invention, it is preferable that the number of base frequencies X is a number different from the M.

  When the number of base frequencies X is the same as the M, the motion of the (M + 1) th and subsequent frame images in the time series order in the calculation of the predicted motion vector YV0 (M + 1) is M frame images F. The periodic nature of the movement of (1) to F (M) is a periodic repetition, and an error occurs. When the number of base frequencies X is different from the M, the problem of periodicity does not occur, and the motion prediction accuracy of the frame image F (M + 1) is further improved.

  The number X of the base frequencies is preferably smaller than the M.

  In this way, the time for displaying the M frame images F (1) to F (M) is longer than the period of the base frequency, and the M frame images F (1) to F (M) are displayed. The information of the frequency component that is the same as or higher than the base frequency is accurately included therein. Therefore, the motion prediction accuracy of the frame image F (M + 1) is further improved.

  The period of the base frequency excluding the DC component among the X base frequencies is the time from the time when the frame image F (1) should be displayed to the time when the frame image F (M) should be displayed. Shorter than that.

  According to this, the periodicity problem does not occur, and information on frequency components that are the same as or higher than the base frequency is accurately included in the M frame images F (1) to F (M). become. Therefore, the motion prediction accuracy of the frame image F (M + 1) is further improved.

  Further, a baseline that is a frequency component lower than the base frequency excluding the direct current component of the X base frequencies from the motion trajectories X0 (1) to X0 (M) calculated by the motion trajectory calculation means is obtained. Baseline calculation means for correcting the removed trajectory as the motion trajectories X0 (1) to X0 (M) is further included, and the motion vector prediction means is the M motion trajectories corrected by the baseline calculation means. X0 (1) to X0 (M) are decomposed into X base frequencies, and the X sine waves defined by the frequencies are converted into the (M + 1) th frame image F ( The coordinate position obtained by extending to the time to display M + 1) and the baseline are extended to the time to display the (M + 1) th frame image F (M + 1) on the time axis. The motion trajectory X0 (M + 1) is predicted by adding the calculated coordinate position, and the motion vector XV (M + 1) of the frame image F (M + 1) is predicted from the motion trajectory X0 (M) and the motion trajectory X0 (M + 1). ).

  According to this, the low frequency component is removed from the base frequency excluding the direct current component from the trajectory to be processed by the predicted motion trajectory calculating means and the process of extending the sine wave on the time axis. Therefore, the corrected motion trajectories X0 (1) to X0 (M) have fewer frequencies lower than the X base frequencies, and thus the prediction accuracy is improved.

  Moreover, it is preferable that the base line is a straight line having an inclination obtained by averaging the inclinations between the frames of the movement trajectories X0 (1) to X0 (M).

  According to this, the base line is a straight line having an inclination between the frames of the motion trajectories X0 (1) to X0 (M), that is, an average of the motion vectors V0 (2) to V0 (M). Therefore, the calculation of the baseline can be simplified.

  Further, before the motion vector predicting means uses the motion trajectories X0 (1) to X0 (M), the motion trajectory X0 is given a greater weight as it is closer to the (M) th frame in time series order. It is preferable to further include motion trajectory weighting means for correcting (1) to X0 (M).

  According to this, the motion trajectory weighting means assigns a larger weight to the motion trajectories X0 (1) to X0 (M) as it is closer to the Mth frame in time series order. Then, the X sine waves obtained by decomposing the motion trajectories X0 (1) to X0 (M) into X base frequencies by the predicted frame image motion calculation means are the Mth frame images in time series order. The motion of the frame image close to is emphasized. For this reason, the motion vector prediction means means that the motion vector predictor YV0 (M + 1) reflects the motion of the frame images that are closer in time series order than the motion of the frame images that are farther in time series order. Is possible.

  The motion trajectory weighting means has a function W (n) in which the value corresponding to the first frame in time series order is closest to 0, and the value corresponding to the Mth frame in time series order is closest to 1. It is preferable to correct the motion trajectories X0 (1) to X0 (M) by multiplying the coordinates of the motion trajectory X0 (n) and the value of W (n) for each frame.

  According to this, the motion trajectory weighting means corrects the motion trajectories X0 (1) to X0 (M) by applying the function W (n) and the motion trajectory X0 (n). Therefore, the movement trajectory can be corrected easily and quickly.

  In addition, image display means for sequentially displaying the partial images HFG (M + 1) in chronological order.

  According to this configuration, an image display device having an effect equivalent to that of the image processing device of the present invention can be obtained.

  In addition, the image processing method according to the present invention provides (M + 1) frame images F (1) to F (M + 1) to be displayed (M is an integer equal to or larger than 2), which are continuous in time series. A frame image receiving step for sequentially receiving the frame images F (1) to F (M + 1) that may cause image shaking, and a frame image F (p−1) (1 < (M + 1) frame images F (1) to F (M + 1) are detected by detecting a motion vector V0 (p) indicating the motion of the frame image F (p) with reference to p ≦ (M + 1)). Among them, a motion vector detecting step for obtaining M motion vectors V0 (2) to V0 (M + 1) indicating motions of M frame images F (2) to F (M + 1), and the M motion vectors V0 ( (M-1) of 2) to V0 (M + 1) X pieces (X is an integer) of the overall movement of the M frame images F (1) to F (M) indicated by the motion vectors V0 (2) to V0 (M) specified by the image shake. The frame image F is decomposed into base frequencies, and X sine waves defined by the frequencies are extended to the time at which the (M + 1) th frame image F (M + 1) is to be displayed on the time axis. A motion vector prediction step of predicting a predicted motion vector YV0 (M + 1), which is a motion vector of the frame image F (M + 1), indicating the motion of the frame image F (M + 1) with reference to (M); A cut-out position calculating step of calculating a cut-out position S (k) when cutting out a partial image HFG (k) that is a part of the frame image F (k) from (k) (k is an integer indicating a time-series order) And said Using the motion vector YV0 (M + 1), a cutout position calculating step for calculating a cutout position S (M + 1) for attenuating the image fluctuation, and using the cutout position S (M + 1), the frame image F ( A partial image cutting step of cutting out the partial image HFG (M + 1) from M + 1), and prior to cutting out the partial image HFG (M + 1) by the partial image cutting step, Receiving the frame image F (M) of the Mth frame in sequence order, detecting the motion vector V0 (M) by the motion vector detection step, and predicting the prediction motion vector YV0 (M + 1) by the motion vector prediction step; And the cut-out position S (M + 1) is calculated by the cut-out position calculating step, and the partial image HFG (M + 1) by the partial image cut-out step is calculated. ) And a part of the detection of the motion vector V0 (M + 1) by the motion vector detection step in parallel.

  According to this configuration, an effect equivalent to that of the image processing apparatus of the present invention can be obtained.

  In addition, the image processing program according to the present invention causes a computer to display (M + 1) (M + 1) frame images F (1) to F (M + 1) (in parentheses) to be displayed in time series. Is an integer indicating a time-series order), and frame image receiving means for sequentially receiving the frame images F (1) to F (M + 1) that may cause image shaking, and a frame image F (p-1) By detecting a motion vector V0 (p) indicating the motion of the frame image F (p) based on (1 <p ≦ (M + 1)), the (M + 1) frame images F (1) to F ( M + 1), M motion vectors V0 (2) to V0 (M + 1) indicating motion of the M frame images F (2) to F (M + 1), and the M motions Vectors V0 (2) to V0 (M + 1) The overall motion of M frame images F (1) to F (M) indicated by (M-1) motion vectors V0 (2) to V0 (M) is specified by the image shaking. The (M + 1) th frame image F (M + 1) should be displayed on the time axis with X sine waves defined by the frequency divided into X base frequencies (X is an integer). By extending to the time, the predicted motion vector YV0 (M + 1), which is the motion vector of the frame image F (M + 1), indicating the motion of the frame image F (M + 1) with respect to the frame image F (M) is predicted. And a cutout position S () when a partial image HFG (k) that is a part of the frame image F (k) is cut out from the frame image F (k) (k is an integer indicating the time series order). k) cutting position calculation to calculate Means for calculating a cutout position S (M + 1) for attenuating the image fluctuation using the predicted motion vector YV0 (M + 1), and the cutout position S (M + 1). A program for causing the partial image HFG (M + 1) to be cut out from the frame image F (M + 1), and cutting out the partial image HFG (M + 1) by the partial image cutting out means Prior to performing the reception of the frame image F (M) of the Mth frame in time-series order by the image receiving means, detection of the motion vector V0 (M) by the motion vector detection means, and the motion vector prediction means Prediction of the predicted motion vector YV0 (M + 1) by the calculation of the cutout position S (M + 1) by the cutout position calculation means The gist is that a part of the partial image HFG (M + 1) cut out by the partial image cutout unit and a part of the detection of the motion vector V0 (M + 1) by the motion vector detection unit are performed in parallel.

  According to this configuration, an effect equivalent to that of the image processing apparatus of the present invention can be obtained.

  The present invention can also be understood as a computer-readable recording medium on which an image processing program is recorded. According to this configuration, an effect equivalent to that of the image processing apparatus of the present invention can be obtained.

  Next, an embodiment of the present invention will be described. FIG. 1 is an explanatory diagram showing a schematic configuration of an image processing apparatus as an embodiment of the present invention. The image processing apparatus is a general-purpose computer 100, and includes an input interface (I / F) 101, a CPU 103, a RAM 102, a ROM 104, a hard disk 105, and an output interface (I / F) 106, which are connected to each other via a bus 107. It is connected. The input I / F 101 is connected to a digital video camera 200 (image capturing device) and a DVD player 210 as devices that input moving images, and the output I / F 106 is a device (image display device) that outputs moving images. Video projector 300 and display 310 are connected. In addition, if necessary, a drive device that can read data from a storage medium that stores a moving image and an image display device that can output a moving image can be connected. Note that moving images include time-sequential still images.

  In addition, the image capturing apparatus and the image display apparatus may each include an image processing apparatus according to the present invention.

  The input I / F 101 acquires a frame image from a moving image having a predetermined number of pixels and a predetermined pixel value such as an RGB format, and converts it into frame image data that can be processed by the computer 100. In the present embodiment, it is assumed that frame image data is acquired from a predetermined number of field images constituting the frame image. The frame image is usually composed of a plurality of predetermined number of field images according to the recording method. For example, in the case of a moving image recorded by the NTSC system, it is an interlace system, and 60 frame images per second constitute 30 frame images per second. Therefore, the input I / F 101 acquires one frame image using the pixel values of the two field images. Of course, in the case of a progressive method (non-interlace method) in which one field image constitutes one frame image, one frame image is acquired using one field image.

  The converted frame image data is stored in the RAM 102 or the hard disk 105. The CPU 103 executes predetermined image processing on the stored frame image data, and stores it again in the RAM 102 or the hard disk 105 as corrected frame image data. In the case of predetermined image processing performed until the corrected frame image data is generated, the RAM 102 and the hard disk 105 are used as a working memory for the image processing data as necessary. Then, the stored corrected frame image data is converted into predetermined image data via the output I / F 106 and sent to the video projector 300 or the like.

  An application program that records predetermined image processing may be stored in advance in the hard disk 105 or the ROM 104. For example, the recording medium may be supplied from the outside by a computer-readable recording medium such as a CD-ROM and stored in the hard disk 105 via a DVD-R / RW drive (not shown). Of course, the data may be stored in the hard disk 105 by accessing a server or the like that supplies an application program via network means such as the Internet and downloading data.

  The CPU 103 functions as an image processing apparatus by reading an application program stored in the hard disk 105 or the ROM 104 via the bus 107 and executing the read application program under a predetermined operating system. FIG. 2 shows a functional block diagram of an image processing apparatus as an embodiment of the present invention. As shown in FIG. 2, by executing this application program, the CPU 103 causes the frame image receiving means 110, the motion vector detecting means 120, the motion trajectory calculating means 121, the baseline calculating means 122, the motion trajectory weighting means 123, It functions as a motion vector prediction unit 124, a memory 126, a cutout position calculation unit 130, a partial image cutout unit 140, and an image display unit 150.

Each part manages the following processes. The frame image receiving unit 110 acquires frame images from the moving image input to the computer 100 in chronological order. The motion vector detection unit 120 calculates a motion vector of the acquired frame image, stores it in the memory 126, and supplies it to the motion trajectory calculation unit 121. The motion trajectory calculation means 121 calculates a motion trajectory indicating the motion of the frame image for the past M frames from the current frame using the motion trajectory for the past M frames from the current frame. The baseline calculation unit 122 calculates a baseline that is a low frequency component of the motion trajectory, and corrects the motion trajectory calculated by the motion trajectory calculation unit 121 by removing the baseline component as a motion trajectory. Next, the motion trajectory weighting means 123 calculates a window function W (n) that is 1 near the current frame and approaches 0 as it moves away from the current frame with respect to the motion trajectory corrected by the baseline calculation means 122. Use to correct the motion trajectory by weighting the motion trajectory. With this processing, the closer to the current frame the closer to the current trajectory, the closer to the trajectory before correction, and the closer to the current frame, the less trajectory the amplitude is. Next, the motion vector prediction unit 124 decomposes the motion trajectory into X base frequencies. Then, the motion trajectory can be expressed by superposition of sine waves defined by the base frequency. Next, by extending this sine wave to the time of the next frame of the current frame, the motion trajectory of the frame image of the next frame is predicted. Then, the baseline component is added to the motion trajectory, and the motion vector of the next frame is predicted from the motion trajectory of the current frame and the predicted motion trajectory of the next frame.
The cut-out position calculating unit 130 calculates an image position for cutting out a predetermined image from the frame image as a correction position vector using a motion vector or the like. The partial image cutout unit 140 generates a correction frame image based on the calculated correction position vector. The image display means 150 converts the generated corrected frame image into predetermined image data and outputs it.

  Next, processing performed by each unit in the image processing apparatus according to the present embodiment will be described with reference to a processing flowchart of FIG. When the process shown in FIG. 3 is started, first, in step S10, the frame image receiving unit 110 performs a process of fetching the frame image F (n) into the computer 100 in order of time series from the input moving image. In the following description of the image processing apparatus according to the present embodiment, (n) means the nth (n is an integer of 1 or more) in time series order. In FIG. 3, m is an integer, and the initial values of n and m are n = 1 and m = 0, respectively.

  In this embodiment, it is assumed that frame images are automatically captured sequentially in time-series order with the input of a moving image as a trigger. Of course, the frame image to be captured may be designated. As a method for specifying a frame image, an identification number of a frame image that is normally added may be specified when a frame image is acquired from a moving image and imported into the computer 100. Alternatively, the user may directly designate an image while viewing a frame image displayed on a display (not shown) such as a monitor provided in the video camera.

  Next, in step S <b> 20, the frame image receiving unit 110 determines whether or not there is a frame image F (n) acquired by being taken into the computer 100. If the frame image F (n) acquired from the moving image exists (YES), the process proceeds to step S24. On the other hand, if the frame image F (n) acquired from the moving image does not exist (NO), the process proceeds to step S22, m is updated (m = m + 1), and the process proceeds to step S24.

  In step S24, it is determined whether or not m = 2. If m = 2 (S24, YES), the processing in the image processing apparatus ends.

  On the other hand, if m = 2 is not satisfied (S24, NO), (A) the processes in S26, S30, S35 and S40, (B) the processes in S28 and S50, and (C) the processes in S65 and S70 are performed in S10. This is done in parallel while reading information for one frame.

First, the processes in S26, S30, S35, and S40 will be described.
(A) Processing in S26, S30, S35, and S40 In step S26, it is determined whether m = 1 or not. If m = 1 (S26, YES), the process proceeds to step S80.

  On the other hand, if m = 1 is not satisfied (S26, NO), in step S30, the motion vector detection means 120 performs a process for detecting the motion vector V0 (n) of the acquired frame image F (n). The relative position of the frame image F (n) with respect to the n−1th frame image F (n−1) in time series order is detected, and the frame image F (n) is arranged at the relative position. Then, the shift amount of the screen position of the frame image F (n) with respect to the screen position of the frame image F (n−1) is detected as the number of pixels in the X and Y directions. A motion vector V0 (n) of the frame image F (n) is represented by a vector having the number of pixels in the X direction as an X component and the number of pixels in the Y direction as a Y component. Data V0 (n) of the calculated motion vector V0 (n) is stored in the memory 126 (such as the RAM 102).

  Next, with reference to FIGS. 4 and 5, the motion vector detection processing of the nth frame image in S30 of FIG. 3 by the motion vector detection means 120 will be described in detail. Here, FIG. 4 is a flowchart for explaining in detail the motion vector detection processing of the n-th frame image in S30 of FIG. FIG. 5 is an explanatory diagram for explaining in detail the motion vector detection processing of the nth frame image in S30 of FIG.

  First, the motion vector detection unit 120 resets a difference value matrix described below with reference to FIG. 5 (step S300). That is, all the component values of the difference value matrix are set to zero.

  Next, the motion vector detection means 120 reads the coordinate data of the difference calculation area shown in FIG. 5 (step S302). That is, the motion vector detection unit 120 reads from the memory 126 coordinate data indicating the range of the difference calculation area in the frame image set for each sampling point. In the present embodiment, the “difference calculation area” is a square area indicated by a dotted line in FIG. 5 and is an area for performing the processing in steps S310 to S316 described below. Each process will be described later.

  Next, the motion vector detection means 120 reads data at the sampling point of the (n−1) th frame image (step S304). That is, the motion vector detection unit 120 reads the coordinate data and pixel data of the sampling point (shown in FIG. 5) in the (n−1) th frame image from the memory 126. In the present embodiment, the pixel data to be read is a value of one of RGB channels or a luminance value.

  Next, the motion vector detection means 120 reads each pixel data (input signal) and its coordinate data in the nth frame image (step S306). Here, the pixel data to be read is a value or luminance value of one of the RGB channels, and the coordinate data indicates the position of each pixel in the frame image. As shown in FIG. 5, the order of reading is read from the upper left of the frame image to the right lateral direction, and sequentially read downward line by line. Eventually, it reaches the lower right point EP2 of the frame image, and the reading process is terminated.

  Next, the motion vector detection means 120 determines whether or not each pixel read in step S306 exists in the difference calculation area (step S308). Specifically, the coordinate data of the pixel read in step S306 and the coordinate data of the difference calculation area read in step S302 are compared, and it is determined whether or not the pixel read in step S306 exists in the difference calculation area. To do.

  If the result of determination is that the pixel read in step S306 is outside the difference calculation area (No in step S308), the process returns to step S306 to read the next pixel.

  As a result of the determination, when the pixel read in step S306 is present in the difference calculation area (step S308, Yes), the motion vector detection unit 120 reads the pixel coordinate data read in step S306 and the pixel data read in step S304 ( It is determined whether or not the coordinate data of the sampling point in the n-1) th frame image matches (step S310). If the coordinate data of the pixel read in step S306 matches the coordinate data of the sampling point in the (n−1) th frame image read in step S304 as a result of the determination (step S310, Yes), motion vector detection is performed. The means 120 stores the pixel data of the pixels in the nth frame image read in step S306 in the memory 126 as sampling data (step S312).

  Then, after the process of step S312, or as a result of the determination in step S310, the coordinate data of the pixel read in step S306 and the coordinate data of the sampling point in the (n−1) th frame image read in step S304. If they do not match (No in step S310), the motion vector detection means 120 calculates the absolute value of the difference between the sampling point data in the (n-1) th frame image and the pixel data in the nth frame image. (Step S314), the calculated absolute value is added to the corresponding component of the difference value matrix (Step S316).

  In this embodiment, as shown in FIG. 5, the difference value matrix is configured to store the same number of data as the total number of pixels in the difference calculation area, and the coordinates of the center of the difference value matrix correspond to the coordinates of the sampling points. To do. The component to which the absolute value of the difference value is added is a component at the same position as the pixel coordinate read in step S306 in the positional relationship with respect to the sampling point in the difference value matrix.

  The motion vector detecting means 120 determines whether or not reading of all the pixels in the difference calculation area is completed (step S318), and the coordinates of the pixel read in step S306 are “difference value calculation is completed” in FIG. It is determined whether or not the point is “EP1”. When reading of all the pixels in the difference calculation area has not been completed (No at Step S318), the motion vector detection unit 120 repeats the processes at Steps S306 to S316.

  When the reading of all the pixels in the difference calculation area is completed and the coordinates of the pixel read in step S306 have reached the “point EP1 at which the difference value calculation ends” in FIG. 5 (step S318, Yes), the frame image receiving means 110 continues reading the remaining pixels in the nth frame image, and the motion vector detection means 120 detects the motion vector of the nth frame image (step S320). That is, the motion vector detection unit 120 obtains a position (vector) from the sampling point of the minimum value component in the difference value matrix, detects the position (vector) as a motion vector of the nth frame image, and Save to 126.

  Thus, the motion vector detection process for the nth frame image in step S30 in FIG. 3 is completed, and the motion vector prediction process for the (n + 1) th frame image in step S35 in FIG. 3 is performed. Using the flowchart for explaining the motion vector prediction process of the (n + 1) th frame image in FIG. 6 and the explanatory diagram for explaining the motion vector prediction process by the motion trajectory in FIG. The motion vector prediction process of the frame image will be described.

  First, the motion trajectory calculating means 121 reads the motion vectors V0 (n−M + 1) to V0 (n) of M frame images from the (n−M + 1) th to the nth from the memory 126 (step S340). Further, since the frame image starts from the first in the time series order, the motion vector of the frame image before the second in the time series order is not calculated. Therefore, when (n−M + 1) ≦ 1, it is possible to read the M motion vectors (n−M + 1) to V0 (n) with V0 (q) = 0 (q ≦ 1) and the motion trajectory calculation unit 121. Like that. M is a number determined in advance by the frequency of image shake such as camera shake, and is set to 8 in the present embodiment. Next, the motion trajectory calculation means 121 uses the equation (1) to obtain motion trajectories X (−M + 1) to X (0) indicating the coordinates of the M frame images from the (n−M + 1) th to the nth. Calculate (step S342). The motion trajectory X (i), modified motion trajectories X1 (i), X2 (i), X3 (i), X4 (i), which will be described later, the window function W (i), and the baseline B (i The integer i in parentheses in) is an integer indicating the time-series order of the frame images. i is an integer indicating a time-series order in which the origin is different from the time-series order used in the frame image F (n) and the n-th frame image F (n) is the origin 0. That is, the coordinates indicated by the movement locus X (0) indicate the coordinates of the frame image F (n). In the following description, the time-series order with the nth frame image F (n) as the origin 0 is referred to as time-series order i, and this time-series order is indicated using a variable i, such as n. To distinguish.

  Expression (1) calculates the motion trajectory by calculating the coordinates of the M frame images from the M motion vectors, and the origin of the coordinates of each frame image is the frame image F (n−M). Also, using equation (2), the motion trajectory X (i) may be calculated for i in the range of (−M + 2) ≦ i ≦ 0 with the frame image F (n−M + 1) as the origin.

  In this embodiment, the motion of the frame image F (n + 1) of the next frame that has not yet been acquired is predicted from the motion of M frame images, and image shake such as camera shake is corrected based on the predicted motion. Therefore, in order to sufficiently remove image shake such as camera shake, it is necessary to increase the prediction accuracy of the frame image F (n + 1). The prediction accuracy of the frame image F (n + 1) depends on whether image shake information such as camera shake can be accurately extracted from the movements of the M frame images. In general, it is said that the frequency of unpleasant image shaking such as camera shake is a motion with a frequency higher than 2 to 4 Hz. Therefore, the time for displaying M frame images (hereinafter referred to as reference time) may be determined based on the fluctuation period (1/2 to 1/4) of the image to be removed. In the case of a movie with a frame rate of 30 frames / second, it is better to set M = 8 to 16 (reference time = 0.27 seconds to 0.53 seconds), and among these, M = 8 is optimal. . As described above, the value of M may be determined from the frequency of image shaking to be removed and the frame rate.

  Next, the baseline calculation unit 122 calculates the baseline B (i) of the movement trajectories X (−M + 1) to X (0), and the baseline B ( The motion trajectories X1 (−M + 1) to X1 (0) from which the component i) is removed are calculated (step S344). The baseline is expressed as shown in Expression (3) by calculating an average vector VB of M motion vectors V0 (n−M + 1) to V0 (n) as shown in Expression (4). The average vector VB may be calculated from the average of (M−1) motion vectors V0 (n−M + 2) to V0 (n).

  B (i) = X (0) + i × VB (3)

  FIG. 7A is a graph showing the movement trajectories X (−M + 1) to X (0) and the baseline B (i). From FIG. 7A, it can be seen that the baseline B (i) is a large movement of the movement locus X (i), that is, a low-frequency movement locus. For the baseline B (i), an equation that can represent a low frequency component of the motion trajectory X (i) different from the equation (3) may be used. Next, the baseline calculation means 122 removes the component of the baseline B (i) from the motion trajectories X (−M + 1) to X (0) using the equation (5), and corrects the motion trajectory X1 (i ) Is calculated.

  X1 (i) = X (i) -B (i) (5)

  In the corrected motion trajectory X1 (i), since the large motion represented by the baseline B (i) is removed, the shake component of the high-frequency image derived from camera shake is emphasized. Become. Therefore, by using the corrected motion trajectory X1 (i), it is possible to accurately calculate the motion related to the shaking of the high-frequency image derived from camera shake or the like.

Next, the motion trajectory weighting means 123 weights the motion trajectory X1 (i) using the window function represented by Equation (6) as shown in Equation (7), and weighted motion trajectory X2 (i ) Is calculated (step S346).
W (i) = exp {− (2i / M) ² } (6)
X2 (i) = X1 (i) × W (i) (7)

  FIG. 7B is a graph showing the shape of the window function. The window function has a shape in which the value of the window function W (0) corresponding to the newest frame image F (n) is 1, and approaches 0 as it goes from the old frame in time series order. FIG. 7C shows a motion trajectory X2 (i) obtained by weighting the corrected motion trajectory X1 (i) with a window function using Expression (7). As is clear from FIG. 7C, the weighted motion trajectory X2 (i) has a smaller amplitude for an older frame and a larger amplitude for a new frame.

  Next, the motion vector predicting unit 124 decomposes the weighted motion trajectory X2 (i) into X base frequencies using Equation (8), Equation (9), and Equation (10), and X This is expressed by superposing the sine waves (step S348). X3 (i) in Expression (8) is a motion locus expressed by superposition of sine waves.

  Since the weighted motion trajectory X2 (i) is weighted such that the closer the frame is to the nth frame in time series order, the larger the amplitude is, the xth sine wave is also nth in time series order. The movement of the large amplitude of the frame close to this frame is strongly reflected. For this reason, the X sine waves contain many frame image motions close to (n + 1) in time series order, that is, many image shakes due to camera shake in a time zone close to (n + 1). Therefore, the prediction accuracy of the position of the next frame predicted from X sine waves is improved.

  The X base frequencies are 0 (Hz), f / X (Hz), 2f / X (Hz),..., (X-1) f /, where f (frame / second) is the frame rate. X (Hz). If X / f, which is the base period when u = 1, is equal to the reference time M / f, periodicity is generated in Equation (8), and the prediction accuracy of the position of the (n + 1) -th frame image is high. Deteriorate. Therefore, it is better that X and M have different values. Further, if X / f, which is the base period when u = 1, is shorter than the reference time M / f, the base frequency is included in the reference time M / f by one period or more. Prediction accuracy is improved. From the above, in Equation (8), in order to improve the prediction accuracy of the position of the (n + 1) th frame image, it is better that X ≠ M, and the basis of when u = 1 is longer than the reference time. It is even better if the period is long, ie X <M. In this embodiment, since M = 8, the number of base frequencies X is preferably 7 or less.

  Next, the motion vector predictor 124 extends each sine wave in the time direction to the next frame (the (n + 1) th frame in time series order) using the equation (11), and adds the sine wave of each frequency. (Step S350). X3 (1) in Expression (11) is a coordinate extended to the (n + 1) th frame in time series order. At this time, since the origin 0 of the time series order i is set to the nth frame, the calculation of the trigonometric function can be omitted, and the calculation is simplified as shown in Expression (11). FIG. 7D is a graph showing the X sine waves that have been decomposed and how each sine wave is extended to the next frame.

  Next, the motion vector prediction means 124 uses the equation (12) to calculate the motion trajectory X4 (1) of the (n + 1) th frame from the extended motion trajectory X3 (1) and the baseline B (i). Prediction is made (step S352). FIG. 7E is a diagram showing the synthesized sine wave, the baseline, and the motion locus X4 (1) of the (n + 1) th frame.

  Finally, using equation (13), a motion vector V0 (n + 1) indicating the motion of the (n + 1) th frame image with respect to the nth frame image is calculated (step S354).

  Thus, the motion vector prediction process for the (n + 1) th frame image in step S35 in FIG. 3 is completed, and the correction position vector S (n + 1) calculation process for the (n + 1) th frame image in step S40 in FIG. Is done.

  Here, regarding the calculation processing of the corrected position vector S (n + 1) in the next step S40 and the calculation processing of the corrected motion vector V1 (n) in the following step S420 (FIG. 10), it is easy to understand the explanation of the processing contents. Therefore, the relationship between the motion vector V0 (n) of the frame image F (n), the corrected position vector S (n), and the corrected motion vector V1 (n) will be described in advance with reference to FIGS.

  FIG. 8 shows the first frame image F (1), the second frame image F (2), and F (3) acquired in the time series order first. The present invention provides an image processing technique for generating a corrected frame image by a correction process for cutting out a predetermined image from a frame image based on a correction amount, and when the correction process is not performed or when the correction amount is zero. The frame images cut out in this case are frame images FK (1), FK (2), and FK (3) shown by broken lines in FIG.

  Here, as shown in FIG. 8, the frame image FK (n) has a screen size smaller than the frame image F (n) by a predetermined correction limit value Smax. Specifically, the image has a small screen size by a predetermined correction limit value Sxmax in the left and right directions in the X direction and by a predetermined correction limit value Symax in the upper and lower directions in the Y direction. That is, the image to be cut out has a screen size that is 2 × Sxmax smaller in the X direction and 2 × Symax smaller in the Y direction.

  In this embodiment, the predetermined correction limit value Smax (X component is Sxmax, Y component is Symax) is set in advance as a default value in the application program and stored in the hard disk 105 or the like. Of course, the user may set and store the data by inputting data using an input means (not shown) connected to the computer 100.

  Similarly, FIG. 8 (lower side) shows frame images FK (1), FK (2), and FK (3) arranged according to the obtained relative positions. As apparent from FIG. 8, for example, the motion vector V0 (2) of the frame image F (2) is equal to the motion vector of the frame image FK (2). That is, the motion vector V0 (2) of the frame image F (2) is obtained as the screen position of the frame image FK (2) with respect to the frame image FK (1). Similarly, the motion vector V0 (3) of the frame image FK (3) is obtained as the screen position of the frame image FK (3) with respect to the frame image FK (2). The X components of the motion vectors V0 (2) and V0 (3) are represented by the number of pixels in the X direction, and the Y component is represented by the number of pixels in the Y direction.

  In FIG. 8, since the frame image F (1) has no previous frame image in chronological order, in the present embodiment, the frame image F (1) is treated as not moving, and the frame image FK (1 The motion vector V0 (1) in FIG.

  Next, the corrected position vector S (n) and the motion vector V0 (n) of the frame image F (n) and the corrected motion vector V1 (n) calculated in step S420 (FIG. 10) are used with reference to FIG. Explain the relationship. FIG. 9 is an explanatory diagram showing an enlarged upper left corner portion of the frame images FK (1) to FK (3) shown in FIG. 8 (lower part). Here, the corrected frame image HFG (2) indicates a state in which the frame image FK (2) is corrected by the correction position vector S (2) of the frame image F (2) calculated in step S40 (FIG. 3). ing.

  As is apparent from FIG. 9, the frame image FK (2) is obtained by moving the screen position of the frame image FK (1) according to the number of pixels of the X component and Y component of the motion vector V0 (2). The image HFG (2) is obtained by moving the screen position of the frame image FK (2) according to the number of pixels of the X component and the Y component of the correction position vector S (2). Similarly, the frame image FK (3) is obtained by moving the screen position of the frame image FK (2) according to the number of pixels of the X and Y components of the motion vector V0 (3), and the corrected frame image HFG ( 3) shows the screen position of the frame image FK (3) moved according to the number of pixels of the X and Y components of the correction position vector S (3).

  As shown in FIG. 9, the corrected motion vector V1 (2) indicates a relative screen position of the corrected frame image HFG (2) with respect to the frame image FK (1), and the corrected motion vector V1 ( 3) shows a relative screen position of the correction frame image HFG (3) with respect to the correction frame image HFG (2). Since the motion vector V0 (1) is 0, the correction position vector S (1) is also 0 (described later). Therefore, the frame image FK (1) is the same as the corrected frame image HFG (1).

  Here, in the motion vector V0 (n) and the corrected motion vector V1 (n), the X component is positive in the drawing right direction and the Y component is positive in the drawing upward direction, but the corrected position vector S (n) is moving. Contrary to the motion vector, the X component is positive in the left direction of the drawing, and the Y component is positive in the downward direction of the drawing. Accordingly, the correction position vectors S (2) and S (3) of the frame images FK (2) and FK (3) are opposite to the motion vectors V0 (2) and V0 (3) as shown in FIG. It goes in the positive direction.

  In this embodiment, the motion prediction vector of the (n + 1) th frame image obtained in S35 is used instead of the motion vector V0 (n + 1).

  Next, returning to FIG. 3, the cut-out position calculation means 130 performs a correction position vector S (n + 1) calculation process in step S <b> 40. The process in step S40 will be described in detail with reference to the flowchart of FIG.

  When the process of step S40 is started, first, in step S410, a process of inputting the predicted motion vector V (n + 1) of the target frame image F (n + 1) obtained in S35 is performed. Here, the CPU 103 reads out the predicted motion vector V (n + 1) obtained in step S35 and stored in the memory 126 such as the RAM 102, and performs input processing.

  Next, in step S420, the corrected motion vector V1 (n) of the previous frame image (hereinafter simply referred to as “previous frame image”) in time series order with respect to the frame image F (n + 1) of interest is obtained. Processing to calculate is performed. The corrected motion vector V1 (n) is calculated using the following equation as a vector of the X component V1x (n) and the Y component V1y (n).

V1x (n) = V0x (n) -Sx (n)
V1y (n) = V0y (n) -Sy (n)
It is calculated as described with reference to FIGS. 8 and 9 and is stored in the memory 126 such as the RAM 102.

  In step S430, a process of inputting the correction position vector S (n) of the frame image F (n) is performed. Specifically, the corrected position vector S (n) calculated in step S40 (FIG. 3) while the previous frame image F (n−1) is input and stored in the memory 126 such as the RAM 102. Are read out by the CPU 103 to perform input processing.

  In step S420 and step S430, in the case of “n = 1”, that is, the first frame image in time series order, the previous frame image does not exist. In this case, as described above, the frame image F (n) is not moved in the present embodiment, and as a result, the corrected motion vector V1 (1) is 0 and the corrected position vector S (1) is also 0. .

  Next, in step S440, attenuation coefficient K and D values are determined for each of the X and Y components. The attenuation coefficients K and D are coefficients used in a predetermined calculation formula used in the next processing step S490, and are coefficients that determine the size of the correction position vector, that is, the correction amount. The attenuation coefficients K and D are both decimal numbers of 1 or less.

In this embodiment, the values of the attenuation coefficients K and D are determined using at least one of the following three examples.
“First Example of Decay Factor Determination Method”
The values of the attenuation coefficients K and D depend on the magnitude of the movement amount of the image calculated for the correction frame image obtained by correcting the previous frame image (hereinafter simply referred to as “pre-correction frame image”) and the movement amount of the frame image of interest. To decide. Further, when the movement amount of the frame image of interest is equal to or greater than the movement amount of the previous correction frame image, the attenuation coefficient is based on the direction of the predicted motion vector of the frame image of interest and the direction of the correction motion vector of the previous correction frame image. A method for determining the values of K and D.
“Second Embodiment of Damping Coefficient Determination Method”
A method of determining the values of the attenuation coefficients K and D by determining the ratio of the correction amount of the previous frame image to the correction limit value indicating the correctable range and determining whether the correction amount of the previous frame image is “increase” or “decrease”. .
“Third embodiment of damping coefficient determination method”
What percentage of the difference between the predicted motion vector of the frame image of interest and the corrected motion vector of the previous frame image with respect to the difference between the correction limit value that is the correctable range and the correction amount of the previous frame image A method of determining the values of the attenuation coefficients K and D depending on whether or not.

“First Example”
First, regarding the processing performed in step S440 (FIG. 10), details of the determination processing of the attenuation coefficients K and D in the first embodiment will be described using the processing flowchart shown in FIG. The method for determining the values of the attenuation coefficients K and D in the first embodiment is based on the size and orientation of the predicted motion vector of the frame image of interest and the size and orientation of the corrected frame image one time before in the chronological order. The correction amount is calculated, and the correction amount of the frame image to be noticed is calculated in accordance with the direction change and the magnitude of the predicted motion vector. Here, the processing for the X component of the frame image of interest will be described. However, in processing step S440, the same processing as the processing for the X component shown in FIG. Determine the values of the damping coefficients K, D for. Then, in the next processing step S490 (FIG. 10) to be described later, when the X component and the Y component of the correction position vector are calculated by a predetermined arithmetic expression, the values of the determined attenuation coefficients K and D are used.

  When the process is started, first, in step S441, an X component Vx (n + 1) extraction process of the predicted motion vector V (n + 1) of the target frame image obtained in S35 is performed. The X component Vx (n + 1) is indicated by the number of pixels in the X direction, and is positive if the vector direction is the right direction of the drawing as shown in FIG. Note that the Y component Vy (n + 1) of the predicted motion vector V (n + 1) of the frame image of interest is positive if the vector direction is upward in the drawing as shown in FIG. 9, and negative if downward. .

  Next, in step S442, the X component V1x (n) of the corrected motion vector V1 (n) of the previous frame image in time series order is extracted from the frame image of interest. Of course, as described above, V1x (0) = 0.

  Next, in step S443, the absolute value of Vx (n + 1) and the absolute value of V1x (n) are calculated. Then, in the next processing step S444, it is determined whether or not the absolute value of Vx (n + 1) is greater than or equal to the absolute value of V1x (n). If NO in step S444, that is, if the moving amount of the focused frame image is smaller than the moving amount of the previous correction frame image, the process proceeds to step S445, where the values of the attenuation coefficients K and D are determined as K1 and D1, and step S490 is performed. Proceed to (FIG. 10).

  On the other hand, if “YES” in the step S444, that is, if the moving amount of the focused frame image is equal to or larger than the moving amount of the previous correction frame image, the process proceeds to a step S446 to calculate the product BSx of Vx (n + 1) and V1x (n). . Then, in the next processing step S447, it is determined whether BSx is 0 or more. That is, it is checked here whether the vector Vx (n + 1) and the vector V1x (n) are in the same direction (0 or more) or in the opposite direction (less than 0).

  If NO in step S447 (that is, in the reverse direction), the process proceeds to step S448, where the values of the attenuation coefficients K and D are determined as K2 and D2, and the process proceeds to step S490 (FIG. 10). Here, the value of K2 is larger than the value of K1, and the value of D2 is larger than the value of D1.

  On the other hand, if YES in step S447 (that is, in the same direction), the process proceeds to step S449, where the values of the attenuation coefficients K and D are determined as K3 and D3, and the process proceeds to step S490 (FIG. 10). Here, the value of K3 is larger than the value of K2, and the value of D3 is larger than the value of D2.

  An example of the values of the attenuation coefficients K and D in the first embodiment is shown in the attenuation coefficient table GKT1 in FIG. Here, the value of the attenuation coefficient K is desirably a value satisfying K1 <K2 <K3 in a range of 0.9 ≦ K ≦ 1. Further, the value of the attenuation coefficient D is desirably a value satisfying D1 <D2 <D3 in a range of 0.5 ≦ D ≦ 1. When the movement amount Vx (n + 1) of the frame image to be corrected is smaller than the movement amount V1x (n) of the previous correction frame image, the movement amount Vx (n + 1) of the frame image to be corrected is the movement amount of the previous frame image. It means that it is decreasing with respect to V1x (n). Therefore, suppose that the correction amount Sx (n + 1) of the frame to be corrected so that the movement amount V1x (n + 1) of the correction frame image of the frame image to be corrected is the same as the movement amount V1x (n) of the previous frame image. ) Is designated (that is, K = D = 1), the probability that the correction amount Sx (n + 1) reaches the correction limit value is high. Therefore, a small amount of attenuation coefficient is used to reduce the correction amount and make it difficult to reach the correction limit value.

  Even if the movement amount of the frame image to be corrected is equal to or larger than the movement amount of the previous correction frame image, if the movement amounts of these two frame images are in different directions, the frame image to be corrected The correction amount Sx (n + 1) of the frame to be corrected is designated so that the movement amount V1x (n + 1) of the correction frame image is the same as the movement amount V1x (n) of the previous frame image (that is, K = D = 1) ), The probability that the correction amount Sx (n + 1) reaches the correction limit value is high. Therefore, the amount of correction of the frame image to be corrected is reduced by reducing the value of the attenuation coefficient compared to the case of the same direction. The values of the attenuation coefficients K1 to K3 and D1 to D3 are not limited to the present embodiment, and any values may be used as long as the above-described conditions are satisfied.

"Second Example"
Next, details of the determination processing of the attenuation coefficients K and D in the second embodiment will be described using the processing flowchart shown in FIG. In the method of determining the values of the attenuation coefficients K and D in the second embodiment, the correction amount is calculated based on a predetermined ratio RS as to whether or not it is close to the correction limit value, and further, the correction amount is increased by a predetermined arithmetic expression. It is intended to calculate the correction amount of the frame image to be noticed by determining whether it is reduced or not. Here, the processing for the X component of the frame image of interest is also described. However, in processing step S440, the same processing as the processing for the X component shown in FIG. Determine the values of the damping coefficients K, D for. Then, in the next processing step S490 (FIG. 10) to be described later, when the X component and the Y component of the correction position vector are calculated by a predetermined arithmetic expression, the values of the determined attenuation coefficients K and D are used. is there.

  When the processing here is started, input processing of the correction limit value Sxmax is first performed in step S451. More specifically, the CPU 103 reads the value of the Smax X component stored in the hard disk 105 or the like.

  Next, in step S452, the X component Sx (n) of the correction position vector S (n) of the frame image F (n) is extracted. Specifically, the corrected position vector S (() calculated in step S40 (FIG. 3) while the previous frame image F (n-1) is input in S10 and stored in the memory 126 such as the RAM 102 is stored. n) is read by the CPU 103 to perform input processing. Here, as described above, the X component Sx (n) is represented by the number of pixels with the left direction on the screen as the positive direction. Of course, the Y component Sy (n) is represented by the number of pixels with the screen lower direction being positive.

  Next, in step S453, the value of the ratio RS obtained by dividing the absolute value of Sx (n) by Sxmax is calculated. The larger the RS value, the closer the correction amount is to the correction limit value.

  Next, in step S454, an X component Vx (n + 1) extraction process of the predicted motion vector V (n + 1) obtained in S35 is performed. The X component Vx (n + 1) is indicated by the number of pixels in the X direction, and is positive if the vector direction is the right direction of the drawing as shown in FIG. As shown in FIG. 9, the Y component Vy (n + 1) of the predicted motion vector V (n + 1) is positive if the direction of the vector is upward in the drawing and negative if it is downward.

  Next, in step S455, an X component V1x (n) extraction process of the corrected motion vector V1 (n) of the frame image F (n) is performed. As described above, when n = 1, V1x (1) = 0.

  Using each vector extracted in steps S452, S454, and S455, in the next step S456, a process for calculating the tendency value KHx from the equation (14) is performed.

KHx = Sx (n) × (Vx (n + 1) −V1x (n)) (14)
Next, in step S457, it is determined whether or not the tendency value KHx is 0 or more. If YES, the correction is determined to be “increase” in step S459, and if NO, the correction is determined to be “decrease” in step S458. Sx (n), that is, the correction direction based on the correction position vector of the previous frame image, and Vx (n + 1) −V1x (n), that is, the image based on the difference between the predicted motion vector of the target frame image and the correction vector of the previous frame image. Compared with the moving direction, it is determined that the direction is increased if the direction is opposite, and decreased if the direction is the same.

  Next, processing for determining the values of the attenuation coefficients K and D using the attenuation coefficient table GKT2 is performed in step S460 from the calculated ratio RS and the determination information of “increase” or “decrease”. An example of the attenuation coefficient table GKT2 is shown in FIG. The attenuation coefficient table is stored in the hard disk 105 or the like. From the data of the ratio RS and the information of “increase” or “decrease”, the CPU 103 reads the value of the corresponding address from the attenuation coefficient table GKT2, and stores the attenuation coefficients K and D. Determine the value.

  As shown in FIG. 14, in the second example, in the attenuation coefficient table GKT2, the value of the ratio RS is rounded off to the second decimal place to a value of 0.1. Then, the values of the attenuation coefficients K and D were set so as to become smaller values than the previous values as the ratio RS became larger. Furthermore, the value corresponding to the “increase” information is set to a value smaller than the value corresponding to the “decrease” information. Even when the ratio RS is the same, if the increase is determined, the correction direction and the moving direction of the image are opposite to each other. , D is reduced to reduce the correction amount. Of course, the value of the ratio RS may be a finer step size, or conversely, a coarser step size. Furthermore, the step width is not the same and can be changed.

  The attenuation coefficient table GKT2 shown in FIG. 14 shows an example of specific numerical values for the attenuation coefficients K and D. The value of the attenuation coefficient K is in the range of 0.9 ≦ K ≦ 1, and the attenuation coefficient The value of D may be changed in accordance with the value of RS in the range of 0.5 ≦ D ≦ 1. At this time, it is preferable that the values of the attenuation coefficients K and D are set to be smaller than the previous values as the value of RS becomes larger. When the RS value is large, the correction amount is closer to the correction limit value than when the RS value is small. Therefore, the attenuation coefficient is reduced to reduce the correction amount.

“Third Example”
Next, details of the process of determining the attenuation coefficients K and D in the third embodiment will be described with reference to the process flowchart shown in FIG. The method of determining the values of the attenuation coefficients K and D in the third embodiment is that when calculating the correction amount of the frame image of interest, when correcting with the correction amount of the previous correction frame image in time series order, The number of corrections up to the correction limit value is examined to calculate the correction amount of the frame image to be noted. Here, the processing for the X component of the frame image of interest is also described. However, in processing step S440, the same processing as the processing for the X component shown in FIG. Determine the values of the damping coefficients K, D for. Then, in the next processing step S490 (FIG. 10) to be described later, when the X component and the Y component of the correction position vector are calculated by a predetermined arithmetic expression, the values of the determined attenuation coefficients K and D are used. is there.

  When the processing here is started, input processing of the correction limit value Sxmax is first performed in step S461. More specifically, the CPU 103 reads the value of the Smax X component stored in the hard disk 105 or the like.

  Next, in step S462, the X component Vx (n + 1) of the predicted motion vector V (n + 1) obtained in S35 is extracted. The X component Vx (n + 1) is indicated by the number of pixels in the X direction, and is positive if the vector direction is the right direction of the drawing as shown in FIG. As shown in FIG. 9, the Y component Vy (n + 1) of the predicted motion vector V (n + 1) is positive if the direction of the vector is upward in the drawing and negative if it is downward.

  Next, in step S463, an X component V1x (n) extraction process of the corrected motion vector V1 (n) is performed. Of course, as described above, when n = 1, V1x (1) = 0.

  Next, in step S464, an X component Sx (n) extraction process of the correction position vector S (n) is performed. Here, as described above, the X component Sx (n) is represented by the number of pixels with the left direction on the screen as the positive direction. Of course, the Y component Sy (n) is represented by the number of pixels with the screen lower direction being positive.

  Next, in step S465, the vector difference value BST is calculated from equation (15).

BST = | Vx (n + 1) −V1x (n) | (15)
Next, in step S466, a correction remaining amount value SZT is calculated from equation (16).

SZT = Sxmax− | Sx (n) | (16)
In step S467, the correction limit frame number T is calculated from equation (17).

T = SZT ÷ BST (17)
From the value of the correction limit frame number T calculated in step S467, the values of the attenuation coefficients K and D are determined using the attenuation coefficient table GKT3 in step S468. An example of the attenuation coefficient table GKT3 is shown in FIG. The attenuation coefficient table GKT3 is stored in the hard disk 105 or the like, and from the data of the correction limit frame number T, the CPU 103 reads the value of the corresponding address from the attenuation coefficient table GKT3 and determines the values of the attenuation coefficients K and D.

  In the attenuation coefficient table GKT3 shown in FIG. 16, the correction limit frame number T is a value obtained by rounding the first decimal place to an integer value. The attenuation coefficient table GKT3 shows an example of specific numerical values. The value of the attenuation coefficient K is in the range of 0.9 ≦ K ≦ 1, and the value of the attenuation coefficient D is 0.5 ≦ D ≦. In the range of 1, it may be changed according to the value of the correction limit frame number T. At this time, it is preferable to set the attenuation coefficients K and D to be larger as the value of T becomes larger. When the value of T is large, there is a margin in the amount of correction until the correction limit value is reached compared to when the value of T is small, so the attenuation coefficient is increased and the amount of correction of the frame image is increased. .

  As described above, the processing for the X component described in the first to third embodiments is similarly performed for the Y component, the values of the attenuation coefficients K and D are determined for each of the X and Y components, and the process proceeds to processing step S490. move on.

  When the values of the attenuation coefficients K and D are determined for the X and Y components according to the first to third embodiments, the process returns to step S490 (FIG. 10), and the corrected position vector S (n + 1) is used using a predetermined arithmetic expression. ) Is calculated. In the present embodiment, the following expression (18) is used as the predetermined arithmetic expression.

S (n + 1) = K × S (n) + D × (V (n + 1) −V1 (n)) (18)
Specifically, the correction position vector S (n + 1) is obtained by calculating the correction position vectors Sx (n + 1) and Sy (n + 1) for the X component and the Y component, respectively, using Expression (18).

  As apparent from the equation (18), the difference between the predicted motion vector V (n + 1) and the corrected motion vector V1 (n) of the corrected frame image HFG (n) obtained by correcting the frame image F (n) is obtained. Since a value obtained by multiplying the difference by the attenuation coefficient D is used to calculate the correction position vector S (n + 1), between the corrected frame image HFG (n + 1) obtained by correcting the frame image F (n + 1) and the previous corrected frame image HFG (n). The probability of smooth image movement increases.

In the equation (18), when the frame image of interest is the first in chronological order (n = 1), since the frame image F (1) is not moved in the present embodiment as described above, S ( 1) = 0. By the way,
When n = 2, S (2) = K × 0 + D × (V (2) −0) = D × V (2)
In the case of n = 3, S (3) = K × S (2) + D × (V (3) −V1 (2))
Thus, a corrected position vector is calculated.

(B) Processing in S28 and S50 Next, returning to FIG. 3, (A) the processing in S26, S30, S35 and S40 and (C) the processing in S28 and S50 performed in parallel with the processing in S65 and S70. explain.

  First, in step S28, it is determined whether m = 1. If m = 1 (S28, YES), the process proceeds to step S80.

  On the other hand, if m = 1 is not satisfied (S28, NO), in step S50, the partial image cutting means 140 performs a process of generating a corrected frame image HFG (n).

  FIG. 17 shows an example of the corrected frame image HFG (3) generated by cutting out a predetermined image from the frame image F (3) when n = 3. On the upper side of FIG. 17, a frame image FK (3) that becomes a predetermined image when correction is not performed, that is, when the correction amount is 0, is indicated by a broken line. Then, on the lower side of FIG. 17, correction is performed by correcting the screen position by Sx (3) in the X direction and Sy (3) in the Y direction according to the correction position vector S (3) calculated in step S40. A frame image HFG (3) is shown.

  Similarly, generally, in accordance with the corrected position vector S (n) calculated in step S40 while the (n−1) -th frame image F (n−1) is input in S10, the frame image F ( The corrected frame image HFG (n) is generated by correcting and cutting out the screen position of n) by correcting only Sx (n) in the X direction and Sy (n) in the Y direction.

  Thus, by correcting the screen position, the amount of movement of the image with respect to the frame image can be suppressed.

(C) Processing in S65 and S70 Next, returning to FIG. 3, (A) the processing in S26, S30, S35 and S40 and (B) the processing in S65 and S70 performed in parallel with the processing in S28 and S50. explain.

  First, in S65, the image display means 150 receives the corrected frame image HFG (n-1) generated in S50 while the (n-1) th frame image F (n-1) is input in S10. The enlargement process is performed while the nth frame image F (n) is input in S10.

  The screen size of the corrected frame image generated in S50 is an image that is smaller by 2 × Sxmax in the horizontal direction and 2 × Symax in the vertical direction than the screen size of the acquired frame image. The screen size of the corrected frame image generated in S50 is enlarged to the same screen size as the acquired frame image.

  The screen is enlarged by making the number of pixels of the corrected frame image the same as the number of pixels of the frame image. An increase in the number of pixels constituting an image can be realized by pixel interpolation processing. In addition to the bi-linear method, the interpolation process uses a known method such as the bi-cubic method or the nearest neighbor method.

  Next, in step S70, the image display means 150 performs an output process of the corrected frame image HFG (n-1) enlarged in S65. In the present embodiment, the enlarged corrected frame image HFG (n−1) is converted into an image signal required by an image display device such as the video projector 300 by the output I / F 106 (FIG. 1), and the corrected frame image is converted. Every time it is generated, it is output immediately. In this way, the time from acquisition of the frame image to be corrected to output of the corrected frame image is shortened, and correction can be performed in real time. Of course, the output destination of the enlarged corrected frame image HFG (n−1) may be a predetermined image file provided in the RAM 102, the hard disk 105, or the like, and a series of a plurality of generated corrected frame images may be stored.

  Thus, the processes for S26 to S70 are finished, and n is updated in step S80 (n = n + 1). Then, returning to step S10, the same processing is repeated for the next frame image in time series order, and corrected frame images are output for all the acquired frame images.

  FIG. 18 shows an operation timing chart of the image processing apparatus according to the embodiment of the present invention. The horizontal axis represents time, and n and n + 1 in “(1) Input Image Signal” indicate the time during which the nth and (n + 1) th frame images are input in S10. As shown in FIG. 5, Δt is the time from when the pixel read in S306 reaches from EP1 (the point where the difference value calculation ends) to EP2 (the point where the data reading in the frame ends). In the meantime, a motion vector prediction process and a correction position calculation process for the next frame are performed. If there is a time such as a vertical blanking period between the end of reading the current frame (EP2) and the start of reading the next frame, the motion vector of the next frame is predicted. The processing and the correction position calculation processing may be performed before reading of the next frame is started. Each arrow in “(2) Difference calculation area” indicates that the processes of S310 to S318 in FIG. 4 are being performed in each difference calculation area. The arrows {n− (n−1)} and {(n + 1) −n} in “(3) motion vector detection” are performed by the motion vector detection processing (S30) of the nth and (n + 1) th frame images, respectively. It shows while being broken. The arrows {(n + 1) −n} and {(n + 2) − (n + 1)} in “(4) Next frame motion vector prediction” indicate motion vector prediction processing of the (n + 1) th and (n + 2) th frame images, respectively ( It shows during S35). The arrows {(n + 1) −n} and {(n + 2) − (n + 1)} in “(5) Correction position calculation processing” indicate correction position calculation processing (S40 of the (n + 1) th and (n + 2) th frame images, respectively. ) Is shown. “N, n + 1” in “(6) Frame image cutout” indicates that the nth and (n + 1) th frame image cutout processing (correction frame image generation processing in S50) is being performed in S10. “N−1” and “n” in “(7) Frame image enlargement” indicate that the (n−1) -th and n-th frame images are being enlarged and output (S65 and S70) in S10, respectively.

  As is clear from FIG. 18, the (n + 1) th frame image is predicted from motion vectors obtained based on a plurality of frame images before the nth frame image while the nth frame image is read in S10. A motion vector can be obtained. A correction position calculation process is performed using the predicted motion vector of the obtained (n + 1) th frame image. As a result, image blur due to camera shake is corrected for the input video.

  While the nth frame image is being read in S10, the motion vector of the nth frame image is calculated, the predicted motion vector of the (n + 1) th frame image is calculated, and the readout position of the (n + 1) th frame image. Perform the calculation. If there is a time such as a vertical blanking period between the nth frame image and the (n + 1) th frame image, the calculation of the predicted motion vector of the (n + 1) th frame image and the (n + 1) th frame image are performed. The calculation of the reading position of the frame may be performed until the reading of the (n + 1) th frame image is started. The (n + 1) th frame image is read in parallel with the (n + 1) th frame image read using the read position of the (n + 1) th frame image obtained while the nth frame image is read in S10. A clipped image of the image can be output.

  In this way, the motion vector in the next frame image can be predicted from the motion vector obtained based on the plurality of frame images before the nth frame image. Therefore, by performing in parallel while reading the information for one frame image, from the detection of the motion vector to the prediction of the motion vector of the next frame image and the calculation of the correction amount for the next frame image, the camera shake correction processing and the like are performed in real time. Real-time processing is possible while suppressing the amount of memory required to hold the frame image, which is necessary when performing the above.

  As mentioned above, although one embodiment of the present invention has been described, the present invention is not limited to such an embodiment, and can of course be implemented in various forms without departing from the spirit of the present invention. is there.

  (Modification 1) For example, in the above-described embodiment, the relative positional relationship between the frame image F (n) and the frame image F (n−1) is determined based on the motion vector of the frame image F (n) acquired from the moving image. If the screen position data exists as frame image metadata by means of detecting the amount of movement, such as a speed sensor or an acceleration sensor, when moving images are captured, the screen position data is used for motion. A vector may be calculated. In this way, it is not necessary to perform image processing such as image pattern matching or feature point tracking, and the processing load relating to motion vector calculation can be reduced.

  (Modification 2) In addition, the image processing apparatus in the embodiment is configured by a general-purpose computer, but the present invention is not limited to this, and is configured by a mobile computer, a workstation, or the like. May be. Alternatively, in a digital camera, a video camera, a DVD recorder / player, a video projector, a TV, or a mobile phone, such as a device that can record or play a moving image, or various devices that can record or play a still image in chronological order, The image processing apparatus of the present invention may be incorporated.

  (Modification 3) Further, the image processing apparatus according to the embodiment acquires frame images from all the predetermined number of field images constituting the frame image, and corrects the position where the predetermined image is cut out from the acquired frame images. However, the present invention is not limited to this, and when the frame image is composed of N (N is an integer of 2 or more) field images, 1 to N−1 of these N field images. Of these, an image obtained from an arbitrary number of field images may be used as a frame image, and the position where a predetermined image is cut out may be corrected.

1 is an explanatory diagram showing a schematic configuration of an image processing apparatus as an embodiment of the present invention. 1 is a functional block diagram of an image processing apparatus as an embodiment of the present invention. It is a flowchart explaining the process in the image processing apparatus as one Embodiment of this invention. 4 is a flowchart for explaining in detail a motion vector detection process of an nth frame image in S30 of FIG. 3. It is explanatory drawing for demonstrating in detail the detection process of the motion vector of the nth frame image in S30 of FIG. It is a flowchart for demonstrating the prediction process of the motion vector of the (n + 1) th frame image. It is explanatory drawing for demonstrating the prediction process of a motion vector by a motion locus. It is explanatory drawing for demonstrating the motion vector of a frame image. It is explanatory drawing for demonstrating the relationship between the motion vector of a frame image, a correction | amendment motion vector, and a correction | amendment position vector. It is a flowchart explaining the calculation process of a correction position vector. It is a flowchart explaining the determination process of the attenuation coefficient in a 1st Example. It is an attenuation coefficient table for demonstrating the value of the attenuation coefficient in a 1st Example. It is a flowchart explaining the determination process of the attenuation coefficient in a 2nd Example. It is an attenuation coefficient table for demonstrating the value of the attenuation coefficient in a 2nd Example. It is a flowchart explaining the determination process of the attenuation coefficient in a 3rd Example. It is an attenuation coefficient table for demonstrating the value of the attenuation coefficient in a 3rd Example. It is explanatory drawing explaining the method to produce | generate a correction | amendment frame image from a frame image. It is an operation | movement timing diagram of the image processing apparatus by one Embodiment of this invention. It is explanatory drawing explaining the data flow at the time of the conventional image blurring correction process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Computer, 101 ... Input I / F, 102 ... RAM, 103 ... CPU, 104 ... ROM, 105 ... Hard disk, 106 ... Output I / F, 107 ... Bus line, 110 ... Frame image receiving means, 120 ... Motion vector Detection means 121... Motion trajectory calculation means 122. Baseline calculation means 123... Motion trajectory weighting means 124... Motion vector prediction means 126 126 Memory 130 Extraction position calculation means 140 140 Partial image extraction means 150 ... Image display means, 200 ... Video camera, 210 ... DVD player, 300 ... Projector, 310 ... Display, F (n) ... Frame image, FK (n) ... Frame image, HFG (1) ... Corrected frame image, Smax: Correction limit value, Sxmax: X component of correction limit value, Symax: Correction limit Y component, V0 (n) ... motion vector, V1 (n) ... corrected motion vector, S (n) ... corrected position vector, Sx (n) ... X component of corrected position vector, Sy (n) ... corrected position vector Y component, K ... attenuation coefficient, D ... attenuation coefficient, GKT1-3 ... attenuation coefficient table.

Claims (13)

(M + 1) frame images F (1) to F (M + 1) (M is an integer equal to or greater than 2) to be displayed, which are continuous in time series (in parentheses are integers indicating the time series order), and image shaking Frame image receiving means for sequentially receiving the frame images F (1) to F (M + 1) that may cause
By detecting the motion vector V0 (p) indicating the motion of the frame image F (p) with reference to the frame image F (p−1) (1 <p ≦ (M + 1)), the (M + 1) frames are detected. Motion vector detection for obtaining M motion vectors V0 (2) to V0 (M + 1) indicating the motion of M frame images F (2) to F (M + 1) among the images F (1) to F (M + 1). Means,
Of the M motion vectors V0 (2) to V0 (M + 1), M frame images F (1) to F (1) to (M−1) motion vectors V0 (2) to V0 (M) are shown. The overall motion of F (M) is decomposed into X (X is an integer) base frequency specified by the image fluctuation, and X sine waves defined by the frequency are expressed on the time axis. The frame image F (M + 1) indicating the movement of the frame image F (M + 1) with reference to the frame image F (M) by extending the (M + 1) th frame image F (M + 1) to the time to be displayed. Motion vector predicting means for predicting a predicted motion vector YV0 (M + 1) that is a motion vector of
A cutout for calculating a cutout position S (k) when cutting out a partial image HFG (k) that is a part of the frame image F (k) from the frame image F (k) (k is an integer indicating the time series order). Extraction position calculation means, which uses the predicted motion vector YV0 (M + 1) to calculate a extraction position S (M + 1) for attenuating image fluctuation of the frame image F (M + 1);
Partial image cutout means for cutting out the partial image HFG (M + 1) from the frame image F (M + 1) using the cutout position S (M + 1);
Prior to cutting out the partial image HFG (M + 1) by the partial image cutting means, the frame image F (M) of the Mth frame is received in time series by the image receiving means, and the motion vector detection is performed. Detecting the motion vector V0 (M) by the means, predicting the predicted motion vector YV0 (M + 1) by the motion vector predicting means, and calculating the cut-out position S (M + 1) by the cut-out position calculating means,
Image processing characterized in that a part of the partial image HFG (M + 1) cut out by the partial image cutout unit and a part of the detection of the motion vector V0 (M + 1) by the motion vector detection unit are performed in parallel. apparatus. The image processing apparatus according to claim 1,
Using the (M-1) motion vectors V0 (2) to V0 (M), the entire M frame images F (1) to F (M) having an arbitrary frame image as a reference position. Motion trajectory calculating means for calculating M motion trajectories X0 (1) to X0 (M), which are trajectories that indicate correct movement,
The motion vector prediction means decomposes the M motion trajectories X0 (1) to X0 (M) into the X base frequencies, and converts the X sine waves defined by the frequencies into a time axis. The image processing apparatus according to claim 1, wherein the predicted motion vector YV0 (M + 1) is predicted by extending the frame image F (M + 1) to a time to be displayed. The image processing apparatus according to claim 2,
An image processing apparatus according to claim 1, wherein the number of base frequencies X is different from the M. The image processing apparatus according to claim 3,
The number X of the base frequencies is a number smaller than the M pieces. The image processing apparatus according to claim 2,
Of the X base frequencies, the period of the base frequency excluding the DC component is longer than the time from the time when the frame image F (1) should be displayed to the time when the frame image F (M) should be displayed. An image processing apparatus characterized by being short. An image processing apparatus according to any one of claims 2 to 5,
The baseline, which is a frequency component lower than the base frequency excluding the DC component of the X base frequencies, is removed from the motion trajectories X0 (1) to X0 (M) calculated by the motion trajectory calculation means. Baseline calculation means for correcting a trajectory as the motion trajectories X0 (1) to X0 (M),
The motion vector prediction means decomposes the M motion trajectories X0 (1) to X0 (M) corrected by the baseline calculation means into the X base frequencies, and is defined by the frequencies. The coordinate position obtained by extending X sine waves to the time to display the (M + 1) -th frame image F (M + 1) on the time axis, and the base line on the time axis ( The motion trajectory X0 (M + 1) is predicted by adding the coordinate position obtained by extending to the time at which the M + 1) th frame image F (M + 1) should be displayed, and the motion trajectory X0 (M) and the motion An image processing apparatus that predicts the predicted motion vector YV0 (M + 1) of the frame image F (M + 1) from a locus X0 (M + 1). The image processing apparatus according to claim 6,
The image processing apparatus according to claim 1, wherein the base line is a straight line having an inclination obtained by averaging inclinations between frames of the motion trajectories X0 (1) to X0 (M). The image processing apparatus according to any one of claims 2 to 7,
Before the motion vector predictor uses the motion trajectories X0 (1) to X0 (M), the motion trajectory X0 (1) is weighted so as to be closer to the (M) th frame in time series order. ) To X0 (M), further comprising a motion trajectory weighting means. The image processing apparatus according to any one of claims 2 to 8,
The motion trajectory weighting means uses a function W (n) in which the value corresponding to the first frame in time series order is closest to 0 and the value corresponding to the Mth frame in time series order is closest to 1. The image processing apparatus is characterized by correcting the motion trajectories X0 (1) to X0 (M) by multiplying the coordinates of the motion trajectory X0 (n) and the value of W (n) for each frame. An image processing apparatus according to any one of claims 1 to 9,
An image processing apparatus comprising image display means for sequentially displaying the partial images HFG (M + 1) in chronological order. (M + 1) frame images F (1) to F (M + 1) (M is an integer equal to or greater than 2) to be displayed, which are continuous in time series (in parentheses are integers indicating the time series order), and image shaking Frame image receiving step for sequentially receiving the frame images F (1) to F (M + 1) that may generate
By detecting the motion vector V0 (p) indicating the motion of the frame image F (p) with reference to the frame image F (p−1) (1 <p ≦ (M + 1)), the (M + 1) frames are detected. Motion vector detection for obtaining M motion vectors V0 (2) to V0 (M + 1) indicating the motion of M frame images F (2) to F (M + 1) among the images F (1) to F (M + 1). Process,
Of the M motion vectors V0 (2) to V0 (M + 1), M frame images F (1) to F (1) to (M−1) motion vectors V0 (2) to V0 (M) are shown. The overall motion of F (M) is decomposed into X (X is an integer) base frequency specified by the image fluctuation, and X sine waves defined by the frequency are expressed on the time axis. The frame image F (M + 1) indicating the movement of the frame image F (M + 1) with reference to the frame image F (M) by extending the (M + 1) th frame image F (M + 1) to the time to be displayed. A motion vector prediction step of predicting a predicted motion vector YV0 (M + 1) that is a motion vector of
A cutout for calculating a cutout position S (k) when cutting out a partial image HFG (k) that is a part of the frame image F (k) from the frame image F (k) (k is an integer indicating the time series order). A position calculation step, which uses the predicted motion vector YV0 (M + 1) to calculate a cut position S (M + 1) that attenuates the image shake;
A partial image cutting step of cutting out the partial image HFG (M + 1) from the frame image F (M + 1) using the cutting position S (M + 1),
Prior to cutting out the partial image HFG (M + 1) by the partial image cutting step, receiving the frame image F (M) of the Mth frame in time series order by the image receiving step, and detecting the motion vector Detecting a motion vector V0 (M) by a step, predicting the predicted motion vector YV0 (M + 1) by the motion vector prediction step, and calculating the cut position S (M + 1) by the cut position calculation step,
Image processing characterized in that a part of the partial image HFG (M + 1) cut out by the partial image cutout step and a part of the detection of the motion vector V0 (M + 1) by the motion vector detection step are performed in parallel. Method. Computer
(M + 1) frame images F (1) to F (M + 1) (M is an integer equal to or greater than 2) to be displayed, which are continuous in time series (in parentheses are integers indicating the time series order), and image shaking Frame image receiving means for sequentially receiving the frame images F (1) to F (M + 1) that may cause
By detecting the motion vector V0 (p) indicating the motion of the frame image F (p) with reference to the frame image F (p−1) (1 <p ≦ (M + 1)), the (M + 1) frames are detected. Motion vector detection for obtaining M motion vectors V0 (2) to V0 (M + 1) indicating the motion of M frame images F (2) to F (M + 1) among the images F (1) to F (M + 1). Means,
Of the M motion vectors V0 (2) to V0 (M + 1), M frame images F (1) to F (1) to (M−1) motion vectors V0 (2) to V0 (M) are shown. The overall motion of F (M) is decomposed into X (X is an integer) base frequency specified by the image fluctuation, and X sine waves defined by the frequency are expressed on the time axis. The frame image F (M + 1) indicating the movement of the frame image F (M + 1) with reference to the frame image F (M) by extending the (M + 1) th frame image F (M + 1) to the time to be displayed. Motion vector predicting means for predicting a predicted motion vector YV0 (M + 1) that is a motion vector of
A cutout for calculating a cutout position S (k) when cutting out a partial image HFG (k) that is a part of the frame image F (k) from the frame image F (k) (k is an integer indicating the time series order). A position calculation unit, which uses the predicted motion vector YV0 (M + 1) to calculate a cutout position S (M + 1) that attenuates the image shake;
A program for functioning as a partial image cutout means for cutting out the partial image HFG (M + 1) from the frame image F (M + 1) using the cutout position S (M + 1),
Prior to cutting out the partial image HFG (M + 1) by the partial image cutting means, the frame image F (M) of the Mth frame is received in time series by the image receiving means, and the motion vector detection is performed. Detecting the motion vector V0 (M) by the means, predicting the predicted motion vector YV0 (M + 1) by the motion vector predicting means, and calculating the cut-out position S (M + 1) by the cut-out position calculating means,
Image processing characterized in that a part of the partial image HFG (M + 1) cut out by the partial image cutout unit and a part of the detection of the motion vector V0 (M + 1) by the motion vector detection unit are performed in parallel. program. A computer-readable recording medium on which the image processing program according to claim 12 is recorded.

Images