JP5364927B2 - Motion compensation apparatus and method - Google Patents

Description

  The present invention relates to a motion correction apparatus and method for obtaining a motion between images from a plurality of different image information and correcting the display position of the image from the motion information.

  The block matching method is the most common method for obtaining movement (movement amount) between different images.

  The block matching method can detect the movement (movement amount) with high accuracy when the moving direction of the image is only the X-axis and the Y-axis, but has a drawback that the detection accuracy is low with respect to the rotation of the image. As a countermeasure for rotation correction by block matching, a technique described in Patent Document 1 is proposed.

As a method for obtaining the rotational movement amount of the image with high accuracy, there is a phase correlation method.
In this method, it is possible to detect not only the X axis and Y axis but also rotation and enlargement / reduction of an image with high accuracy. However, since the calculation amount is large, it has not been used so far.

However, it has been attracting attention in recent years due to the speeding up of arithmetic processing due to recent technological developments.
Further, image blur correction is performed by correcting the image display position in the direction opposite to the movement amount from the obtained image movement amount information.

Japanese Patent Application No. 2004-564419

The method described above is effective in detecting and correcting the movement (movement amount) between two images.
However, in the above-described method, when the detection / correction is continued in time as in a moving image, the angle of the image gradually shifts due to the accumulation of angle detection errors, and the correction is not stable. There was a problem.

  In addition, in the motion blur correction by the image cutout method, in the case of blur correction in the X and Y directions, the center position of the blur is basically a detection error caused when correction is started centering on the position at the start of shooting. There is a problem that the position of the image gradually shifts due to the accumulation, and the blur correction possible range becomes small.

  In addition, when correcting the rotational movement amount, if the rotational blur is corrected around the position at the start of shooting, if the image is tilted at the start of shooting, the image is corrected using the original image as a reference. Therefore, an unnatural image is displayed.

A method using template matching can be considered as a measure for accumulation of image blur errors and a method for determining a blur center position.
However, since this is a comparison with a fixed template image, there is a premise that the conditions of the image to be shot must be unchanged, and it is necessary to use it in an environment where the shooting situation changes such as outdoors, There is a disadvantage that it cannot be used when there is a tilt or the like.

  When detecting and correcting a continuous motion such as a moving image, the present invention does not fluctuate the display angle or position due to accumulation of errors over time, and does not cause an unnatural tilt or shift of the display position. It is an object of the present invention to provide a motion correction apparatus and method capable of displaying an image without any problem.

A motion correction device according to a first aspect of the present invention determines a motion detection unit that detects a motion between temporally continuous images, and an angle that is a reference for correcting a rotation direction of the detected motion. A reference angle determination unit, a frame-by-frame rotation angle information storage unit that stores information about the rotation angle of each frame from the reference angle, and a plurality of the frame-by-frame rotation angle information are used to calculate a correction amount of the movement in the rotation direction. to possess the correction amount calculating unit, wherein the reference angle determination unit determines the angle as a reference for the correction value obtained by subtracting the average value from the frame every rotation angle information as history data.

The second aspect of the motion compensation device of the present invention, determines a motion detector for detecting a movement between images, the position serving as a reference for horizontal or vertical correction of said detected movement temporally consecutive A reference position determination unit that performs the above-described correction, a frame-by-frame motion amount information storage unit that stores information about a horizontal or vertical motion amount of each frame from the position serving as a reference for correction, and a plurality of pieces of the motion amount information for each frame. have a, a correction amount calculation unit for calculating a correction amount of horizontal or vertical motion Te, the reference position determination unit, the correction value obtained by subtracting the average value from the frame each motion amount information as history data Determine the reference position .

A motion correction method according to a third aspect of the present invention determines a motion detection step for detecting a motion between temporally continuous images, and an angle serving as a reference for correcting a rotation direction of the detected motion. A reference angle determination step, a frame-by-frame rotation angle information storage step for storing information on the rotation angle of each frame from the reference angle, and a plurality of the frame-by-frame rotation angle information are used to calculate a correction amount of the movement in the rotation direction. to possess a correction amount calculating step, and in the reference angle determination step determines the angle as a reference for the correction value obtained by subtracting the average value from the frame every rotation angle information as history data.

According to a fourth aspect of the present invention, there is provided a motion correction method for detecting a motion between temporally continuous images, and determining a position serving as a reference for horizontal or vertical correction among the detected motions. A reference position determining step, a frame-by-frame motion amount information storing step for storing horizontal or vertical motion amount information for each frame from the position serving as a reference for correction, and a plurality of pieces of frame-by-frame motion amount information. A correction amount calculating step of calculating a correction amount of horizontal or vertical motion, and in the reference position determination step, the correction is performed using a value obtained by subtracting an average value from the motion amount information for each frame as history data. Determine the reference position .

  According to the present invention, when detecting and correcting a continuous motion such as a moving image, there is no fluctuation in the display angle and position due to accumulation of errors over time, and there is no natural fluctuation of the display position. It is possible to perform image display without any problem.

Arrangement of an image processing apparatus employing the motion correction apparatus of the image according to the implementation embodiments of the present invention is a block diagram showing the. It is a figure which shows the example that the angle of an image will shift | deviate gradually by accumulation | storage of an angle detection error when correction | amendment is continued continuously. It is a figure for demonstrating the basic X-axis and Y-axis blur correction. It is a figure for demonstrating the blurring correction | amendment of a rotating shaft when a rotation center exists in a screen. It is a figure for demonstrating the blurring correction | amendment of a rotating shaft when a rotation center is not in a screen. It is a figure which shows the case where a photography start time is the center of blurring. It is a figure which shows the case where imaging | photography is started when the image inclines. It is a figure which shows the case where accumulation | storage of the movement amount detection error of a horizontal or vertical direction generate | occur | produced. It is a figure for demonstrating the principle of this system. It is a figure which shows the case where the rotation center position (reference | standard position) is calculated | required with this system with respect to the waveform which added the angle detection error of FIG. 2 to the transition of the angle of a sine wave. The rotation center position (reference position) is obtained by this method for the waveform transition in which the frequencies 3, 5, and 7 Hz are mixed and the phase is shifted by π / 6 and the angle detection error in FIG. 2 is added. It is a figure which shows the case where it calculated | required. It is a figure which shows the case where the panning of a camera generate | occur | produced with the constant speed with respect to the same waveform as FIG. Is a flow chart for explaining the operation of the image processing apparatus according to the present implementation embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Figure 1 is a block diagram showing the arrangement of an image processing apparatus employing the motion correction apparatus of the image according to the implementation embodiments of the present invention.

The image processing apparatus 10 includes an image input unit 11, a storage unit 12, an inter-frame movement amount calculation unit 13 as a motion detection unit, an accumulated movement amount calculation unit 14 including a function as a rotation angle detection unit, and a center rotation angle calculation unit. As a reference position determination unit 15, a shake correction unit 16, and an image output unit 17.
The history information storage unit 12b, the accumulated movement amount calculation unit 14, and the reference position determination unit 15 of the storage unit 12 form a reference angle determination unit.

When detecting and correcting a continuous motion such as a moving image, the image processing apparatus 10 is free from fluctuations in the display angle due to accumulation of errors over time and unnatural tilting of the display position. It is possible to display an image without any image.
The image processing apparatus 10 obtains a movement amount including a rotation angle between the images from information between different consecutive images, and then stops the image from the rotation movement amount data and displays the angle (rotation center angle). Ask for. As a result, the image processing apparatus 10 performs natural and unsharp image display with no unnatural inclination of the display position and no change in display angle due to accumulation of errors over time.
The image processing apparatus 10 stores the rotation angle information for each frame, which is movement amount information of the rotated angle, as history information, calculates the average value of the rotation angle information for each frame, which is movement amount information on the history, and calculates the average value The motion vector of the minute is corrected and displayed. Thereby, the image processing apparatus 10 performs natural and unsharp image display.

The image processing apparatus 10 basically obtains a movement amount including a rotation angle between different images from information between different consecutive images using the first acquired image as a reference.
The obtained movement amount is stored as history information for a plurality of times from the latest time.
Then, an average value of the movement amount information on the history is calculated, and the motion vector corresponding to the average value is corrected from the movement center position as a correction value.

  The image input unit 11 receives an input signal based on the captured image (video) and supplies the input image signal to the storage unit 12 and the image output unit 17.

  The storage unit 12 includes an image storage unit 12a and a history information storage unit 12b.

The image storage unit 12a sequentially stores the temporally continuous image data supplied from the image input unit 11 in units of frames, for example.
The image data stored in the image storage unit 12 a is read by the motion detection unit 13.

The history information storage unit 12b functions as a frame-by-frame rotation angle information storage unit (rotation angle storage unit), and stores information on the rotation angle of each frame from the reference angle.
History information storage unit 12b as the rotation angle storage unit sequentially stores the detected rotation angle accumulated movement amount calculating section 14 as a rotation angle detecting unit.

  The inter-frame movement amount calculation unit 13 as a motion detection unit detects a movement (movement amount) including the rotation angle between the current frame and the previous frame stored in the image storage unit 12a of the storage unit 12, and rotates the detection result. The accumulated movement amount calculation unit 14 as an angle detection unit is supplied.

The accumulated movement amount calculation unit 14 as a rotation angle detection unit detects a rotation angle for each frame from the reference angle among the movements detected by the inter-frame movement amount calculation unit 13 as a movement detection unit, and detects the detected frame. Each rotation angle is output and stored in the history information storage unit 12b as a frame-by-frame rotation angle information storage unit.
The accumulated movement amount calculation unit 14 has a function as a reference angle determination unit that determines an angle serving as a reference for correcting the rotation direction of the detected motion, and stores information on the rotation angle of each frame from the reference angle. The integrated value is stored in the history information storage unit 12b.

As an example of processing for detecting the rotation angle of the inter-frame movement amount calculation unit 13, a phase correlation method is shown below.
In this movement amount and rotation angle detection processing, for example, an Fourier image according to the following (Equation 1) is performed on an image in which a predetermined window function is set to obtain an amplitude image.

Then, the amplitude image obtained by the Fourier transform and expressed in the orthogonal coordinate system is converted into a polar coordinate system (log-polar). Converting to polar coordinates means performing complex logarithmic polar coordinates.
Further, the detection processing unit detects the amount of change between images based on θ and r converted to the polar coordinate system. At this time, the amount of change is detected when any of the operations of rotation, enlargement, and reduction is performed.

In the log-polar conversion processing, the pixel position Y (x, y) on the orthogonal coordinates is converted into polar coordinates θ, r and further logarithmically converted to obtain the coordinates Y (ρ, θ) after log-polar conversion.
The formula is shown below.

Therefore, it is possible to obtain the rotation angle of the image by obtaining the shift in the θ direction between the comparison target images in the detection process. Incidentally, enlargement / reduction can be obtained by obtaining the displacement in the r direction.
Then, the rotation angle of each frame from the obtained reference angle is stored in the history information storage unit 12b as the rotation angle storage unit.

The reference position determination unit 15 integrates a plurality of rotation angle information for each frame stored in the history information storage unit 12b serving as a rotation angle storage unit, calculates an average value, and sets this as a reference angle.
The blur correction unit 16 has a function as a correction amount calculation unit that calculates a correction amount of motion in the rotation direction using the frame-by-frame rotation angle information and the reference angle stored in the history information storage unit 12b.
This calculation process will be described in detail later.

  Based on the reference angle based on the center rotation angle calculated by the reference position determination unit 15, the shake correction unit 16 performs blurring of an image (moving image in the present embodiment) at the first rotation angle in the image output unit 17. Motion).

The image processing apparatus 10 having the above configuration uses a method of estimating the rotation center position from the history information of the movement of the rotation angle of the image.
Hereinafter, this method will be described in further detail.

First, FIG. 2 shows an example in which the angle of the image gradually shifts due to the accumulation of the angle detection error when the correction is continuously performed.
Here, the detection accuracy of the rotation angle is ± 0.3 °. In this result, the error for each frame is within ± 0.3 °, but as time elapses, the error is accumulated, and finally the error is 3.0 °. Recognize.

In general, in motion blur correction, in the case of correcting motion in the X direction (horizontal direction or horizontal direction) and Y direction (vertical direction or vertical direction), basically, as shown in FIG. It is only necessary to start correction around the position of.
However, in the case of correcting the rotational movement amount, if the rotational blur is corrected around the position at the start of shooting, as shown in FIGS. In addition, since the correction is performed based on the image as it is, an unnatural image is displayed.
3 to 5, RIM represents a reference image area, and in FIGS. 4 and 5, VPST represents a vertical position of the image.

For example, in a surveillance camera, when the camera is installed on a vertically standing pole, the camera may shake due to wind or surrounding vibration.
In this case, it rotates repeatedly around the lower part of the image, but the position where the rotation should be stopped should be in the direction perpendicular to the ground.
However, if the correction is started in the state of being rotated with respect to the vertical direction at the start of shooting or the start of blur correction, the shifted position is corrected based on the reference, resulting in an unnatural image.

When the shooting start time is the center of blurring, the image is not uncomfortable as shown in FIG. 6, but when shooting is started when the image is tilted, the image is tilted as shown in FIG. Will be corrected with reference to. Even when shooting is started with an image that is not tilted, a phenomenon in which the image is gradually tilted due to accumulation of detection errors may occur.
Further, in the correction in the X direction and the Y direction, if an error is accumulated, as shown in FIG. 8, the correction is performed based on an image in which a part of the subject OBJ is cut.

Therefore, in this implementation form, a method of estimating the rotational center position from the history information of the motion of the rotation angle.

  As the history information, the rotation center at the initial image position is set to 0 °, and the movement amount from the position as the rotation center is stored for each frame. Then, the average value of the movement amount information on the history is calculated, and the center of rotation is moved using the motion vector for the average value as a correction value.

Specific examples are shown below.
The rotation center position to be obtained is D, the number of stored angle information is m + 1, the stored angle information is current time t, and n = 0, 1, 2, 3,. Then, the amount of movement from the rotation center for each frame is f (t), f (t-1), f (t-2),... F (tm).
Here, when the average Ave of the history data of the stored movement amount is taken, Equation (3) is obtained.

From this average value Ave, the rotation center position D _{t at} time t is _expressed by equation (4).

In addition, the values of the movement amounts f (t), f (t-1), f (t-2),... F (tm) from the rotation center for each frame are also saved by subtracting the average Ave value each time. To do.
When the history data saved by subtracting the average Ave is f ′ (t), f ′ (t−1), f ′ (t−2),... F ′ (tm), the following equation (5) is obtained.

  Here, the reason why accumulation of errors can be eliminated by using the method of estimating the rotation center position as described above will be described with an example.

Normally, in the method of adding motion vectors obtained between frames as needed, if the error is biased in the plus or minus direction, the error is accumulated as shown in FIG. Determine the cutout position and output the image.
For this reason, if the cutout position exceeds the correctable range, it cannot be corrected.

  When the method according to this embodiment is used, the number of saved history data is variable, but here, a case where the number of history data is 10 is shown as an example. The frame rate is 30 fps as an example.

The principle will be described by taking the case of FIG. 9 as an example.
A solid line A in FIG. 9 shows an actual shake waveform. Here, the sine wave becomes an actual waveform. Here, if an error of −0.2 ° occurs every second to third period of the sine wave, the amount of motion between frames including the error is integrated into a waveform as shown by a broken line B in FIG. And the center of rotation gradually shifts.

Here, the recognized position of the center of rotation when this method is used is indicated by a thick line C in the figure.
As can be seen from this result, the position recognized as the center of rotation is slightly shifted while an error between frames continues to occur, but it converges at a certain value and does not become an error of a certain level or more.
It can also be seen that when the error no longer occurs, it gradually returns to the original rotational position.
This is because, for example, when an error occurs in the minus direction, the value in the history table is shifted in the plus direction opposite to the error.

By this shift, the apparent center position of rotation shifts in the minus direction. By shifting the rotation center position, the position of the waveform of the motion that occurs after this is recognized not as a broken line part but as a solid line part.
If the error is continuously the same way direction, until the same number of times as the number of history table reaches the number of successive, the error in the opposite direction, the rotational center position is shifted.

Since the value to be shifted is a value divided by the number of times of history, if the error continuation exceeds the number of tables, the shift value does not increase any more.
Since the shift value is subtracted from the history table for each frame, the position recognized as the center converges without increasing any more.
If the error disappears, the value to be shifted gradually decreases, so that it gradually returns to the original center position.

Here, the case where the error continues in a certain direction is taken as an example, but in such a case, the position recognized as the center and the actual center position are gradually shifted, and the correction processing will eventually fail. It will be.
However, since it cannot be such an extreme error transition, there is no problem in actual use.

Another example is shown below.
FIG. 10 shows a case where the rotation center position (reference position) is obtained by this method for a waveform obtained by adding the angle detection error of FIG. 2 to the angle transition of the sine wave.
From FIG. 10, it can be seen that the rotation center transitions around zero.
In FIG. 10, a sine wave is used as a typical example, but in many cases, an actual blur waveform includes various frequencies.

As an example, FIG. 11 is rotated by this method for the waveform transition in which the frequencies 3, 5, and 7 Hz are mixed and the phase is shifted by π / 6 and the angle detection error of FIG. 2 is added. The case where the center position (reference position) is obtained is shown.
Even in the case of such a waveform, it can be seen that the rotation center moves within a certain range without accumulating errors.

Next, FIG. 12 shows a case where panning of the camera occurs at a constant speed with respect to the waveform of FIG.
In this case, it can be seen that the rotation center position also moves in accordance with panning.
When the number of history data is increased, past data will be referred to, so for example, when the camera shifts from a state with blurring or when the camera is panned, the latest state is reflected. Will take longer.
From these characteristics, if smoother movement is required, increase the number of history data, and if more rapid movement is required, reduce the number of history data. You can choose.

  FIG. 13 is a flowchart for explaining the operation of the image processing apparatus according to the present embodiment.

Through the image input unit 11, temporally captured images (videos) are sequentially stored in the image storage unit 12a, for example, in units of frames (ST1, ST2). Of the two images to be compared, the first image (previous frame) is stored in step ST1, and the second image (current frame) is stored in step ST2.
The image data stored in the image storage unit 12a is read by the inter-frame movement amount calculation unit 13 as a motion detection unit.

The inter-frame movement amount calculation unit 13 detects a movement (movement amount) including the rotation angle between the current frame and the previous frame stored in the image storage unit 12 a of the storage unit 12, and the detection result is the accumulated movement amount calculation unit 14. To be supplied.
In the accumulated movement amount calculation unit 14, the rotation angle of each frame from the reference angle is obtained as an angle change amount from the motion detected by the inter-frame movement amount calculation unit 13 (ST3).
Then, the integrated value of the angle change amount is stored in the history information storage unit 12b (ST4).
Next, in the reference position determination unit 15, an average value of history data is obtained (ST5).
In the shake correction unit 16, the amount of change in angle is corrected by the average value, and the blur of the image (moving image in this embodiment) at the first rotation angle in the image output unit 17 is corrected and displayed (ST6).
Then, the average value is subtracted from each history data (ST7), and the process is repeated from step ST2.

  As described above, the image processing apparatus 10 to which the image motion correction apparatus of this embodiment is applied can obtain the following effects.

That is, by obtaining the amount of movement including the rotation angle between the images from information between different consecutive images, and then obtaining the angle (rotation center angle) to be displayed after stopping the image from the amount of movement data of the rotation. Thus, it is possible to perform natural and unsharp image display with no unnatural inclination of the display position and no change in display angle due to accumulation of errors over time.
In addition, the movement amount information of the rotated angle is saved as history information, the average value of the movement amount information on the history is calculated, and the motion vector corresponding to the average value is corrected and displayed, so that there is no natural fluctuation Image display can be performed.

  That is, in the present embodiment, the movement amount detection and correction of the rotational movement in the Z-axis direction with respect to the screen has been described. However, this method is not limited to the rotation of the Z-axis, but the parallel movement of the X, Y, and Z axes. It can also be applied to the rotational movement of the X and Y axes.

  In this case, instead of the rotation angle, the change position when the movement in the Y direction (vertical direction) or the X direction (horizontal direction) among the detected movements changes in the direction opposite to the direction in which the movement state is detected. To do.

  As described above, by applying the image movement amount detection method of the present embodiment to the movement amount other than the rotation, it is possible to perform image display with higher accuracy and without shaking.

  DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus, 11 ... Image input part, 12 ... Memory | storage part, 13 ... Inter-frame movement amount calculation part, 14 ... Accumulated movement amount calculation part, 15 ... Reference position Determining unit, 16 ... blur correction unit, 17 ... image output unit.

Claims (4)

A motion detector that detects motion between temporally continuous images;
A reference angle determination unit for determining an angle serving as a reference for correcting a rotation direction of the detected movement;
A frame-by-frame rotation angle information storage unit that stores information about the rotation angle of each frame from the reference angle;
A correction amount calculation unit for calculating a correction amount of the rotational movement by using a plurality of said frame each rotation angle information, the possess,
The reference angle determination unit
A motion correction apparatus that determines an angle serving as a reference for the correction using a value obtained by subtracting an average value from the rotation angle information for each frame as history data . A motion detector that detects motion between temporally continuous images;
A reference position determination unit for determining a position that is a reference for correction in the horizontal or vertical direction among the detected movements;
A frame-by-frame motion amount information storage unit that stores information on the amount of motion in the horizontal or vertical direction of each frame from a position serving as a reference for the correction;
A correction amount calculation unit for calculating a correction amount of horizontal or vertical movement by using a plurality of frames each motion amount information, the possess,
The reference position determination unit
A motion correction apparatus that determines a position serving as a reference for the correction using a value obtained by subtracting an average value from the motion amount information for each frame as history data . A motion detection step for detecting motion between temporally continuous images;
A reference angle determining step for determining an angle serving as a reference for correcting the rotation direction of the detected movement;
A frame-by-frame rotation angle information storing step for storing information about the rotation angle of each frame from the reference angle;
A correction amount calculating step of calculating a correction amount of the rotational movement by using a plurality of said frame each rotation angle information, the possess,
In the reference angle determination step,
A motion correction method for determining an angle serving as a reference for the correction using a value obtained by subtracting an average value from the rotation angle information for each frame as history data . A motion detection step for detecting motion between temporally continuous images;
A reference position determining step for determining a position serving as a reference for correction in the horizontal or vertical direction of the detected movement;
A frame-by-frame motion amount information storing step for storing horizontal or vertical motion amount information from the position serving as a reference for the correction ;
A correction amount calculating step of calculating a correction amount of motion in the horizontal direction or the vertical direction by using a plurality of the motion amount information for each frame, and
In the reference position determining step,
A motion correction method for determining a position serving as a reference for the correction using a value obtained by subtracting an average value from the motion amount information for each frame as history data .

Images