JP5415862B2 - Motion amount detection apparatus and method, and imaging apparatus - Google Patents

Description

  The present invention relates to a motion amount detection apparatus and method, and an imaging apparatus capable of detecting and correcting moving image blur in fields such as image measurement and high-resolution image generation.

In recent years, in image measurement and image generation techniques, there is a need for a technique for detecting the amount of movement between successive images of moving image data with high accuracy.
In detecting the amount of motion, there are a block matching method, a phase correlation method, and the like as a method for obtaining a correlation between images.

In the phase correlation method, the phase components of two Fourier-transformed images are synthesized and inverse Fourier transformed to synthesize the correlation function of the image, that is, the correlation indicating the amount of motion between images and the degree of correlation between images. The value can be detected.
Since the amount of motion between images at this time is the amount of motion in units of pixels, it is possible to detect the amount of motion with high accuracy by estimating the amount of motion with a precision equal to or less than the unit of pixels from here.
The estimation of the amount of motion in units of pixels or less includes a method of interpolating between pixel unit positions of the obtained correlation function, a method of estimating by function approximation, and the like (for example, see Patent Documents 1 and 2).

JP 2001-175864 A JP 2004-240931 A

  However, when the amount of motion of the two images is large, the proportion of the same subject in the image is small, so the correlation value between the images is small, and the sharpness of the detected correlation function is small. It gets smaller.

When estimation is performed by function approximation, since it is performed by the least square method or the like, a larger amount of sharpness can correctly estimate a motion amount in units of pixels or less.
Therefore, a small correlation value adversely affects estimation accuracy, leading to a decrease in reliability.

  An object of the present invention is to provide a motion amount detection apparatus, a method thereof, and an imaging apparatus capable of accurately detecting a motion amount between consecutive images in units of pixels or less.

A first aspect of the present invention is a motion amount detection apparatus that obtains a correlation between successive images from a series of moving image data and obtains a motion amount of the entire image between the images. If the correlation value falls below a preset threshold, the arbitrary block is cut out based on the amount of motion. A processing unit that shifts the position and calculates the motion amount and the correlation value again between the blocks cut out by the shift is included.

A first aspect of the present invention is a motion amount detection device that obtains a correlation between successive images from a series of moving image data and obtains a motion amount of the entire image between the images ,
An arbitrary block is cut out from the screen of the moving image, a correlation value that is a degree of correlation between the amount of motion between successive images is calculated, and when the amount of motion exceeds a preset threshold value, based on the amount of motion A processing unit that shifts a position where the arbitrary block is cut out and calculates the motion amount and the correlation value again between the blocks cut out after the shift.

A second aspect of the present invention is a motion amount detection method that obtains a correlation between successive images from a series of moving image data and obtains a motion amount of the entire image between the images. A first calculation step of calculating a correlation value that is a degree of correlation between the amount of motion between successive images and the correlation value falls below a preset threshold value, and based on the amount of motion A shift step of shifting a position to cut out an arbitrary block, and a second calculation step of calculating the motion amount and the correlation value again between the blocks cut out by the shift.
A second aspect of the present invention is a motion amount detection method that obtains a correlation between successive images from a series of moving image data and obtains a motion amount of the entire image between the images. A first calculation step of calculating a correlation value that is a degree of correlation between the amount of motion between successive images and the amount of motion exceeds a preset threshold, and based on the amount of motion A shift step for shifting the position to cut out an arbitrary block;
A second calculation step of calculating the motion amount and the correlation value again between the blocks cut out by the shift.

An imaging apparatus according to a third aspect of the present invention includes an optical system, an imaging element that captures a subject image that has passed through the optical system, and a correlation between consecutive images from a series of moving image data obtained by the imaging element. And a motion amount detection device that obtains the motion amount of the entire image between the images, and the motion amount detection device cuts out an arbitrary block from the moving image screen and moves the motion amount between consecutive images. If the correlation value that is the degree of correlation is calculated and the correlation value falls below a preset threshold value, the position to cut out the arbitrary block is shifted based on the amount of motion, and the block cut out by the shift A processing unit for calculating the motion amount and the correlation value again.
An imaging apparatus according to a third aspect of the present invention includes an optical system, an imaging element that captures a subject image that has passed through the optical system, and a correlation between consecutive images from a series of moving image data obtained by the imaging element. And a motion amount detection device that obtains the motion amount of the entire image between the images, and the motion amount detection device cuts out an arbitrary block from the moving image screen and moves the motion amount between consecutive images. If the correlation value that is the degree of correlation is calculated, and the amount of motion exceeds a preset threshold value, the position to cut out the arbitrary block is shifted based on the amount of motion, and the block cut out by the shift A processing unit for calculating the motion amount and the correlation value again.

  According to the present invention, it is possible to accurately detect the amount of motion between consecutive images in units of pixels or less.

It is a block diagram showing one embodiment of an imaging device concerning the present invention. It is FIG. 1 for demonstrating the cutting-out of the image which concerns on this embodiment. It is FIG. 2 for demonstrating the cutting-out of the image which concerns on this embodiment. It is a figure which shows a correlation function when the deviation | shift amount between images is small as a result of correlation calculation. It is a figure which shows a correlation function in case the deviation | shift amount between images is large as a result of correlation calculation. It is a figure which shows the process which cuts out a block again in the position shifted by the amount of motion detected in the cut-out position of FIG. It is a figure which shows the method of calculating | requiring with the method of least squares etc. and calculating | requiring a peak value with an approximation function. It is a flowchart for demonstrating the 1st motion amount detection operation | movement which concerns on this embodiment. It is a flowchart for demonstrating the 2nd motion amount detection operation | movement which concerns on this embodiment. It is a flowchart for demonstrating the 3rd motion amount detection operation | movement which concerns on this embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing an embodiment of an imaging apparatus according to the present invention.

The imaging apparatus 10 includes an optical system 11, an imaging control unit 12, an imaging unit 13, an image processing unit 14, a motion amount detection unit 15, and a memory 16.
Among these components, the image processing unit 14, the motion amount detection unit 15, the memory 16, and the like constitute a processing unit.

The imaging apparatus 10 has a motion amount detection function that obtains a correlation between successive images from a series of moving image data obtained by the imaging unit 13 and obtains a motion amount of the entire image between the images.
In the motion amount detection function, when extracting an arbitrary block in a moving image and taking a correlation between frames and performing motion detection, the detection accuracy decreases if the motion amount between frames is large or the correlation is low. There is a risk of it.
Therefore, in the motion amount detection function of the present embodiment, fine motion amount detection of an image is realized by obtaining a correlation with a block cut out at a position shifted by the motion amount.

Specifically, the motion amount detection function of the imaging apparatus 10 cuts out an arbitrary block from the moving image screen, and calculates a correlation value that is the amount of motion and the degree of correlation between successive images. including.
Next, the motion amount detection function has a shift function that shifts the position where an arbitrary block is cut out based on the motion amount.
The motion amount detection function has a second calculation function for calculating the motion amount and the correlation value again between the blocks cut out by shifting.

The motion amount detection function is formed including a comparison function for comparing the calculated correlation value with a preset threshold value.
The motion amount detection function shifts the position of extracting an arbitrary block based on the motion amount when the correlation value falls below a preset threshold value as a result of the comparison by the comparison function in the shift function.
Then, in the second calculation function, the motion amount and the correlation value are calculated again between the blocks cut out by shifting.
Alternatively, the motion amount detection function is formed including a comparison function that compares the calculated motion amount D with a preset threshold value.
In the shift function, when the amount of motion exceeds a preset threshold value as a result of comparison by the comparison function, the position where an arbitrary block is cut out is shifted based on the amount of motion.
Then, in the second calculation function, the motion amount and the correlation value are calculated again between the blocks cut out by shifting.

  The imaging control unit 12 performs exposure control and has operation inputs such as an operation unit (not shown), determines the operation of the entire system according to those inputs, controls the optical system 11, the imaging unit 13, and the like, It governs mediation control of the entire system.

  The imaging unit 13 is formed to include an imaging element 131 such as a CCD or CMOS sensor, receives an optical image of a subject passing through the optical system 11 and converts it into an image signal, and converts YUV format image data from the converted image signal. Generate and output to the image processing unit 14.

  The image processing unit 14 stores the image data supplied from the imaging unit 13 in the memory 16, and performs signal processing on the image signal stored in the memory 16.

In the motion amount detection unit 15, correlation between successive images is obtained from a series of moving image data obtained by the imaging unit 13 and subjected to image processing by the image processing unit 14, and a motion amount of the entire image between the images is obtained. .
The motion amount detection unit 15 cuts out an arbitrary block BLK from the moving image screen, and calculates a correlation value V that is the degree of correlation between the motion amount D between successive images.
Next, the motion amount detection unit 15 shifts the position where an arbitrary block BLK is cut out based on the motion amount D.
Then, the motion amount detection unit 15 calculates the motion amount Da and the correlation value Va again between the blocks that are shifted out.

The motion amount detection unit 15 compares the calculated correlation value V with a preset threshold value Vth. If the correlation value V falls below the preset threshold value Vth as a result of comparison, an arbitrary block is determined based on the motion amount D. Shift the position to cut out BLK.
Then, the motion amount detection unit 15 calculates the motion amount Da and the correlation value Va again between the blocks that are shifted out.

In addition, the motion amount detection unit 15 compares the calculated motion amount D with a preset threshold value Dth, and if the motion amount D exceeds the preset threshold value Dth as a result of the comparison, the motion amount detection unit 15 selects an arbitrary value based on the motion amount D. The position where the block BLK is cut out is shifted.
Then, the motion amount detection unit 15 calculates the motion amount Da and the correlation value Va again between the blocks that are shifted out.

As described above, the motion amount detection unit 15 obtains a correlation between successive images, and detects a correlation value V representing the amount of motion D between images and the degree of correlation between images.
Based on this result, the imaging apparatus 10 can display or record a moving image without blurring on a display unit (not shown) or perform other image processing.

  Hereinafter, the motion amount detection process in the present embodiment will be described in more detail.

FIG. 2 is a first diagram for explaining image clipping according to the present embodiment.
FIG. 3 is a second diagram for explaining image clipping according to the present embodiment.

As shown in FIG. 2, the motion amount detection unit 15 detects the motion amount between successive images from a series of input motion image data. The block BLK is cut out using the whole or part of the image as a detection target area.
As shown in FIG. 3, the blocks BLKn and BLKn + 1 are cut out at the same coordinate position in the image n and the image n + 1 in the series of moving image data.
The motion amount detection in this block may be performed at one location in the image, but may be performed at a plurality of locations where the image is divided into blocks.

FIG. 4 is a diagram illustrating a correlation function when a shift amount between images is small as a result of correlation calculation.
FIG. 5 is a diagram illustrating a correlation function when the amount of deviation between images is large as a result of correlation calculation.
FIG. 6 is a diagram illustrating a process of cutting out a block again at a position shifted by the amount of motion detected at the cutting position in FIG.
FIG. 7 is a diagram illustrating a method of approximating by a method such as a least square method and obtaining a peak value by an approximation function.

The motion amount detection unit 15 performs a correlation operation on the two cut out blocks BLKn and BLKn + 1.
If the shift amount between images is small as a result of the calculation, a correlation function having a high correlation value is obtained as shown in FIG.
The correlation value obtained at this time is a correlation value in pixel units.
On the other hand, as shown in FIG. 7, a function such as a sinc function or a Gaussian function is approximated by a method such as a least square method, and the peak position of the approximate function is obtained to detect the amount of motion with a precision less than a pixel unit. can do.
In FIG. 7, the horizontal axis indicates the amount of movement, and the vertical axis indicates the correlation value.

However, if the shift amount between images is large, a function having a small correlation value can be obtained as shown in FIG.
In this case, since the sharpness is small, the estimation accuracy of the motion amount in the pixel unit or less is lowered. If the imaging device is greatly shaken during moving image shooting, the proportion of the same subject in the cut out block is reduced, and the correlation value is reduced.
Therefore, as shown in FIG. 6, the block is cut out again at the position shifted by the amount of motion detected at the cutting position in FIG.

As a result, the proportion of the same subject in the cut out block increases, and a high correlation value is obtained when the correlation calculation is performed again in this block.
Since the sharpness is large, it is possible to accurately estimate the amount of motion on a pixel-by-pixel basis based on this.

FIG. 8 to FIG. 10 show the above-described series of processes as motion detection operations in flowcharts.
The first, second, and third motion detection operations according to this embodiment will be described below in association with FIGS.

First, the first motion amount detection operation with the above configuration will be described with reference to the flowchart of FIG.
FIG. 8 is a flowchart for explaining the first motion amount detection operation according to the present embodiment.

<Step ST1>
In step ST1, the motion amount detection unit 15 acquires an image n.

<Step ST2>
In step ST2, the motion amount detector 15 acquires an image n + 1.

<Step ST3>
In step ST3, the motion amount detection unit 15 cuts out a detection target block from the image.

<Step ST4>
In step ST <b> 4, the motion amount detection unit 15 performs a phase correlation calculation with the cut out blocks, and detects a motion amount D and a correlation value V in units of pixels.

<Step ST5>
In step ST5, the motion amount detector 15 cuts out the block of the image n + 1 again based on the detected motion amount.

<Step ST6>
In step ST <b> 6, the motion amount detection unit 15 performs a phase correlation calculation with the cut out blocks, and detects the motion amount Da and the correlation value Va in units of pixels.

<Step ST7>
In step ST7, the motion amount detection unit 15 estimates the motion amount with an accuracy equal to or less than a pixel unit based on the correlation calculation result.

<Step ST8>
In step ST8, the motion amount detection unit 15 repeats the processes in steps ST2 to ST7 until the photographing is completed.

Next, the second motion amount detection operation will be described with reference to the flowchart of FIG.
FIG. 9 is a flowchart for explaining the second motion amount detection operation according to the present embodiment.
In the second motion amount detection operation, as a result of performing the correlation calculation on the block cut out at the cut-out position in FIG. 3, when the detected correlation value is equal to or less than the preset threshold value Vth, the blur amount is large or the image Judge that the contrast is low.
When the amount of blur is large, the proportion of the same subject in the two images is small, so the correlation value is small.
If it can be determined that the amount of blur is small, the amount of motion in pixel units or less is estimated from the correlation calculation result.
If the shake amount is large, the block is cut out again at the position shifted by the detected motion amount, and the correlation calculation is performed.
The overall process of the second motion amount detection operation will be described.

<Step ST11>
In step ST11, the motion amount detection unit 15 acquires the image n.

<Step ST12>
In step ST12, the motion amount detection unit 15 acquires an image n + 1.

<Step ST13>
In step ST13, the motion amount detection unit 15 cuts out a detection target block from the image.

<Step ST14>
In step ST <b> 14, the motion amount detection unit 15 performs a phase correlation calculation on the cut out blocks, and detects a motion amount D and a correlation value V in units of pixels.

<Step ST15>
In step ST15, the motion amount detection unit 15 determines whether or not the detected correlation value V is greater than the threshold value Vth (whether or not it has fallen below).

<Step ST16>
In step ST15, when the detected correlation value V is equal to or less than the threshold value Vth, in step ST16, the motion amount detection unit 15 cuts out the block of the image n + 1 again based on the detected motion amount.

<Step ST17>
In step ST <b> 17, the motion amount detection unit 15 performs a phase correlation calculation with the cut out blocks, and detects the motion amount Da and the correlation value Va in units of pixels.

<Step ST18>
In step ST18, the motion amount detection unit 15 estimates the motion amount with an accuracy equal to or less than a pixel unit based on the correlation calculation result.

<Step ST19>
In step ST19, the motion amount detection unit 15 repeats the processes in steps ST12 to ST18 until the photographing is completed.

Next, the third motion amount detection operation will be described with reference to the flowchart of FIG.
FIG. 10 is a flowchart for explaining the third motion amount detection operation according to the present embodiment.
In the third motion amount detection operation, when the correlation calculation is performed on the block cut out at the cut-out position in FIG. 3, it is determined that the shake amount is large when the detected motion amount is larger than the preset value Dth. To do.
If the amount of blur is small, the amount of motion in pixel units or less is estimated from the correlation calculation result.
If the shake amount is large, the block is cut out again at the position shifted by the detected motion amount, and the correlation calculation is performed.
The overall process of the third motion amount detection operation will be described.

<Step ST21>
In step ST21, the motion amount detection unit 15 acquires the image n.

<Step ST22>
In step ST22, the motion amount detection unit 15 acquires an image n + 1.

<Step ST23>
In step ST23, the motion amount detection unit 15 cuts out a detection target block from the image.

<Step ST24>
In step ST <b> 24, the motion amount detection unit 15 performs a phase correlation calculation on the cut out blocks, and detects a motion amount D and a correlation value V in units of pixels.

<Step ST25>
In step ST25, the motion amount detection unit 15 determines whether or not the detected motion amount D is greater than the threshold value Dth (whether or not it exceeds).

<Step ST26>
In step ST25, when the detected motion amount D is larger than the threshold value Dth, in step ST26, the motion amount detection unit 15 cuts out the block of the image n + 1 again based on the detected motion amount.

<Step ST27>
In step ST <b> 27, the motion amount detection unit 15 performs a phase correlation calculation with the cut out blocks, and detects the motion amount Da and the correlation value Va in units of pixels.

<Step ST28>
In step ST28, the motion amount detection unit 15 estimates the motion amount with an accuracy equal to or less than a pixel unit based on the correlation calculation result.

<Step ST29>
In step ST29, the motion amount detection unit 15 repeats the processes in steps ST22 to ST28 until shooting is completed.

As described above, according to the present embodiment, in the motion amount detection unit 15, the correlation between successive images obtained from the series of moving image data obtained by the imaging unit 13 and subjected to image processing by the image processing unit 14 is calculated. Then, the movement amount of the entire image between the images is obtained.
The motion amount detection unit 15 cuts out an arbitrary block BLK from the moving image screen, and calculates a correlation value V that is the degree of correlation between the motion amount D between successive images.
Next, the motion amount detection unit 15 shifts the position where an arbitrary block BLK is cut out based on the motion amount D.
Then, the motion amount detection unit 15 calculates the motion amount Da and the correlation value Va again between the blocks that are shifted out.
Therefore, according to the present embodiment, the following effects can be obtained.

The amount of motion between successive images can be detected accurately with a pixel unit or less.
That is, it is possible to realize fine motion amount detection of an image by obtaining a correlation with a block cut out at a position shifted by the motion amount.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 11 ... Optical system, 12 ... Imaging control part, 13 ... Imaging part, 14 ... Image processing part, 15 ... Motion amount detection part, 16 ... memory.

Claims (6)

A motion amount detection device that obtains a correlation between successive images from a series of moving image data and obtains a motion amount of the entire image between the images,
An arbitrary block is cut out from the screen of the moving image, a correlation value that is the degree of correlation between the amount of motion between successive images is calculated, and if the correlation value falls below a preset threshold value , based on the amount of motion And a processing unit that shifts a position to cut out the arbitrary block and calculates the motion amount and the correlation value again between the blocks cut out by the shift. A motion amount detection device that obtains a correlation between successive images from a series of moving image data and obtains a motion amount of the entire image between the images,
An arbitrary block is cut out from the screen of the moving image, a correlation value that is a degree of correlation between the amount of motion between successive images is calculated, and when the amount of motion exceeds a preset threshold value , based on the amount of motion And a processing unit that shifts a position to cut out the arbitrary block and calculates the motion amount and the correlation value again between the blocks cut out by the shift. A motion amount detection method that obtains a correlation between successive images from a series of moving image data and obtains a motion amount of the entire image between the images,
A first calculation step of cutting out an arbitrary block from the screen of a moving image and calculating a correlation value that is a degree of correlation and a motion amount between consecutive images;
When the correlation value falls below a preset threshold value, a shift step of shifting a position to cut out the arbitrary block based on the amount of movement;
A motion amount detection method comprising: a second calculation step of calculating the motion amount and the correlation value again between the blocks cut out by shifting. A motion amount detection method that obtains a correlation between successive images from a series of moving image data and obtains a motion amount of the entire image between the images,
A first calculation step of cutting out an arbitrary block from the screen of a moving image and calculating a correlation value that is a degree of correlation and a motion amount between consecutive images;
When the amount of motion exceeds a preset threshold, a shift step of shifting the position to cut out the arbitrary block based on the amount of motion;
A motion amount detection method comprising: a second calculation step of calculating the motion amount and the correlation value again between the blocks cut out by shifting. Optical system,
An image sensor that images a subject image that has passed through the optical system;
A motion amount detection device that obtains a correlation between successive images from a series of moving image data obtained by the image sensor and obtains a motion amount of the entire image between the images, and
The movement amount detection device includes:
An arbitrary block is cut out from the screen of the moving image, a correlation value that is the degree of correlation between the amount of motion between successive images is calculated, and if the correlation value falls below a preset threshold value , based on the amount of motion And a processing unit that shifts a position where the arbitrary block is cut out and calculates the motion amount and the correlation value again between the blocks cut out after the shift. Optical system,
An image sensor that images a subject image that has passed through the optical system;
A motion amount detection device that obtains a correlation between successive images from a series of moving image data obtained by the image sensor and obtains a motion amount of the entire image between the images, and
The movement amount detection device includes:
An arbitrary block is cut out from the screen of the moving image, a correlation value that is a degree of correlation between the amount of motion between successive images is calculated, and when the amount of motion exceeds a preset threshold value , based on the amount of motion And a processing unit that shifts a position where the arbitrary block is cut out and calculates the motion amount and the correlation value again between the blocks cut out after the shift.

Images