JP6727902B2 - Reference frame generation device, motion estimation device, and program - Google Patents

Description

The present invention relates to a reference frame generation device that generates a new reference frame, a motion estimation device, and a program.

In inter coding in video coding, motion compensation using the result of motion estimation (motion detection) between frames is performed in order to improve coding efficiency. In this motion estimation, a block matching method is generally used for reasons such as easy hardware implementation. However, the block matching method has low accuracy of rotation and depth direction motion estimation. Therefore, a motion estimation method involving geometric transformation has been proposed (for example, see Non-Patent Documents 1 and 2).

In addition, a method of performing encoding using super-resolution processing is known. For example, Patent Document 1 describes a method of performing distributed video coding using super-resolution processing. Specifically, the original image sequence is encoded every few frames using a conventional intra-encoding method such as JPEG, and this is used as a Key frame, and the other frames are used as Wyner-Ziv frames. Parity syndrome bits are generated and transmitted by Wolf encoding. Further, on the encoding side, first, the Key frame is locally decoded to generate a Wyner-Ziv frame prediction signal as side information. On the decoding side, error correction is performed using the parity syndrome for side information. In addition, the prediction efficiency is improved by applying the super-resolution processing to the translation region using the decrypted key frame and Wyner-Ziv frame.

Japanese Patent No. 5484378

R. Takada, S. Orihashi, Y. Matsuo, and J. Katto, "Improvement of 8K UHDTV Picture Quality for H.265/HEVC by Global Zoom Estimation using Motion Vectors", Proceedings of IEEE ICCE, pp. 64-65, 2015 VCEG C1016, Huawei technologies Co. Ltd., "Affine transform prediction for next generation video coding", Video Coding Experts Group-C1016

However, with conventional motion estimation involving geometric transformation, blurring that occurs when moving from the inside to the outside of the depth of field in the movement in the depth direction of the object, and from the outside to the inside of the depth of field. Due to the sharpening that occurs when there is motion, the accuracy of motion estimation becomes low. Even if the detection can be performed with high accuracy, the difference information amount at the time of motion compensation tends to be large.

Further, although the method of Patent Document 1 can improve the prediction efficiency by super-resolution processing, it does not improve the accuracy of motion estimation for the motion in the depth direction because the depth of field is not taken into consideration. ..

An object of the present invention made in view of such circumstances is a reference frame generation device that generates a reference frame for improving the accuracy of motion estimation with respect to a motion in the depth direction, and a motion estimation device that performs motion estimation using the reference frame. , And providing those programs.

In order to solve the above-mentioned problems, a reference frame generation device according to the present invention is a reference frame generation device which inputs one reference frame from an original image sequence and outputs a plurality of new reference frames, wherein the reference frame and the reference frame From the depth of the standard frame of the original image sequence and the angle of view of the camera, the scaling factor of the object of the standard frame with respect to the reference frame is obtained, and from the depth and the depth of field of the camera, the standard from the reference frame. the depth-direction motion estimation unit for estimating a motion in the depth direction with respect to the depth of field of the object to the frame, relative to the reference frame, performs one or more geometric transformation processing with the variable magnification processing in the magnification ratio And a geometric transformation processing unit for generating one or more geometric transformation images, and one or more low-pass filter processings for the geometric transformation processing by changing cutoff frequencies one or more times. A filter processing unit that generates an image and outputs it as a new reference frame, and a high-resolution process that is performed once or more by changing the high-frequency band to be added to the geometric transformation process, and one or more super-resolution processes And a super-resolution processing unit that generates an image and outputs it as a new reference frame, and the filter processing unit is determined by the depth direction motion estimation unit that the object has moved from inside to outside the depth of field. In this case, the greater the distance the object of the reference frame is from the depth of field, the lower the cut-off frequency is set, and the super-resolution processing unit causes the depth direction motion estimation unit to detect the object in the depth of field. If it is determined that moved within the outside of the depth, the object of the reference frame, characterized in that you set high frequency band to the additional greater the distance that deviates from the depth of field.

Further, in order to solve the above problems, a motion estimation device according to the present invention uses the reference frame generation device, the one or more filtered images and the one or more super-resolution processed images as reference frames, and the standard frame. And a motion estimator that detects a motion vector between and.

In order to solve the above problems, the program according to the present invention causes a computer to function as the reference frame generation device.

In order to solve the above-mentioned subject, the program concerning the present invention makes a computer function as the above-mentioned motion estimating device.

According to the present invention, it becomes possible to generate a reference frame for improving the accuracy of motion estimation with respect to a motion in the depth direction.

It is a block diagram which shows the structural example of the reference frame production|generation apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the reference frame production|generation apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structural example of the reference frame production|generation apparatus which concerns on the 3rd Embodiment of this invention. It is a block diagram which shows the structural example of the motion estimation apparatus which concerns on the 4th Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
First, a reference frame generation device according to the first embodiment of the present invention will be described. FIG. 1 shows a configuration example of a reference frame generation device according to the first embodiment of the present invention. In the example illustrated in FIG. 1, the reference frame generation device 10 includes a geometric conversion processing unit 101, a filter processing unit 102, and a super-resolution processing unit 103.

The reference frame generation device 10 inputs one reference frame from an original image sequence composed of a base frame and a plurality of reference frames, and generates and outputs a plurality of new reference frames (new reference frames).

The geometric conversion processing unit 101 performs geometric conversion processing on the input reference frame at least once (n times) by changing geometric conversion parameters including scaling factors, and performs at least one (n) geometric conversion image. (Geometrically transformed image group) is generated. Then, the geometrically transformed image group is output to the filter processing unit 102 and the super-resolution processing unit 103. For example, the geometric transformation processing unit 101 performs rotation, scaling (scaling), and skew by affine transformation. Equation (1) shows an affine transformation equation. Here, (u, v) are horizontal/vertical coordinates before conversion, and (u′, v′) are horizontal/vertical coordinates after conversion. Further, the rotation angle θ, the scaling factors (s _x , s _y ) in the horizontal and vertical directions, and the skew angle δ are used as geometric transformation parameters. The geometrically transformed image group includes an image that is simulated by simulating a movement in the depth direction and an image that is enlarged by simulating a movement in the front direction with respect to the reference frame input to the reference frame generation device 10.

The filter processing unit 102 performs low pass filtering processing once or more (j times) on the geometrically transformed image generated by the geometric transformation processing unit 101 by changing the cutoff frequency, and then performs one or more (j (n×j) filtered images (filtered image group) are generated. Then, the filtered image group is output to the outside as a new reference frame. The filtered image group is a blurred image in which high frequency components have been removed from the reference frame input to the reference frame generation device 10.

The super-resolution processing unit 103 performs super-resolution processing once or more (k times) on the geometrically transformed image generated by the geometric transformation processing unit 101 by changing the high frequency band to be added, and then performs one or more ( (n×k) super-resolution processed images (super-resolution processed image group) are generated. Then, the super-resolution processed image group is output to the outside as a new reference frame. Any known processing can be used for the super-resolution processing, and for example, the super-resolution processing described in Japanese Patent No. 5417290 can be used. The super-resolution processed image group is a sharpened image in which a high-frequency component exceeding the Nyquist frequency is added to the reference frame input to the reference frame generation device 10.

The filtered image group and the super-resolution processed image group output from the reference frame generation device 10 can be used as a reference frame for inter coding. As an example, when used in HEVC (High Efficiency Video Coding)/H.265 inter coding, the distance between the standard frame and the reference frame is about 1 frame in the Low-delay mode and 8 frames in the Random access mode. (However, it depends on the set value). Therefore, in the geometric conversion processing in the geometric conversion processing unit 101, it is not necessary to perform rotation, scaling, and skew processing of such a large value. For example, the rotation angle θ is {−π/8,0,π/8}, the scaling factors s _x and s _y are {0.75,1.0,1.25}, and the skew angle δ is {−π. /8,0,π/8}.

Also, in the filter processing in the filter processing unit 102, it is not necessary to reduce the cutoff frequency so much. For example, the cutoff frequency is {1, 0.8, 0.6} times the Nyquist frequency of the original image sequence. Also, in the super-resolution processing in the super-resolution processing unit 103, it is not necessary to add a high frequency component exceeding the Nyquist frequency of the geometrically transformed image.

It should be noted that a computer can be preferably used to cause the above-described reference frame generation device 10 to function, and such a computer can execute a program describing the processing content for realizing each function of the reference frame generation device 10 of the computer. It can be realized by storing it in a storage unit and reading and executing this program by the CPU of the computer. The program can be recorded on a computer-readable recording medium.

As described above, the reference frame generation apparatus 10 and the program thereof perform the scaling processing and the low-pass filter processing on the original reference frame and the scaling processing for the original reference frame. And a super-resolution processed image group that has undergone processing and super-resolution processing. Therefore, it is possible to increase the number of reference frame candidates in consideration of the movement in the depth direction from the inside to the outside of the depth of field and the movement in the depth direction from the outside to the depth of field, that is, the depth direction. It is possible to generate a reference frame for improving the accuracy of motion estimation with respect to the motion of the.

(Second embodiment)
Next, a reference frame generation device according to the second embodiment of the present invention will be described. FIG. 2 shows a configuration example of a reference frame generation device according to the second embodiment of the present invention. In the example illustrated in FIG. 2, the reference frame generation device 11 includes a geometric conversion processing unit 101, a filter processing unit 102, a super-resolution processing unit 103, and a depth direction motion estimation unit 104. The reference frame generation device 11 of the second exemplary embodiment differs from the reference frame generation device 10 of the first exemplary embodiment in that a depth direction motion estimation unit 104 is further provided.

The depth direction motion estimation unit 104 inputs a standard frame and a reference frame, estimates a motion in the depth direction between the standard frame and the reference frame, and indicates a depth direction motion of the object from the reference frame to the standard frame. Generate directional motion estimation information. Then, the depth direction motion estimation information is output to the geometric conversion processing unit 101. In addition, when an object moving in the front direction and an object moving in the back direction are mixed, it is comprehensively determined in which direction many objects move. Further, when the movement of the object in the depth direction is less than or equal to the threshold value, information indicating that there is no movement of the object in the depth direction may be output as the depth direction motion estimation information.

For the motion estimation in the depth direction, it is preferable to use a technique that is robust to the motion in the depth direction, such as a technique using a particle filter. The particle filter predicts the transition of each frame of time series data based on the weighted probability distribution. The particle filter is capable of global tracking of the object and estimating the global size change of the object (represented by a box surrounding the object). It is presumed that a movement in the front direction occurs when this size increases, and a movement in the back direction occurs when this size decreases. For details of the particle filter, refer to, for example, Artech House Publishers, "Beyond the Kalman Filter: Particle Filters for Tracking Applications" (ISBN 978-1-58083-631-8).

In addition, as the motion estimation in the depth direction, a method of performing matching of SURF (Speeded-Up Robust Features) features that is robust to the motion in the depth direction or a technique of performing matching of SIFT (Scale-Invariant Feature Transform) features is used. May be. For details of SURF, for example, H. Bay, A. Ess, T. Tuytelaars, and LV Gool, "Speeded-Up Robust Features (SURF)", Elsevier, vol. 110, Issue 3, p. 346- See 359, Jun. 2008.

Also, the camera that has captured the original image sequence is equipped with an RGB-D sensor, and the camera detects the depth (Depth) from the RGB-D sensor to the subject, that is, the shooting distance, in units of pixels at every few pixels. There is a case. In this case, the depth direction motion estimation unit 104 may acquire the depths of the reference frame and the base frame from the camera and use the depths to estimate the depth direction motion of the object.

When the depth direction motion estimation information generated by the depth direction motion estimation unit 104 indicates that the object is moving in the depth direction, the geometric conversion processing unit 101 performs geometric conversion processing accompanied by reduction. Further, when the depth direction motion estimation information indicates that the object is moving in the front direction, geometric conversion processing accompanied by enlargement is performed.

The processes of the filter processing unit 102 and the super-resolution processing unit 103 are the same as those in the first embodiment. When the depth direction motion estimation information indicates that there is no movement in the depth direction of the object, the filtering process by the filter processing unit 102 and the super-resolution processing by the super-resolution processing unit 103 are not performed, The geometric transformation image generated by the geometric transformation processing unit 101 is output as it is.

It should be noted that a computer can be preferably used to cause the above-described reference frame generation device 11 to function, and such a computer can execute a program describing the processing content for realizing each function of the reference frame generation device 11 of the computer. It can be realized by storing it in a storage unit and reading and executing this program by the CPU of the computer. The program can be recorded on a computer-readable recording medium.

As described above, the reference frame generation device 11 and its program perform motion estimation of the original image sequence in the depth direction, and perform geometric transformation processing based on the result. Therefore, the generated new reference frame is an image having a smaller difference from the reference frame, and thus it is possible to generate a reference frame for improving the accuracy of motion estimation with respect to the movement in the depth direction.

(Third Embodiment)
Next, a reference frame generation device according to the third embodiment of the present invention will be described. FIG. 3 shows a configuration example of the reference frame generation device according to the third embodiment of the present invention. In the example illustrated in FIG. 3, the reference frame generation device 12 includes a geometric conversion processing unit 101, a filter processing unit 102, a super-resolution processing unit 103, and a depth direction motion estimation unit 104. The reference frame generation device 12 of the third embodiment has the same configuration blocks as the reference frame generation device 11 of the second embodiment, but is different in that processing using camera parameters is performed.

The reference frame generation device 12 inputs a reference frame, a base frame, and camera parameters. The camera parameters include the angle of view and depth of field of the camera in addition to the depths of the reference frame and the base frame.

The depth direction motion estimation unit 104 obtains the scaling factor of the object of the reference frame with respect to the reference frame from the depths of the reference frame and the reference frame, and the angle of view of the camera, and outputs the scaling factor to the geometric conversion processing unit 101. The depth direction motion estimation unit 104 may calculate the depth of field of the camera from the focal length of the lens, the aperture value, and the sensor size.

Further, the depth direction motion estimation unit 104 estimates the motion in the depth direction with respect to the depth of field of the object from the reference frame to the base frame from the depths of the reference frame and the base frame and the depth of field of the camera. That is, it is determined whether the object has moved from inside to outside the depth of field of the camera or from outside to the inside of the depth of field with respect to the reference frame from the reference frame. Then, the determination result is output to the filter processing unit 102 and the super-resolution processing unit 103.

The geometric transformation processing unit 101 generates a geometric transformation image group by performing, on the input reference frame, one or more times of geometric transformation processing accompanied by scaling processing of the scaling ratio obtained by the depth direction motion estimation unit 104. Then, it outputs to the filter processing unit 102 and the super-resolution processing unit 103. When performing the geometric conversion process using the above equation (1), the scaling factor obtained by the depth direction motion estimation unit 104 is set to the scaling factors (s _x , s _y ) in the horizontal and vertical directions.

When the depth-direction motion estimation unit 104 determines that the object has moved out of the depth of field of the camera, the filter processing unit 102 has a large distance that the object of the reference frame deviates from the depth of field. The cutoff frequency is set lower as much as possible, and the low-pass filter processing is performed once or more on the geometric transformation image generated by the geometric transformation processing unit 101 to generate a filtered image group. Then, the filtered image group is output to the outside as a new reference frame. If the depth direction motion estimation unit 104 determines that the object is not moving from within the depth of field, the filtering process is not performed.

If the depth-direction motion estimation unit 104 determines that the object has moved from outside to inside the depth of field of the camera, the super-resolution processing unit 103 determines the distance at which the object in the reference frame deviates from the depth of field. Is set to a higher value, the super-resolution processing is performed once or more on the geometric transformation image generated by the geometric transformation processing unit 101, and a super-resolution processing image group is generated. If the depth direction motion estimation unit 104 determines that the object does not move from outside to inside the depth of field, the super-resolution processing is not performed.

It should be noted that a computer can be preferably used to cause the above-described reference frame generation device 12 to function, and such a computer can execute a program describing the processing content for realizing each function of the reference frame generation device 12 of the computer. It can be realized by storing it in a storage unit and reading and executing this program by the CPU of the computer. The program can be recorded on a computer-readable recording medium.

As described above, the reference frame generation device 12 and the program thereof determine the scaling factor using the camera parameter and determine whether the object has moved out of the depth of field. Then, since the scaling processing, the filtering processing, and the super-resolution processing are performed according to the result, it is possible to generate a reference frame for improving the accuracy of motion estimation with respect to the motion in the depth direction.

(Fourth Embodiment)
Next, a motion estimation device according to the fourth embodiment of the present invention will be described. FIG. 4 shows a configuration example of the motion estimation device according to the fourth embodiment of the present invention. In the example illustrated in FIG. 4, the motion estimation device 1 includes the reference frame generation device 10, 11, or 12, and a motion estimation unit 20.

The reference frame generation device 10, 11, or 12 generates the filter processing image group and the super-resolution processing image group (new reference frame) as described above. Then, the new reference frame is output to the motion estimation unit 20.

The motion estimation unit 20 uses the filter-processed image group and the super-resolution processed image group (new reference frame) generated by the reference frame generation device 10, 11, or 12 as reference frames to perform motion estimation with the reference frame. The motion vector is detected and output to the outside. The motion estimation can be performed using any conventional method such as a block matching method.

It should be noted that a computer can be preferably used to function as the above-described motion estimation device 1, and such a computer stores a program describing the processing content for realizing each function of the motion estimation device 1 in the storage unit of the computer. It can be realized by storing the program in the computer and reading and executing the program by the CPU of the computer. The program can be recorded on a computer-readable recording medium.

In this way, the motion estimation device 1 and its program perform motion estimation using the new reference frame that takes into account the motion in the depth direction. Therefore, the accuracy of motion estimation with respect to the motion in the depth direction can be improved.

Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, it is possible to combine a plurality of constituent blocks described in the embodiment into one or to divide one constituent block.

1 Motion Estimating Devices 10, 11, 12 Reference Frame Generating Device 20 Motion Estimating Unit 101 Geometric Transformation Processing Unit 102 Filter Processing Unit 103 Super-Resolution Processing Unit 104 Depth Direction Motion Estimating Unit

Claims (4)

A reference frame generation device for inputting one reference frame from an original image sequence and outputting a plurality of new reference frames,
From the depth of the reference frame and the base frame of the original image sequence, and the angle of view of the camera, the scaling ratio of the object of the base frame with respect to the reference frame is obtained, and from the depth and the depth of field of the camera, A depth direction motion estimation unit that estimates a motion in the depth direction with respect to the depth of field of the object from the reference frame to the base frame;
Relative to the reference frame, performs one or more geometric transformation processing with the variable magnification processing in the magnification ratio, the geometric transformation processing unit to generate one or more geometric transformation image,
A filter processing unit that performs one or more low-pass filter processes for each of the geometric conversion processes by changing the cutoff frequency, generates one or more filtered images, and outputs the filtered images as a new reference frame,
A super-resolution processing unit that performs one or more super-resolution processings for each of the geometric conversion processings by changing the high frequency band to be added, generates one or more super-resolution processed images, and outputs as a new reference frame. And ,
When the depth direction motion estimation unit determines that the object has moved from within the depth of field to the outside, the filtering unit has a larger distance from the depth of field of the object of the reference frame. Set the cutoff frequency low,
If the super-resolution processing unit determines that the object has moved from outside to inside the depth of field by the depth direction motion estimation unit, the distance at which the object of the reference frame deviates from the depth of field is reference frame generating apparatus characterized that you set high frequency band to be the additional larger. A reference frame generation device according to claim 1 ;
A motion estimation unit that detects a motion vector between the one or more filtered images and the one or more super-resolution processed images as a reference frame and a standard frame of the original image sequence;
A motion estimation device comprising: A program for causing a computer to function as the reference frame generation device according to claim 1 . A program for causing a computer to function as the motion estimation device according to claim 2 .

Images