KR101371826B1 - Zoom motion estimation apparatus and method - Google Patents

Abstract

A zoom motion estimation apparatus and an estimation method thereof are disclosed. The zoom motion estimation apparatus according to the present invention includes: an information obtaining unit which obtains color information from a color camera and depth information from a depth camera; an average calculating unit which calculates averages of color information and depth information about a block in a current screen and averages of color information and depth information about a search block in a reference screen based on the color information and the depth information which are obtained by the information obtaining unit; a zoom ratio calculating unit which calculates a zoom ratio from the average value of the depth information about the current block and a reference block; and a difference signal calculating unit which calculates a movement estimation differential signal between the current block and the reference block by zooming in or out the reference screen depending on the zoom ratio which is calculated by the zoom ratio calculating unit.

Description

Expansion motion estimation device and its estimation method {Zoom Motion Estimation Apparatus and Method}

The present invention relates to an apparatus for estimating stretch movement and a method for estimating the same, and more particularly, to an apparatus for estimating stretch movement using a color camera and a depth camera and an estimating method thereof.

H.264 and MPEG-4 Part 10 coding schemes have been enacted as video coding standards, and the HEVC standard is currently in the completion stage. In video coding, motion estimation and compensation methods are essentially used to remove time direction redundancy. For motion estimation, H.264, HEVC, and the like use a block matching method for finding the nearest block in the current picture (a bundle of neighboring pixels in the spatial direction) and the reference picture neighboring in the temporal direction.

In the block matching motion estimation process, an evaluation scale for finding a block, a size of a search area in a reference picture, and a size of a current block should be considered. Since motion estimation occupies more than 70% of the implementation complexity in video encoding, research on a fast motion estimation method for reducing complexity has been conducted since the beginning of video encoding. Such block-matched motion estimation has a high accuracy in estimating the left and right movements of the camera and the spatial movement of objects in the image, but inevitably decreases the accuracy in image stretching.

The stretching movement of the image means that the current screen is enlarged or reduced in the reference screen, and in order to accurately extract the stretching ratio corresponding to the enlargement and reduction, it is necessary to apply all possible stretching ratios. In addition, because the number of possible stretch ratios is too large, it is impossible to apply all cases. Therefore, researches on a method for reducing the complexity of the implementation have been conducted.

In the first step of stretching motion estimation, the first step is to briefly estimate the motion for a few selected pixels, and the second step is to increase the accuracy for all the pixels, and to use it for various iterative least-squares estimations considering the computational efficiency and accuracy. Improvement methods, methods using interpolated reference pictures, and methods of simplifying motion estimation using 3-D diamond search patterns have been proposed.

However, most of the existing researches use only color information of a camera, which makes it difficult to estimate an accurate stretch motion vector, and there is a limit to reducing the complexity of the estimation.

The present invention has been made to solve the above problems, and an object of the present invention is to provide an apparatus for estimating stretch movement and a method for estimating stretch movement using a color camera and a depth camera.

According to an aspect of the present invention, there is provided an apparatus for estimating stretch, comprising: an information obtaining unit obtaining color information from a color camera and obtaining depth information from a depth camera; An average for calculating the average of the color information and the depth information of the block in the current screen and the average of the color information and the depth information of the search block in the reference screen based on the color information and the depth information acquired by the information obtaining unit. A calculator; A stretch ratio calculator for calculating a stretch ratio from an average value of depth information of the current block and the reference block; And a difference signal calculator configured to calculate a motion estimation difference signal between the current block and the reference block by enlarging or reducing the reference screen according to the stretch ratio calculated by the stretch ratio calculator.

The information acquisition unit may acquire RGB or YUV color information of the video from the color camera, and obtain distance information between the camera and the object from the depth camera.

The information obtaining unit may obtain the color information in pixel units and obtain the depth information in mm units.

The depth camera may be implemented as an infrared pulse laser operating in a time-of-flight (TOF) manner.

The expansion ratio calculation unit may calculate the expansion ratio S based on the following equation.

는 현재블록 내 각 화소의 깊이정보로부터 산출된 블록의 평균 깊이정보를 의미하고,

는 참조블록에 대한 평균 깊이정보를 의미하며,

,

,

는 상수값을 의미한다.Where Pc is the current screen and Pr is the reference screen,

Denotes average depth information of a block calculated from depth information of each pixel in the current block,

Means the average depth information for the reference block,

,

,

Means a constant value.

=0으로 가정하여 다음과 같은 관계식을 얻을 수 있다.The telescopic motion estimator,

Assuming = 0, the following relation can be obtained.

.

.

이면, S>1로 되어 참조블록은 확대되고,

이면, S<1로 되어 참조블록은 축소된다.At this time,

, Then S> 1, and the reference block is enlarged,

If S <1, then the reference block is reduced.

Stretch motion estimation method according to an embodiment of the present invention for achieving the above object comprises the steps of obtaining color information from the color camera, depth information from the depth camera; On the basis of the color information and depth information obtained by the color information and depth information acquiring step, the average of the color information and depth information for the block in the current screen, and the average of the color information and depth information for the search block in the reference screen Calculating; Calculating a stretching ratio from an average value of depth information of the current block and the reference block calculated by the average calculating step; And calculating a motion estimation difference signal between the current block and the reference block by enlarging or reducing the reference screen according to the expansion ratio calculated by the expansion ratio calculation step.

In the acquiring color information and depth information, RGB or YUV color information of a video may be obtained from the color camera, and distance information between the camera and an object may be obtained from the depth camera.

In the obtaining of the color information and the depth information, the color information may be obtained in pixel units, and the depth information may be obtained in mm units.

The depth camera may be implemented as an infrared pulse laser operating in a time-of-flight (TOF) manner.

In the stretching ratio calculating step, the stretching ratio S may be calculated based on the following equation.

는 현재블록 내 각 화소의 깊이정보로부터 산출된 블록의 평균 깊이정보를 의미하고,

는 참조블록에 대한 평균 깊이정보를 의미하며,

,

,

는 상수값을 의미한다.Where Pc is the current screen and Pr is the reference screen,

Denotes average depth information of a block calculated from depth information of each pixel in the current block,

Means the average depth information for the reference block,

,

,

Means a constant value.

=0으로 가정하여 다음과 같은 관계식을 얻을 수 있다.Stretch motion estimation method,

Assuming = 0, the following relation can be obtained.

.

.

이면, S>1로 되어 참조블록은 확대되고,

이면, S<1로 되어 참조블록은 축소될 수 있다.
At this time,

, Then S> 1, and the reference block is enlarged,

In this case, S <1 and the reference block can be reduced.

According to the present invention, by obtaining the distance information between the current block and the reference block from the depth camera, and by calculating the expansion ratio between the two blocks from the distance information, it is possible to reduce the motion estimation difference signal by stretching the reference block according to the expansion ratio Will be.

In addition, according to the present invention, it is possible to increase the motion estimation accuracy without having a large complexity for the stretching motion estimation.

FIG. 1 is a diagram illustrating an example of measuring the number of pixels of each diamond object by distance.
2 is a diagram schematically illustrating an apparatus for estimating stretch motion according to an embodiment of the present invention.
3 is a diagram illustrating two screens of a specific time of an image.
4 is a diagram illustrating a stretch ratio for a P-screen.
5 is a flowchart illustrating a stretching motion estimation method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, a detailed description of known techniques well known to those skilled in the art may be omitted.

In describing the constituent elements of the present invention, the same reference numerals may be given to constituent elements having the same name, and the same reference numerals may be given thereto even though they are different from each other. However, even in such a case, it does not mean that the corresponding component has different functions according to the embodiment, or does not mean that the different components have the same function. It should be judged based on the description of each component in the example.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

In describing the components of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are intended to distinguish the constituent elements from other constituent elements, and the terms do not limit the nature, order or order of the constituent elements. When a component is described as being "connected", "coupled", or "connected" to another component, the component may be directly connected or connected to the other component, Quot; may be "connected,""coupled," or "connected. &Quot;

Before describing the present invention, the relationship between depth information and color information will be described.

When the object is closer to the camera, the corresponding color information is enlarged. On the contrary, when the object is far from the camera, the corresponding color information is reduced. Therefore, the stretching ratio of the color information may be determined in consideration of the distance information between the object and the camera. However, the characteristics of the depth and color camera used should be understood. In the present invention, Kinect depth and color cameras are used, and the relationship between the two cameras is examined through experiments.

Depth cameras can be implemented with infrared pulse lasers operating in a time-of-flight (TOF) manner. Here, the infrared pulse laser includes an infrared laser diode, and the time until the pulsed light emitted from the diode is irradiated into the space and scattered on the surface of the subject is detected by the photodetector of the infrared pulse laser. By measuring this time, you can measure the distance to the object. The measured distance information of the subject is composed of a screen, and the screen is configured for each pixel.

In the embodiment of the present invention, as shown in FIG. 1, the diamond-shaped object was disposed between 1m and 4m and moved at 50cm intervals to investigate depth information and color information. The color information measured the number of pixels in the diamond horizontal diagonal direction.

The number of pixels P measured according to the distance R can be approximated as shown in Equation 1 based on the fitting algorithm.

[Equation 1]

,

,

는 상수값이며,

=221.65,

=0.9,

=-9.65로 조사되었다. Kinect 카메라가 올바르게 작동하는 거리 범위가 1m~4m 사이임을 고려하여 그 범위 내로 국한하여 조사하였다.here,

,

,

Is a constant value,

= 221.65,

= 0.9,

= -9.65. Considering that the range in which the Kinect camera works properly is between 1 m and 4 m, the research is limited to that range.

To date, the block standardization method is used for motion estimation. The present invention proposes a stretch motion estimation method that can be used in conjunction with block matching.

Block matching is a process of dividing the current screen into a specific size and estimating the nearest block from the reference screen. The smaller the current intra-block size for motion estimation, the greater the accuracy of the estimation, but the disadvantage is that the complexity required for estimation increases and the additionally transmitted motion vector increases. In H.264, the block size can be variably selected from 16x16, 8x8, and 4x4.

2 is a diagram schematically illustrating an apparatus for estimating stretch motion according to an embodiment of the present invention.

Referring to FIG. 2, the stretch motion estimating apparatus 100 may include an information obtaining unit 110, an average calculating unit 120, a stretching ratio calculating unit 130, and a difference signal calculating unit 140.

The information obtaining unit 110 obtains color information from the color camera and obtains depth information from the depth camera. The information acquisition unit 110 acquires RGB or YUV color information of the video from the color camera, and obtains depth information, that is, distance information between the camera and the object from the depth camera. Here, the color information is in pixel units, and the depth information is generally in mm units.

On the other hand, for motion estimation, a current screen for motion coding and a reference screen for motion searching are required. The reference screen can be both before or after the current screen. At this time, the movement of the current screen is estimated in units of blocks of a specific size.

The average calculating unit 120, based on the color information and the depth information obtained by the information obtaining unit 110, the average of the color information and depth information for the block in the current screen, and the color information for the search block in the reference screen Calculate the average of and depth information. That is, the average calculator 120 obtains color information of a corresponding block in the current screen, and the depth information obtains an average value of the blocks. Similarly, the reference picture gets the color information of the searched block and the depth information is the average value of the block.

The expansion ratio calculating unit 130 calculates the expansion ratio from the average value of the depth information for the current block and the reference block.

The difference signal calculator 140 obtains a motion estimation difference signal between the current block and the reference block by enlarging or reducing the reference picture according to the stretch ratio calculated by the stretch ratio calculator 130.

The detailed description is as follows.

를 구하며 또한, 참조블록에 대해서도 평균 깊이정보

를 구한다. The average calculator 120 averages depth information of a block from depth information of each pixel in the current block.

And average depth information for the reference block

.

The expansion ratio calculation unit 130 obtains the number of pixels for distance information for the current and reference blocks from Equation 1, and calculates the expansion ratio S of the reference block based on the current block as shown in Equation 2.

&Quot; (2) &quot;

The difference signal calculator 140 enlarges or reduces the reference block in consideration of the expansion ratio, and obtains a motion estimation difference signal from the current block with respect to the enlarged or reduced reference block.

=-9.65의 값이 작으므로

=0으로 가정하면, 수학식 3과 같이 간단한 관계식이 얻어진다.In this case, Equation 2 may be directly applied.

Since the value of = -9.65 is small

Assuming = 0, a simple relationship is obtained as shown in equation (3).

&Quot; (3) &quot;

이면, S>1로 되어 참조블록은 확대되고,

이면, S<1로 되어 참조블록은 축소된다.That is, it can be seen that the stretch ratio is directly related to the distance ratio between the current block and the reference block. At this time,

, Then S> 1, and the reference block is enlarged,

If S <1, then the reference block is reduced.

Kinect depth and color cameras were used to acquire real images and to measure stretch ratio. The image is characterized by a person moving back and forth with respect to the camera at a distance of 1 m and 4 m from the camera.

3 shows two screens of a specific time of an image. Each screen consists of 640 pixels in the horizontal direction and 480 pixels in the vertical direction. A total of 120 screens were used for the simulation at 30 screens per second.

The stretching ratio between the current block and the reference block calculated by Equation 3 is shown in FIG. 4. Here, motion estimation uses motion estimation in the H.264 IBP structure, and a reference block refers to a block used for motion compensation and a current block in a search region of a reference picture. It can be seen that only the stretching ratio for the continuous P-screen is expanded, decreased, and expanded from the stretching ratio. There is also an enlargement of 1.02 or more, and a decrease of 0.95 or less.

5 is a flowchart illustrating a stretching motion estimation method according to an embodiment of the present invention.

1 to 5, the information acquisition unit 110 obtains color information from a color camera and obtains depth information from a depth camera (S110). At this time, the information acquisition unit 110 obtains RGB or YUV color information of the video from the color camera, and obtains depth information, that is, distance information between the camera and the object from the depth camera.

The average calculating unit 120, based on the color information and the depth information obtained by the information obtaining unit 110, the average of the color information and depth information for the block in the current screen, and the color information for the search block in the reference screen And the average of the depth information (S120). That is, the average calculator 120 obtains color information of a corresponding block in the current screen, and the depth information obtains an average value of the blocks. Similarly, the reference picture gets the color information of the searched block and the depth information is the average value of the block.

The expansion ratio calculating unit 130 calculates the expansion ratio from the average value of the depth information of the current block and the reference block (S130).

The difference signal calculator 140 obtains a motion estimation difference signal between the current block and the reference block by enlarging or reducing the reference picture according to the stretch ratio calculated by the stretch ratio calculator 130 (S140).

The present invention is not necessarily limited to these embodiments, as all the constituent elements constituting the embodiment of the present invention are described as being combined or operated in one operation. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. In addition, such a computer program may be stored in a computer-readable medium such as a USB memory, a CD disk, a flash memory, etc., and read and executed by a computer, thereby implementing embodiments of the present invention. As the storage medium of the computer program, a magnetic recording medium, an optical recording medium, a carrier wave medium, or the like may be included.

Furthermore, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined in the Detailed Description. Terms used generally, such as terms defined in a dictionary, should be interpreted to coincide with the contextual meaning of the related art, and shall not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present invention.

The foregoing description is merely illustrative of the technical idea of the present invention and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. In addition, the embodiments disclosed in the present invention are not intended to limit the scope of the present invention but to limit the scope of the technical idea of the present invention. Accordingly, the scope of protection of the present invention should be construed according to the claims, and all technical ideas within the scope of equivalents should be interpreted as being included in the scope of the present invention.

Claims (11)

An information obtaining unit obtaining color information from a color camera and obtaining depth information from a depth camera;
An average for calculating the average of the color information and the depth information of the block in the current screen and the average of the color information and the depth information of the search block in the reference screen based on the color information and the depth information acquired by the information obtaining unit. A calculator;
A stretch ratio calculator for calculating a stretch ratio from an average value of depth information of the current block and the reference block; And
A difference signal calculator for estimating a motion estimation difference signal between the current block and the reference block by enlarging or reducing the reference screen according to the stretch ratio calculated by the stretch ratio calculator.
Stretching motion estimation apparatus comprising a.
The method of claim 1,
The information obtaining unit obtains,
And obtaining RGB or YUV color information of the video from the color camera, and obtaining distance information between the camera and the object from the depth camera.
The method of claim 1,
The information obtaining unit obtains,
And stretching the color information in pixel units and acquiring the depth information in mm units.
The method of claim 1,
The depth camera is a telescopic motion estimation device, characterized in that implemented by an infrared pulse laser operating in a time-of-flight (TOF) method.
The method of claim 1,
The expansion ratio calculating unit calculates the expansion ratio S based on the following equation: Expansion movement estimating apparatus, characterized in that:

Here, Pc is depth information of the pixel in the current block in the current picture and Pr is depth information of the pixel in the reference block in the reference picture,

Denotes average depth information of a block calculated from depth information of each pixel in the current block,

Means the average depth information for the reference block,

,

,

Is a constant value.
6. The method of claim 5,

A telescopic motion estimator comprising the following relations assuming = 0:

.
The method according to claim 6,

, Then S> 1, and the reference block is enlarged,

In this case, S <1, and thus the reference block is reduced.
Obtaining color information from a color camera, and obtaining depth information from a depth camera;
On the basis of the color information and depth information obtained by the color information and depth information acquiring step, the average of the color information and depth information for the block in the current screen, and the average of the color information and depth information for the search block in the reference screen Calculating;
Calculating a stretching ratio from an average value of depth information of the current block and the reference block calculated by the average calculating step; And
Calculating a motion estimation difference signal between the current block and the reference block by enlarging or reducing the reference screen according to the expansion ratio calculated by the expansion ratio calculation step;
Stretch motion estimation method comprising a.
The method of claim 8,
The color information and depth information obtaining step,
And obtaining RGB or YUV color information of the video from the color camera, and obtaining distance information between the camera and the object from the depth camera.
The method of claim 8,
The color information and depth information obtaining step,
And obtaining the color information in pixel units and acquiring the depth information in mm units.
The method of claim 8,
The depth camera is a stretch movement estimation method, characterized in that implemented by an infrared pulse laser operating in a time-of-flight (TOF) method.

Images