KR20190027445A - Method and Device of Motion Estimation for Depth Video Coding by curved surface Modeling, and NON-TRANSITORY COMPUTER READABLE RECORDING MEDIUM - Google Patents

Abstract

According to an embodiment of the present invention, provided is a method for estimating a motion in encoding of a depth image through curved surface modeling, which comprises the steps of: dividing a current screen on an image plane having an object and a background projected on a camera coordinate system into a target block of an m*n (m and n are a positive integer) size; searching for an i*j (i and j are a positive integer equal to or greater than m and n) of a reference screen on the image plane to select a reference block of the m*n size; detecting information on first and second curved surfaces by respectively modeling the first curved surface in the target block and the second curved surface in the reference block; converting the second curved surface in the reference block on the basis of information of the first and second curved surfaces; calculating a difference signal between the first curved surface and the converted second curved surface; and encoding at least one of a motion vector in accordance with a displacement of the difference signal, the target block and the reference block, and the information on the first and second curved surfaces.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion estimation method and apparatus for depth image coding using curved surface modeling and a non-transitory computer readable recording medium,

The present invention relates to a motion estimation method and apparatus in depth image coding and a non-transitory computer readable recording medium. And more particularly, to a method and apparatus for detecting the shape information of an object in a depth image and encoding the depth image based on the shape information of the object, and a non-transitory computer readable recording medium.

In general, motion estimation means finding a block in the current screen and a block closest to the current screen in the temporal direction with respect to the color signal or brightness signal. Usually, the smaller the block size, the larger the accuracy of estimation, but the complexity required for estimation is increased. In MPEG-1 and 2, blocks are composed of 16 pixels in the horizontal direction and 16 pixels in the vertical direction with respect to the lightness signal. In H. 264, block sizes of 16x16, 8x8, 4x4, etc. can be variably selected.

MSE (Mean Square Error) and MAE (Mean Absolute Error) are used as evaluation scales for motion estimation. The accuracy of MSE is excellent, but MAE is widely used because of its high complexity.

Recently, motion estimation methods applied to color signals or lightness signals are being applied to depth images. However, since the depth image has a three-dimensional motion in the temporal flow unlike the color image, the method of obtaining the motion vector considering only the planar motion in the existing color image can not be applied to the depth image.

Korean Patent Publication No. 10-2011-0121003 A1

The present invention provides a motion estimation method and apparatus capable of estimating motion between depth signals.

The present invention also provides a method and apparatus for estimating a curved surface through a depth value of an area made up of an in-depth curved surface.

Also, the present invention provides a motion estimation method and apparatus capable of estimating motion for an object showing three-dimensional motion.

In addition, the present invention provides a motion estimation method and apparatus capable of estimating a motion vector in consideration of three-dimensional motion such as rotation, stretching, and the like as well as planar motion of a curved surface.

According to an embodiment of the present invention, there is provided a method for processing an object on a camera coordinate system, the method comprising the steps of: dividing a current screen on an image plane on which an object on a camera coordinate system and a background is projected, into object blocks of m * n (m, n is a positive integer) Selecting a reference block of size m * n by searching an area of i * j (where i and j are positive integers of m, n or more) on a reference screen on the image plane; Detecting information on the first and second curved surfaces by modeling a first curved surface in the target block and a second curved surface in the reference block, respectively; Converting a second curved surface in the reference block based on the information of the first and second curved surfaces; Calculating a difference signal between the first curved surface and the transformed second curved surface; And coding at least one of the difference signal, the motion vector according to the displacement of the target block and the reference block, and the first and second curved surface information. .

The method of claim 1, further comprising detecting shape information of an object based on a depth value of at least one of the current screen and the reference screen, The motion estimation method may be provided in the depth image coding through the curved surface modeling in which each of the second curved surfaces in the reference block is modeled as a spherical surface, a concave surface, or an ellipsoidal surface.

The step of detecting information of the first and second curved surfaces may be performed by modeling a first equation composed of a coordinate of the object corresponding to the object block on the camera coordinate system and a first undetermined parameter, Determining a value of the first microcrystalline parameter to generate a first determination parameter based on the first determination parameter and the coordinate value of the first pixels constituting the target block, 1 prediction depth value; And modeling a second equation consisting of a coordinate of the object corresponding to the reference block on the camera coordinate system and a second undetermined parameter and determining a value of the second undetermined parameter from the second equation, And determining a second predicted depth value of each of the second pixels based on the second determination parameters and the coordinate values of the second pixels constituting the reference block, A motion estimation method may be provided in the depth image encoding.

The converting the second curved surface in the reference block may further include calculating a first error between the second predicted depth value and a second measured depth value of each of the second pixels; And calculating a second error between the first predicted depth value and the first error, wherein the step of calculating a difference signal between the first curved surface and the transformed second curved surface comprises: A motion estimation method may be provided in the depth image encoding through the curved surface modeling that calculates the difference between the first measurement depth value of each of the pixels and the difference of the second error.

Also, it is preferable to calculate the difference signal between each of a plurality of m * n-size reference blocks in the i * j region and the target block, and to calculate a sum of absolute errors of the difference signals or a square error A motion estimation method may be provided in depth image coding through a curved surface modeling that generates a reference block having a minimum sum and a motion vector according to a displacement between the target block and the target block.

In addition, a motion estimation method may be provided in the depth image coding through the curved surface modeling that encodes the difference between the motion vector and the first and second determination parameters.

Also, the first and second equations may provide a motion estimation method in depth image coding through surface modeling, which is one of an equation of a sphere, an equation of a concave surface, and an equation of an ellipse.

In another aspect, at least one memory for storing instructions; And at least one processor, the instructions being executable by the processor to cause the processor to perform operations, the operations comprising: displaying an object on the camera coordinate system and a current screen on the image plane in which the background is projected dividing a target block of m * n (m, n is a positive integer) size; Selecting an m * n-size reference block by searching an area of i * j (where i and j are positive integers of m, n or more) on the reference screen on the image plane; Modeling the first curved surface in the target block and the second curved surface in the reference block to detect information of the first and second curved surfaces; Transform a second curved surface in the reference block based on the information of the first and second curved surfaces; Calculate a difference signal between the first curved surface and the transformed second curved surface; A motion vector according to the displacement of the target block and the reference block, and at least one of the first and second curved surface information; A motion estimation unit may be provided in the depth image encoding through the curved surface modeling.

In another aspect, a non-volatile computer readable medium storing the instructions may be provided.

Embodiments of the present invention can estimate a surface through depth information in a depth image.

In addition, embodiments of the present invention can estimate the motion vector by removing the influence of motion of various curved objects such as spherical, concave, and elliptical surfaces.

In addition, an embodiment of the present invention can estimate a motion vector considering rotation and contraction of an object.

In addition, the embodiment of the present invention enables precise motion estimation by converting a curved surface modeling factor of a reference block into a curved surface modeling factor of a target block.

1 is a block diagram illustrating a depth image processing apparatus configured to process a depth image according to an exemplary embodiment of the present invention.
2A is an exemplary diagram of a motion estimation method according to an embodiment of the present invention.
2B is a flowchart of information detection of the first and second curved surfaces.
2C is a flowchart related to a method of converting the second curved surface.
3 shows that one point on the camera coordinate system is projected onto the image plane.
Fig. 4 is an exemplary diagram for conversion of the second curved surface. Fig.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention and methods of achieving them will be apparent with reference to the embodiments described in detail below with reference to the drawings. However, the present invention is not limited to the embodiments described below, but may be implemented in various forms. In the following embodiments, the terms first, second, and the like are used for the purpose of distinguishing one element from another element, not the limitative meaning. Also, the singular expressions include plural expressions unless the context clearly dictates otherwise. Also, the terms include, including, etc. mean that there is a feature, or element, recited in the specification and does not preclude the possibility that one or more other features or components may be added. Also, in the drawings, for convenience of explanation, the components may be exaggerated or reduced in size. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and thus the present invention is not necessarily limited to those shown in the drawings.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or corresponding components throughout the drawings, and a duplicate description thereof will be omitted .

FIG. 1 is a block diagram illustrating a motion estimation apparatus for depth image coding through surface modeling according to an embodiment of the present invention. Referring to FIG.

The motion estimation apparatus 100 may provide the video data to the data reception apparatus 200. [

The motion estimation apparatus 100 and the data receiving apparatus 200 may be used in various applications such as desktop computers, laptop computers (e.g., laptop computers), tablet computers, set top boxes, telephone handsets such as so- Pads, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, and the like.

In some implementations, the motion estimation apparatus 100 and the data receiving apparatus 200 may be provided with a configuration 10 for wireless communication.

Also, the data receiving apparatus 200 may receive the image data processed through the computer readable medium.

The computer-readable medium may include any type of media or device capable of moving video data imaged from the motion estimation apparatus 100 to the data receiving apparatus 200. In one example, the computer-readable medium may include a communication medium, such as a transmission channel, that enables the motion estimation apparatus 100 to transmit video data directly to the data receiving apparatus 200 in real time.

The processed image data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the data receiving apparatus 200. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency spectrum or one or more physical transmission lines. The communication medium may form part of a packet based network, such as a global network such as a local area network, a wide area network, or the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the motion estimation apparatus 100 to the data receiving apparatus 200. In some instances, image processed image data may be output from the output interface 130 to a computer readable storage medium, such as a non-transitory computer readable storage medium, i.e., a data storage device. Similarly, the image data may be accessed from the storage device by the input interface 230 of the data receiving apparatus 200. [ The storage device may be a variety of distributed or removable media such as hard drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or nonvolatile memory, or any other suitable digital storage media for storing image data. And non-volatile data storage media that are locally accessed. In a further example, the storage device may correspond to a file server or other intermediate storage device that may store the image data generated by the motion estimation apparatus 100.

The data receiving apparatus 200 may access the stored image data from the storage device through streaming or downloading.

In the example of FIG. 1, the motion estimation apparatus 100 may include an image source 110 and an image processing unit 120. In addition, the motion estimation apparatus 100 may further include an output interface 130.

The data receiving apparatus 200 may include an input interface 230 and a data processing unit 220. In addition, the data receiving apparatus 200 may further include a display device 210.

In another example, the motion estimation apparatus 100 and the data processing unit 220 may include other components.

For example, the motion estimation apparatus 100 may receive an external video source, such as an image from an external camera, and the external camera may be a depth image capturing device that generates a depth image. Likewise, the data receiving device 200 may interface with the external display device 210 rather than with the integrated display device 210.

The image source 110 of the motion estimation apparatus 100 may include a depth image capture device, such as a camera, an archive containing previously captured depth images, and / or an interface for receiving depth images from a depth image content provider . ≪ / RTI >

In some embodiments, the depth imaging device may provide a depth image that represents scene depth information in 256-level 8-bit images or the like. The number of bits for representing one pixel of the depth image can be changed instead of 8 bits. The depth image capturing device can measure the distance from the depth image capturing device to the object and the background by using infrared rays or the like to provide a depth image having a value proportional or inversely proportional to the distance.

The pixel value of the depth image may be, for example, depth information in the form of integers in mm, but not limited to, RGB color information, for example.

Each of motion estimation apparatus 100 and data receiving apparatus 200 may include one or more memories and one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof.

The memory includes instructions (e.g., executable instructions) such as computer readable instructions or processor readable instructions. The instructions may include one or more instructions executable by the computer, such as by each of the one or more processors.

For example, the one or more instructions may be executable by one or more processors to perform operations including one or more processors determining a motion vector of a depth screen and processing a depth screen to encode a depth screen have.

In particular, the image processing unit 120 may include one or more memories 121 for storing instructions and at least one processor 122 for executing the instructions.

The processor 122 of the image processing unit 120 may be configured to apply the techniques for determining the motion vector of the depth image.

In some implementations, the processor 122 of the image processing unit 120 may be configured to apply techniques for encoding depth images.

In another embodiment, the processor 122 of the image processing unit 120 may be configured to apply a technique of encoding and decoding a depth image.

The data processing unit 220 may be configured to transmit the depth screen data from the image processing unit 120 to an external device, display, analyze, and the like.

In some implementations, the data processing unit 220 may be configured to decode the encoded depth screen data from the image processing unit 120.

Although not shown in FIG. 1, in some embodiments, the motion estimation apparatus 100 and the data processing apparatus may be an integrated apparatus. For example, the motion estimation apparatus 100 may be configured to encode a depth picture and to decode the encoded depth picture.

FIG. 2A is an exemplary diagram of a motion estimation method according to an embodiment of the present invention, FIG. 2B is a flowchart of information detection of first and second curved surfaces, and FIG. 2C is a flowchart of a second curved surface transformation method. Fig. 3 shows that one point on the camera coordinate system is projected onto the image plane, and Fig. 4 shows an example of conversion of the second curved surface.

Referring to FIGS. 2A and 3, a motion estimation method according to an embodiment of the present invention is a method of estimating a motion of an object on a camera coordinate system and a current screen on an image plane on which a background is projected, using m * n (m, n is a positive integer) (S120) of searching for a reference block of size m * n by searching an area of i * j (where i and j are m and positive integers of n or more) on the reference screen on the image plane, (S130) of detecting the shape information of the object based on the depth value of at least one of the current screen and the reference screen, modeling the first curved surface in the target block and the second curved surface in the reference block, (S140) of transforming a second curved surface in the reference block based on the information of the first and second curved surfaces (S150), a difference signal between the first curved surface and the converted second curved surface (S160), checking whether all the reference blocks in the search area are searched A step S180 of measuring the sum of the absolute errors of the difference signals or squared errors calculated between the curved surfaces of the target block and the curved surfaces of each of the detected reference blocks, (S190) coding at least one of a motion vector according to a displacement of the reference block, a first and a second curved surface information, and a difference of a determination parameter.

Also, based on the shape information of the object detected in step S130, the first curved surface in the target block and the second curved surface in the reference block may be modeled by spherical, concave, and ellipsoidal modeling.

2B, the step S140 of detecting the information of the first and second curved surfaces models the first equation composed of the coordinates of the object corresponding to the target block on the camera coordinate system and the first undetermined parameters S141), a value of the first undetermined parameter is determined from the first equation to generate a first determination parameter (S142), and based on the coordinate values of the first pixels constituting the object block, And determining a first predicted depth value of each of the pixels (S143). Further, a second equation composed of the coordinates of the object corresponding to the reference block on the camera coordinate system and the second undetermined parameter is modeled (S145), and the value of the second undetermined parameter is determined from the second equation, (S146), and determining a second predicted depth value of each of the second pixels based on the second determination parameters and the coordinate values of the second pixels constituting the reference block (S147).

In addition, the first and second equations described above may vary according to each of the shape modeling encoding modes of the object.

Referring to FIG. 2C, the step of transforming the second curved surface in the reference block S150 includes a step S151 of calculating a first error between the second predicted depth value and the second measured depth value of each of the second pixels, And calculating a second error between the first predicted depth value and the first error (S152), and calculating a difference signal between the first curved surface and the transformed second curved surface (S160) A difference signal between the first measurement depth value and the second error of each of the pixels can be calculated.

Each step of the motion estimation method according to the embodiment of the present invention will be described in more detail.

An object on the camera coordinate system and a projected background Flat  The current screen is m * n size To target block  (S110).

(X, Y, Z) is projected onto a point with the (x, y) coordinates of the image plane, and the Z value is given by (x , y) is the depth value in the coordinates. The depth value may be expressed in units of pixels.

Specifically, a scene photographed when a depth image is photographed is projected onto an image plane, which is a virtual plane, with each point in the photographing region. If one point P = (X, Y, Z) in the camera coordinate system is projected to p = (h, w) which is one point in the image plane with the origin as the origin, if the depth value in the pixel containing p is d , The relation between P and p satisfies the following formula (1).

If the world coordinate system, which is a three-dimensional coordinate system, and the camera coordinate system, which is a three-dimensional coordinate system, are inconsistent with each other before performing the steps described in S110, coordinates Conversion may be performed.

Also, coordinate transformation may be performed to match the positions of the centers of the camera coordinate system and the image plane coordinate system when the centers of the camera coordinate system and the image plane coordinate system do not coincide with each other before performing the step described in S110.

[Equation 1]

X = dh / f

Y = dw / f

Z = d

In Equation 1, f is the focal length of the camera, which is defined as the distance between the origin and the image plane on the camera coordinate system.

The processor 122 of the motion estimation apparatus 100 may divide the current screen on the image plane on which the object on the camera coordinate system and the background are projected into blocks of a first predetermined size.

For example, the current screen can be divided into blocks of m * n. Therefore, each of the plurality of blocks can be defined as an area consisting of m * n pixels.

For example, a block of m * n can be defined as an area consisting of 8 * 8 pixels or 16 * 16 pixels. When the resolution is increased, the basic unit can be defined as an area consisting of 32 * 32 or 64 * 64 pixels. However, the present invention is not limited to this, and the processor 122 may calculate the motion estimation difference signal of the current screen without dividing the current screen.

In addition, the predetermined first size may be predetermined in consideration of the accuracy and complexity inversely proportional to each other.

For example, as the size of a block decreases, the accuracy of motion estimation increases. However, since the number of blocks increases, the complexity of an operation required for estimation increases and information of a motion vector increases. Therefore, the first size of the block can be set in consideration of the accuracy and the complexity.

In addition, the processor 122 may divide the current screen into a plurality of blocks, and store information on the positions of the divided blocks and depth values in the pixels in the memory 121. [

The processor 122 Current screen  And the reference screen, the shape information of the object may be detected based on the depth value of at least one of the pixels (S120).

For example, the processor 122 may detect the shape information of an object based on at least one of a current screen (or a target block) and a reference screen (or a reference block) and depth values of adjacent pixels of the current screen. Each of the pixels has a depth value, and can detect the shape information of the object based on the relative depth difference between the pixels and / or the distribution of the depth value of the reference pixel and the pixels around the reference pixel.

The processor 122 can determine that the shape of the object has a shape of a spherical surface, a concave surface, or an elliptical surface based on pixel values in each block, but the present invention is not limited thereto.

For example, when a region corresponding to a target block has a shape such as a spherical surface, a curved surface, a concave surface, or an ellipsoid, the Z value gradually increases or decreases from an arbitrary pixel in a target block toward a surrounding pixel, Or a case where the degree of increase / decrease of the depth value increases or decreases from an arbitrary pixel to a surrounding pixel. Accordingly, the processor 122 can detect the shape information of the object from the distribution relationship of the depth value and the depth value of the pixels in the object block.

The processor 122 performs modeling of either the spherical surface, the concave surface, or the ellipsoidal surface modeling on the first curved surface in the target block and the second curved surface in the reference block based on the shape information of the detected object, It is possible to generate a curved surface modeling factor for each. In addition, the processor 122 may generate the predicted depth values of the pixels of the target block and the reference block, respectively, based on each of the curved surface modeling encoding modes. A detailed method will be described later.

Searching the i * j area of the reference screen on the image plane to select an m * n reference block (S130) .

The processor 122 of the motion estimation apparatus 100 can search a second-sized search area of a reference screen on an image plane on which an object on a camera coordinate system and a background are projected.

Here, the reference screen may be the depth screen of the previous frame of the current screen or the depth screen of the next frame of the current screen, and may be a reference screen, Size reference block can be selected. Here, the search area of the second size may be the size of i * j.

In addition, the reference point of the search area may be located at the same position in space with respect to the position of the target block as a motion estimation target among a plurality of blocks of the current screen. That is, it can be the same position as the position of the target block in the reference screen.

In addition, the processor 122 may select a reference block of a first size in a search area of a second magnitude about a reference point.

In addition, the following operations (S140 to S160) may be performed for each reference block, with each block in the search area being a reference block.

In addition, the processor 122 may store in the memory 121 the information of the position of the reference block and the depth value in the pixel.

Hereinafter, a plurality of modeling methods according to shape information of an object, a decision parameter generation method and a prediction depth value generation method according to each modeling method will be described in detail.

- spherical surface modelling

The first curved surface in the target block and the second curved surface in the reference block Each By modeling  Detecting information of the first and second curved surfaces (S140).

The processor 122 determines the coordinates (X, Y, Z) of an arbitrary point P on the camera coordinate system to be projected to one of the pixels in the object block and the first undetermined parameters a, b, c: center coordinates of the sphere on the camera coordinate system, r: radius of the modeled sphere) (S141).

In addition, the processor 122 may calculate coordinates of an arbitrary point P on the camera coordinate system to be projected to one of the pixels in the reference block and the coordinates of the second undetermined parameters a ', b', c ', r') (step S145).

 &Quot; (2) "

In addition, the processor 122 may transform the equation of the first sphere into the equation of the first sphere 1-1 by performing the coordinate transformation (from the camera coordinate system to the coordinate system of the image plane) of the equation of the first sphere. Then, the value of the first undetermined parameter may be determined from the equation of the first-first-order equation to generate the first determination parameter (S142).

Likewise, the processor 122 may coordinate transform the equations of the second spheres into equations of the second-first sphere. Then, the value of the second undetermined parameter may be determined from the equation of the second-first sphere to generate the second determination parameter (S146).

The method of transforming the equation of the sphere will be described in more detail. When a region composed of spherical spheres on the camera coordinate system is projected onto a plurality of pixels represented by (h, w) coordinates on the image plane, The coordinates (X, Y, Z) can be expressed as (dh / f, dw / f) according to the similarity ratio of the triangle. Thus, the processor 122 may convert the first-round equation into a first-first-order equation that satisfies equation (3), and the second-round equation into a second-first-round equation.

&Quot; (3) "

In Equation (3), h and w are the vertical and horizontal coordinates of the image plane, and f is the focal length. And d is the coordinate of the Z-axis of the projected point, so it becomes the predicted depth value in the corresponding pixel.

Equation (3) can be summarized as Equation (4) which is a quadratic equation of d.

&Quot; (4) "

Further, if the root of the equation (4) expressed by the quadratic equation of d is found, d can satisfy the equation (5).

&Quot; (5) "

In this case, d in Equation (5) can be regarded as a first predicted depth value as a depth value when the parameters a, b, c, and r of the sphere of Equation (3) are given.

Similarly, for the reference block, d in Equation (5) can be regarded as a second predicted depth value as a depth value when the parameters a ', b', c ', and r' of the sphere are given.

Also, the predicted depth value d in Equation (5) is a predicted depth variable whose value is not determined (the first predicted depth variable for the target block and the second predicted depth variable for the reference block).

의 차이가 제일 작을 때의 결정매개변수 a, b, c, r(또는 a', b', c', r')를 구함으로써 대상블록(또는 참조블록)에서 최적의 구의 방정식의 인자를 결정할 수 있다. 그리고 프로세서(122)는 인자와 대상 블록 화소 내의 좌표 값 그리고 초점거리(f)를 수학식 3에 대입하고 연산함에 따라 대상 블록 내의 화소

의 예측깊이변수 d의 값(예측깊이값)을 결정할 수 있어 프로세서(122)는 예측깊이값(대상블록에 대해서는 제1 예측깊이값, 참조블록에 대해서는 제2 예측깊이값)을 생성(S143, S147)할 수 있다.The processor 122 determines the depth value d actually measured from the predicted depth variable d and the pixel (h, w) of the image plane

(Or a ', b', c ', r') of the decision parameters a, b, c, . Then, the processor 122 substitutes the coefficient and the coordinate value in the target block pixel and the focal length f into Equation (3)

The processor 122 may generate a predicted depth value (a first predicted depth value for the target block and a second predicted depth value for the reference block) (S143, S147).

가 최소가 되도록 미결정매개변수의 값을 결정하기 위해 최소자승법을 적용할 수 있다. 이 때

는 미결정매개변수에 대해 비선형식으로 나타나므로 가우스-뉴턴법을 적용할 수 있다.At this time, the processor 122 calculates the error of the estimated depth variable of the approximated spherical surface at (h, w) of the modeled surface and the actually measured depth

The least squares method can be applied to determine the value of the undetermined parameter so that the minimum value is minimized. At this time

Gauss-Newton method can be applied since it appears nonlinearly for the undetermined parameter.

의 차로 이루어진 행렬

과

에서의 자코비안 행렬

, 예측깊이변수 값을 나타내는

은 수학식 6를 충족한다.In the Gauss-Newton method, at step n, the predicted depth variable d and the actually measured depth value at each pixel in the target block (or reference block)

A matrix consisting of

and

Jacobian procession in

, Representing the predicted depth variable value

Satisfies Equation (6).

&Quot; (6) "

,

,

,

,

을 구할 수 있다.Further, by applying the Gauss-Newton method, as shown in Equation 7,

Can be obtained.

&Quot; (7) "

의 예측깊이변수 d의 값을 결정할 수 있다. 따라서 프로세서(122)는 화소의 예측깊이값(대상블록에 대해서는 제1 예측깊이값, 참조블록에 대해서는 제2 예측깊이값)을 생성할 수 있다.The processor 122 may repeat the operation according to Equation 7 P times to obtain the spherical surface closest to the surface having the given depth value and determine the value of the undetermined parameter of the sphere equation to generate the determination parameter. Then, by substituting the coordinate value of the pixel of the sphere and the pixel of the target block (or the reference block) and the focal length f into the equation (3), the pixel of the target block

The value of the predicted depth variable d can be determined. Thus, the processor 122 may generate a predicted depth value of the pixel (a first predicted depth value for the target block and a second predicted depth value for the reference block).

를 정함에 있어

과

값은 블록 중심의 좌표인

와

를 수학식 2에 대입하여 구할 수 있고,

은 임의의 초기값

를 대입한다.

은 블록 중심의 깊이 값

에서

을 더한 수치를 초기 값으로 할 수 있다.At this time,

In determining

and

The value is the coordinate of the block center

Wow

Can be obtained by substituting in Equation (2)

Lt; RTI ID = 0.0 >

.

Is the depth value of the block center

in

Can be used as an initial value.

- Concave surface modelling

The first curved surface in the target block and the second curved surface in the reference block Each By modeling  Detecting information of the first and second curved surfaces (S140).

The processor 122 determines the coordinates (X, Y, Z) of an arbitrary point P on the camera coordinate system to be projected to one of the pixels in the object block and the first undetermined parameters a, b: a parameter forming the equation of the concave surface) of the first concave surface (S141).

In addition, the processor 122 may calculate coordinates of an arbitrary point P on the camera coordinate system to be projected to one of the pixels in the reference block and the second undetermined parameters (a ', b') in order to model the second curved surface The equation of the second concave surface of the constructed Equation (8) can be modeled (S145).

&Quot; (8) "

Further, the processor 122 may convert the equation of the first concave surface into the equation of the 1-1 concave surface by performing coordinate transformation (from the camera coordinate system to the coordinate system of the image plane) of the equation of the first concave surface. Then, the value of the first microcrystalline parameter may be determined from the equation of the first concave surface to generate the first determination parameter (S142).

Likewise, the processor 122 may coordinate transform the equations of the plurality of second concave surfaces and transform them into equations of the 2-1 concave surface. Then, the value of the second undecided parameter may be determined from the equation of the 2-1 concave surface to generate the second determination parameter (S146).

The method of converting the equation of the concave surface will be described in more detail. When a region composed of a curved surface on the camera coordinate system is projected onto a plurality of pixels represented by (h, w) coordinates on the image plane, The coordinates (X, Y, Z) of P can be expressed by (dh / f, dw / f) according to the similarity ratio of the triangle. Therefore, the processor 122 can convert the equation of the first concave surface into the equation of the 1-1 concave surface satisfying the equation (9), and the equation of the second concave surface into the equation of the 2-1 concave surface.

 &Quot; (9) "

h and w are the vertical and horizontal coordinates of the image plane, f is the focal length, and d is the predicted depth variable. And d is the coordinate of the Z-axis of the projected point, so it becomes the predicted depth value in the corresponding pixel.

의 차이(수학식 11에 따라)가 최소가 되도록 하는 매개변수(a, b)의 값을 결정할 수 있다.The processor 122 may also determine the predicted depth variable d and the measured depth value d in accordance with Equation 10,

(A, b) to minimize the difference (in accordance with equation (11)) of the parameter (a, b).

&Quot; (10) "

In Equation (10), d is the predicted depth variable, which is the first predicted depth value of the pixels of the image plane when the values of the parameters a and b of the equation (1-1) of Equation (9) are determined.

Similarly, for the reference block, d in the equation (10) can be regarded as the second predicted depth value as the depth value when the parameters a 'and b' of the concave surface are given.

Also, the predicted depth value d in Equation (10) is a predicted depth variable whose value is not determined (the first predicted depth variable for the target block and the second predicted depth variable for the reference block).

&Quot; (11) "

의 차이(

은 오차)가 제일 작을 때의 a, b(값이 결정된 매개변수: 인자, 결정매개변수)를 구함으로써 대상블록(또는 참조블록)에서 최적의 오목 곡면의 방정식을 구하고 제1 및 제2 예측깊이변수 d의 값(예측깊이값)을 결정할 수 있어 프로세서(122)는 예측깊이값(대상블록에 대해서는 제1 예측깊이값, 참조블록에 대해서는 제2 예측깊이값)을 생성할 수 있다. 이 때 모델링된 표면의 (h, w)에서의 근사된 오목 곡면의 예측깊이변수 d와 실제로 측정된 깊이

의 오차

가 최소가 되도록 미결정매개변수의 값을 결정하기 위해 최소자승법을 적용할 수 있다. 그리고

는 미결정매개변수에 대해 비선형식으로 나타나므로 가우스-뉴턴법을 적용할 수 있다.As indicated in equation (11), the depth value actually measured in the predicted depth variable d and (h, w)

Difference between

(Or reference block) by obtaining a and b (parameters whose parameters are determined as parameters, parameters, and the like) when the error is smallest and the first and second predicted depths The value of the variable d (predicted depth value) can be determined, and the processor 122 can generate a predicted depth value (a first predicted depth value for the target block and a second predicted depth value for the reference block). At this time, the predicted depth variable d of the approximated concave surface at (h, w) of the modeled surface and the actually measured depth

Error of

The least squares method can be applied to determine the value of the undetermined parameter so that the minimum value is minimized. And

Gauss-Newton method can be applied since it appears nonlinearly for the undetermined parameter.

의 차로 이루어진 행렬

과

에서의 자코비안 행렬

, 미결정매개변수 값을 나타내는

은 수학식 12를 충족한다.In the Gauss-Newton method, in step n, the value of d in each pixel in the target block (or reference block) and the actually measured depth value

A matrix consisting of

and

Jacobian procession in

, Indicating the value of the indeterminate parameter

(12) < / RTI >

&Quot; (12) "

을 구할 수 있다.Also, by applying the Gauss-Newton method, as shown in Equation 13,

Can be obtained.

&Quot; (13) "

의 예측깊이값 d(대상블록에 대해서는 제1 예측깊이값, 참조블록에 대해서는 제2 예측깊이값)를 결정(S143, S147)할 수 있다.The processor 122 may repeat the above-described operation P times to obtain a concave surface having a given depth value and a closest concave surface, and determine the value of the undetermined parameter of the concave surface equation to generate a determination parameter. Then, the processor 122 calculates a value by substituting the determined factor, the coordinate value in the target block (or reference block) pixel, and the focal length f into Equation (9). Then, the processor 122 determines whether or not the pixel

(The first predicted depth value for the target block and the second predicted depth value for the reference block) (step S143, S147).

- Ellipse modelling

The first curved surface in the target block and the second curved surface in the reference block Each By modeling  Detecting information of the first and second curved surfaces (S140).

The processor 122 determines the coordinates (X, Y, Z) of an arbitrary point P on the camera coordinate system to be projected to one of the pixels in the object block and the first undetermined parameters a, b, c, A, B, and C are parameters constituting the equation of the first ellipse) (S141).

In addition, the processor 122 may calculate coordinates of an arbitrary point P on the camera coordinate system to be projected to one of the pixels in the reference block and the coordinates of the second undetermined parameters a ', b', c ', A', B ', C') (S145).

 &Quot; (14) "

In addition, the processor 122 may convert the first ellipse equation into a first -1 ellipse equation by performing coordinate transformation (from the camera coordinate system to the image plane coordinate system) of the first ellipse equation. Then, the value of the first undetermined parameter may be determined from the equation of the first ellipse to generate the first determination parameter (S142).

Similarly, the processor 122 may coordinate transform the equations of the plurality of second ellipses into equations of the second-1 ellipse. And the second determinant parameter may be determined from the equation of the 2-1 ellipse to generate a second determinant parameter (S146).

A method of converting an ellipse equation will be described in more detail. When a region of a curved surface on a camera coordinate system is projected onto a plurality of pixels represented by (h, w) coordinates on an image plane, P (X, Y, Z) can be represented by (dh / f, dw / f) according to the similarity ratio of the triangle. Therefore, the processor 122 may convert the equation of the first ellipse to the equation of the 1-1 ellipse satisfying the equation (15), and the equation of the second ellipse to the equation of the 2-1 ellipse.

 &Quot; (15) "

h and w are the vertical and horizontal coordinates of the image plane, f is the focal length, and d is the predicted depth variable. Similarly, for the reference block, d in Equation 15 can be regarded as a second predicted depth value as a depth value when the parameters a ', b', c ', and r' of the ellipse are given.

The processor 122 also calculates the values of the first undetermined parameters a, b, c, A, B, and C based on the first predicted depth variable d and the first measured depth value of the pixels in the object block You can decide. Thus, the processor 122 may generate a first determination parameter (a, b, c, A, B, C where the value is determined).

Similarly, the processor 122 calculates the second undetermined parameters a ', b', c ', A', B ', C' based on the second predicted depth variable d and the second measured depth value of the pixels in the reference block ') Can be determined. Thus, the processor 122 may generate a second determination parameter (values a ', b', c ', A', B ', C').

의 차이(수학식 17에 따라)가 최소가 되도록 하는 결정매개변수(a, b, c, A, B, C)(또는 a', b', c', A', B', C')를 구할 수 있다.In particular, the processor 122 calculates the predicted depth variable d and the measured depth value d in accordance with equation (16)

(Or a ', b', c ', A', B ', C') to minimize the difference (in accordance with equation (17) Can be obtained.

&Quot; (16) "

D in Expression 16 is a predictive depth variable, and when the values of the parameters a, b, c, A, B, and C of the equation of the 2-1 ellipse in Equation 15 are determined, (The first predicted depth value for the target block, and the second predicted depth value for the reference block) of the corresponding pixels. However, since the value of d is not determined at the present stage, it becomes the predicted depth variable (the first predicted depth variable for the target block and the second predicted depth variable for the reference block).

&Quot; (17) "

의 차이(

은 오차)가 제일 작을 때의 a, b, c, A, B, C(값이 결정된 매개변수: 인자)를 구함으로써 대상 블록(또는 참조 블록)에서 최적의 타원의 방정식을 구하고 예측깊이변수 d의 값을 결정할 수 있다. As shown in equation (17), the predicted depth variable d and the actually measured depth value at (h, w)

Difference between

(Or reference block) by obtaining a, b, c, A, B, and C (the parameter whose value is determined) when the error is smallest Can be determined.

의 오차

가 최소가 되도록 미결정매개변수의 값을 결정하기 위해 최소자승법을 적용할 수 있다. 그리고

는 미결정매개변수에 대해 비선형식으로 나타나므로 가우스-뉴턴법을 적용할 수 있다.At this time, the predicted depth variable d of the approximated ellipse at (h, w) of the modeled surface and the actually measured depth

Error of

The least squares method can be applied to determine the value of the undetermined parameter so that the minimum value is minimized. And

Gauss-Newton method can be applied since it appears nonlinearly for the undetermined parameter.

의 차로 이루어진 행렬

과

에서의 자코비안 행렬

, 미결정매개변수 값을 나타내는

은 수학식 18를 충족한다.In the Gauss-Newton method, in step n, the value of d in each pixel in the target block (or reference block) and the actually measured depth value

A matrix consisting of

and

Jacobian procession in

, Indicating the value of the indeterminate parameter

Satisfies the expression (18).

&Quot; (18) "

을 구할 수 있다.Further, by applying the Gauss-Newton method, as shown in Equation 19,

Can be obtained.

&Quot; (19) "

의 예측깊이값 d(대상블록에 대해서는 제1 예측깊이값, 참조블록에 대해서는 제2 예측깊이값)를 결정(S143, S147)할 수 있다.The processor 122 may repeat the above-described operations P times to determine the ellipse closest to the object of the ellipse having a given depth value, and determine the value of the parameter of the first ellipse equation. Then, the processor 122 substitutes the determined factor and the coordinate value in the pixel of the target block (or reference block) and the focal length f into the equation (15). Thus, the processor 122 determines whether the pixel (s) in the target block

(The first predicted depth value for the target block and the second predicted depth value for the reference block) (step S143, S147).

Transforming a second curved surface in the reference block based on information of the first and second curved surfaces (S150).

Referring to FIG. 4, in the depth image, the curved surface modeling factors of the object may vary depending on the three-dimensional motion. Therefore, it is desirable to perform the motion estimation in consideration of the depth image. This method converts the surface modeling factor in the reference screen block into the current screen modeling surface factor and compares the changed depth value with the depth value of the current screen block.

For example, when modeling with a spherical surface for this method is as follows. (A`, b`, c`, r`) of the modeling surface of the target block and the second decision parameter of the modeling surface of the reference block are (a`, b`, c`, (X, y) in the current screen and d` (x, y) in the reference screen, the value obtained by substituting the modeled parameters and the corresponding coordinates into the equation (5) Is the estimated depth value obtained through In this case, x and y are relative coordinates in the block, not the coordinates in the whole image coordinate system. If the size of the block is 4, the range of x, y is 0 ~ 3.

라고 하면, 이 때 참조 블록에서의 모델링된 인자를 통해 계산된 제2 예측깊이값과 실제 측정된 제2 측정깊이값 간에는 오차가 있을 수 있다. 그 오차를 e`(x,y)로 두면 그 식은 수학식 20을 충족한다.Also, the actual measured depth value in the reference block is

There may be an error between the second predicted depth value calculated through the modeled factor in the reference block and the actually measured second measured depth value. If the error is set as e '(x, y), the equation satisfies the expression (20).

&Quot; (20) "

를 구할 수 있다.In this case, the method of converting the modeling factor of the reference block into the modeling factor of the target block is as follows. The second predicted depth value d '(x, y) of the reference block will be the first predicted depth value d (x, y) at the same position in the target block if the modeling factor is converted to the modeling factor of the current scene. And the error between the second predicted depth value and the second measured depth value is the same even after the modeling factor is changed. Accordingly, the processor 122 converts the curved surface modeling factor in the reference block into the first surface factor of the target block by using Equation (21)

Can be obtained.

&Quot; (21) "

와, 대상 블록의 모델링 인자로 변환된 참조 블록의 측정 깊이값

를 비교하여 차 신호를 구할 수 있다.Then, in step S160 of calculating a difference signal between the first curved surface and the converted second curved surface , the processor 122 may calculate a difference signal between the first measured depth value and the second error of each of the first pixels . That is, the first measured depth value actually measured in the target block

And a measurement depth value of a reference block converted into a modeling factor of a target block

The difference signal can be obtained.

quest Within the region  In step S170, it is checked whether all reference blocks are searched.

The processor 122 checks whether all reference blocks in the search area have been searched to perform the above-described process (S160) on the entire intra-reference search area. If not, Step S160 can be performed.

The surface of the target block Explored  The sum of the absolute errors of the difference signals calculated between the curved surfaces of each reference block or Squared error  Measuring the sum (S180) and Differential signal  (S190) encoding at least one of a motion vector according to a displacement of a target block and a reference block, a difference between first and second curved surface information, and a determination parameter.

The processor 122 measures a difference signal between each of a plurality of m * n reference blocks in the i * j region and a target block and determines a position where the SAE (or SSE) of the difference signal among the plurality of reference blocks is the lowest The difference between the difference signal and the motion vector between the target block and the reference block which is the block most similar to the target block, and the decision parameters of the first and second curved surfaces can be encoded.

In addition, based on the parameters of the curved surface (spherical or concave surface or ellipsoidal surface) determined in the step S140 of detecting the information of the first and second curved surfaces, the target block or the reference block may be a curved surface (spherical surface or concave surface or ellipsoidal surface) A step of confirming whether or not it corresponds to a part may be further performed.

More specifically, the processor 122 calculates a value e by adding the square sum of all the parameters in the matrix Fp matrix representing the error obtained after applying the Gauss-Newton method P times, as shown in equation (22). In this case, if e in the target block and e in the reference block are equal to or greater than a predetermined threshold value T, the motion estimation is performed based on the curved surface modeling based on the concrete information of the curved surface. It is possible to apply another motion estimation method without applying the motion estimation method using the motion estimation method. Here, the other motion estimation method may be a known motion estimation method, but is not limited thereto.

&Quot; (22) "

On the other hand, the memory 121 may store instructions to be executed by the processor 122 so that the processor 122 computes Equations (1) to (21) as described above.

The embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention or may be those known and used by those skilled in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, medium, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code, such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be modified into one or more software modules for performing the processing according to the present invention, and vice versa.

The specific acts described in the present invention are, by way of example, not intended to limit the scope of the invention in any way. For brevity of description, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of such systems may be omitted. Also, the connections or connecting members of the lines between the components shown in the figures are illustrative of functional connections and / or physical or circuit connections, which may be replaced or additionally provided by a variety of functional connections, physical Connection, or circuit connections. Also, unless explicitly mentioned, such as " essential ", " importantly ", etc., it may not be a necessary component for application of the present invention.

While the present invention has been described in connection with what is presently considered to be practical and exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

100: motion estimation device
110: image source
120:
121: Memory
122: Processor
130: Output interface
200: Data receiving device
210: Display Device
220:
230: Input Interface

Claims (9)

Dividing a current picture on an image plane on which an object on a camera coordinate system and a background is projected into object blocks of m * n (m, n is a positive integer) size;
Selecting a reference block of size m * n by searching an area of i * j (where i and j are positive integers of m, n or more) on a reference screen on the image plane;
Detecting information on the first and second curved surfaces by modeling a first curved surface in the target block and a second curved surface in the reference block, respectively;
Converting a second curved surface in the reference block based on the information of the first and second curved surfaces;
Calculating a difference signal between the first curved surface and the transformed second curved surface; And
And coding at least one of the difference signal, the motion vector according to the displacement of the target block and the reference block, and the first and second curved surface information
Motion Estimation Method in Depth Image Coding by Surface Modeling. The method according to claim 1,
Detecting the shape information of the object based on the depth value of at least one of the current screen and the reference screen,
Modeling the first curved surface in the target block and the second curved surface in the reference block as one of a spherical surface, a concave surface, and an ellipsoidal surface based on the shape information of the object
Motion Estimation Method in Depth Image Coding by Surface Modeling. The method according to claim 1,
Wherein the step of detecting information on the first and second curved surfaces comprises:
Modeling a first equation consisting of a coordinate of the object corresponding to the object block on the camera coordinate system and a first undetermined parameter, determining a value of the first undetermined parameter from the first equation, Determining a first prediction depth value of each of the first pixels based on the first decision parameter and the coordinate value of the first pixels constituting the target block; And
Modeling a second equation consisting of a coordinate of the object corresponding to the reference block on the camera coordinate system and a second undetermined parameter and determining a value of the second undetermined parameter from the second equation, And determining a second prediction depth value of each of the second pixels based on the second decision parameter and the coordinate value of the second pixels constituting the reference block
Motion Estimation Method in Depth Image Coding by Surface Modeling. The method of claim 3,
Wherein transforming the second curved surface in the reference block comprises:
Calculating a first error between the second predicted depth value and a second measured depth value of each of the second pixels; And
Calculating a second error between the first predicted depth value and the first error,
Wherein calculating the difference signal between the first curved surface and the transformed second curved surface comprises calculating a difference signal between the first measurement depth value of each of the first pixels and the difference signal of the second error
Motion Estimation Method in Depth Image Coding by Surface Modeling. 5. The method of claim 4,
Calculating a difference signal between each of a plurality of m * n-size reference blocks in the i * j region and the target block, and calculating a sum of absolute errors or a square error of the difference signal among the plurality of reference blocks, And generates a motion vector according to a displacement between the reference block that is the smallest and the target block
Motion Estimation Method in Depth Image Coding by Surface Modeling. 6. The method of claim 5,
Coding the difference between the motion vector and the first and second decision parameters
Motion Estimation Method in Depth Image Coding by Surface Modeling. The method of claim 3,
Wherein the first and second equations are any one of an equation of a sphere, an equation of a concave surface and an equation of an ellipse
Motion Estimation Method in Depth Image Coding by Surface Modeling. At least one memory for storing instructions; And
At least one processor,
The instructions being executable by the processor to cause the processor to perform operations,
The operations include:
Dividing the current picture on the image plane on which the object on the camera coordinate system and the background is projected into object blocks of size m * n (m, n is a positive integer);
Selecting an m * n-size reference block by searching an area of i * j (where i and j are positive integers of m, n or more) on the reference screen on the image plane;
Modeling the first curved surface in the target block and the second curved surface in the reference block to detect information of the first and second curved surfaces;
Transform a second curved surface in the reference block based on the information of the first and second curved surfaces;
Calculate a difference signal between the first curved surface and the transformed second curved surface;
A motion vector according to the displacement of the target block and the reference block, and at least one of the first and second curved surface information;
Containing
Motion Estimation in Depth Image Coding by Surface Modeling. 9. A non-transitory computer readable medium storing the instructions according to claim 8.

Images