KR101946715B1 - Adaptive search ragne determination method for motion estimation of 360 degree video - Google Patents

Abstract

A method for determining an adaptive search region in motion estimation for a 360-degree image includes determining a target block in a first frame of a 360-degree image, determining a search region in a second frame for motion estimation based on the first frame, And determining a block having a minimum difference from the target block in the search area. The size of the search area is determined according to the position of the target block.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an adaptive search area determination method,

The techniques described below relate to motion estimation techniques for 360 degree images. More specifically, the technique described below relates to a search area setting technique for motion estimation of a 360 degree image.

Recently, VR (Virtual Reality) devices and contents using them have attracted attention. The VR device provides a 360-degree (all-round) image to the user. 360 degree images can be generated through various models.

On the other hand, a next generation video coding (Future Video Coding) standard having a coding efficiency twice as high as that of HEVC (High Efficiency Video Coding) / H.265, which is the latest video codec standard, is being prepared. The standardization body is also discussing standardization for 360 degree images.

US Published Patent US 2008/0137746

The technique described below is intended to provide a motion estimation technique for a 360 degree image. The technique described below is to provide a technique for adaptively setting a search area for motion estimation on a 360-degree image.

A method for determining an adaptive search region in motion estimation for a 360 degree image includes determining a target block in a first frame of a 360 degree image and determining a search region in a second frame for motion estimation based on the first frame . The size of the search area is determined according to the position of the target block. Or the size of the search area is determined according to the identifiers of the individual images constituting the target block.

The technique described below allows accurate motion estimation to be performed in consideration of distortion occurring in a 360-degree image.

1 is an example of generating a 360-degree image.
Figure 2 is an example of motion estimation based on block matching.
Figure 3 is an example of distortion that occurs in the ERP model.
4 is an example of a search area for motion estimation for a 360 degree image.
Figure 5 is an example of a system for delivering a 360 degree image.

The following description is intended to illustrate and describe specific embodiments in the drawings, since various changes may be made and the embodiments may have various embodiments. However, it should be understood that the following description does not limit the specific embodiments, but includes all changes, equivalents, and alternatives falling within the spirit and scope of the following description.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the following description, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include " should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms " comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner.

Also, in performing a method or an operation method, each of the processes constituting the method may take place differently from the stated order unless clearly specified in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

Recently, VR (Virtual Reality) service has attracted attention. The VR service basically uses a 360-degree image (omni-directional image). A 360-degree image (omnidirectional image) is an image obtained by uniformly combining images acquired by a camera (hereinafter referred to as a 360-degree camera) that captures an omnidirectional scene. A smartphone or a VR-only device provides an image at a specific point in time according to the orientation of the device using a 360-degree image. First, a process of generating a 360-degree image will be described.

1 is an example of generating a 360-degree image. The process of generating a 360-degree image includes (a) image acquisition, (b) a plurality of image stitching, and (c) mapping of combined images. The computer device receives a plurality of images and generates 360-degree images. In some cases, the encoder may generate a 360 degree image and encode the image. It is assumed that the 360-degree image generation is performed by a computer device.

The computer device acquires a plurality of images using a 360-degree camera (a process). A 360 degree camera is a device that acquires images of different directions (viewpoints) based on a specific point. The 360 degree camera includes a plurality of camera units facing different directions with respect to the reference point. Although a 360 degree camera in the form of a sphere is shown in Fig. 1, a 360 degree camera can have various forms.

The computer device combines a plurality of images obtained by the 360 degree camera with a certain reference (step b). The combining process corresponds to a process of combining a plurality of images into one image. The computer device stitches adjacent images together to produce a combined image.

Finally, the computer device performs mapping using a certain model of the combined image (step c). The mapping corresponds to the process of converting a 360-degree azimuth image into a plane image. FIG. 1 shows an example of mapping using an equirectangular projection (ERP) model. The ERP model is a method of projecting an image that is mapped to a 360-degree sphere in 2D based on a 360 degree camera. Furthermore, the computer device can map images using various other models. In addition to ERP, there are cube map projection (CMP), icosahedral projection (ISP), and octachedron projection (OCP).

In video coding, the encoder performs motion estimation on an object between frames. The encoder reduces the amount of data delivered through motion estimation for the same object. The result of the motion estimation is a motion vector. First, general motion estimation will be described. Figure 2 is an example of motion estimation based on block matching. Generally, the encoder performs block matching based motion estimation. The encoder determines the most similar block in two frames and performs motion estimation. An object on which motion estimation is performed in the current frame (first frame) is called a target block. The encoder finds the block that is closest to the target block of the current frame (least difference) in the reference frame (second frame). At this time, the encoder generally compares the similarity with the target block with respect to the block existing in a certain area (search area) in the reference frame. The encoder moves blocks in pixel units or subpixel units in the search area and selects the block with the smallest difference from the target block. A detailed description of the motion estimation process is omitted.

Now, the features of the above-described 360-degree image will be described. The following will focus on the ERP model.

Figure 3 is an example of distortion that occurs in the ERP model. 3 (a) is an example of a 360-degree image generated by an ERP model. 3 (a), the reference line is a line corresponding to the center of the vertical direction in the image. The image generated by the ERP model is distorted when the object moves up and down with respect to the reference line. As described above, ERP is a process of generating a two-dimensional plane based on a sphere such as the earth. Therefore, when describing with reference to the three-dimensional earth coordinate system, the baseline can be said to correspond to the equator. The degree of distortion is increased by 1 / cos? Based on the reference line. Where φ is the latitude. In other words, φ represents the position in the longitudinal direction with respect to a specific region of the image in the global coordinate system.

FIG. 3 (b) shows an example of the distortion generated in the image generated by the ERP model. It is assumed that the position of the reference frame is different from the position of the current frame due to the movement of a certain object in the image. For example, assume that the object in the first block in the current frame (first frame) is the same as the object in the second block in the reference frame. As shown in FIG. 3 (b), the size of the area occupied by the same object varies due to distortion occurring in the 360-degree image even though the object is the same object. The images generated by the ERP model are displayed on the equator as 1 / cos? Therefore, the accuracy of the motion estimation may be degraded if a general block matching algorithm is used for the image generated by the EPR model.

A motion estimation process for a 360-degree image will now be described. We propose a new search area setting method for block matching. 4 is an example of a search area for motion estimation for a 360 degree image. It is assumed that the encoder performs block matching based on the target block in the current frame (first frame). The encoder finds a block that matches the target block in a constant search area of the reference frame (second frame). The encoder determines the size of the search area based on the position of the target block. That is, if the position of the target block is located at a position where the distortion is severely generated according to the ERP mapping, the encoder sets the size of the search area to be wider. It is assumed that the position of the search area (the position of the center point) is basically a position used in conventional block matching. The encoder can determine only the horizontal size L of the search area according to the position of the target block.

(i) The encoder can determine the size (L) of the search area in proportion to the distance d away from the reference line of the target block. (ii) If the size of the block is determined in advance, the distance from the reference line can be determined according to the coordinate position of the block. Therefore, the encoder may determine the size L of the search area according to the position (x, y) of the target block. At this time, the coordinates of the block can be set based on the center point of the block. Depending on the embodiment, the coordinates may be set relative to other points in the block. (iii) If there is information that previously maps an identifier of a block and a position of a block, the encoder may determine the position of the target block with an identifier (ID) of the target block.

When the coordinates of the block are used, the encoder can determine the size of the search area according to Equation (1) below.

At this time,? Can be determined according to the following Equation 2 according to the coordinates of the block.

Here, it is assumed that the frame has a size of W (horizontal length) × H (vertical length). y is the vertical coordinate of the target block.

However, as the search area increases, the complexity of block matching may increase. In this case, the complexity problem can be solved by adjusting the number of search points for matching according to the extent of the search area.

Furthermore, distortions that occur in 360-degree images can occur in other models than ERP models. The encoder can set the search area according to the position of the target block even in a 360-degree image generated by using another model. The encoder can adaptively set the size of the search area in consideration of the distortion caused by the model.

In FIG. 4, the position of the target block is used as a reference. The location of the target block is related to the type of image that makes up the 360 degree image. The 360-degree image is composed of a plurality of images as described with reference to FIG. A plurality of images are images for different viewpoints (directions). A plurality of images are images obtained by different cameras (or camera units). The 360 degree image is arranged at a constant position of each of the plurality of images. The positions of the plurality of images constituting one plane image can be determined in advance according to the mapping technique.

Therefore, the encoder may determine the size of the search area according to the type of the image forming the target block. In other words, the encoder may determine the size of the search area according to the direction (viewpoint) of the image constituting the target block. In other words, the encoder may determine the size of the search area according to the type of camera that has captured the image forming the target block. In this case, the encoder can determine the size of the search area based on any one of an identifier for a plurality of images, an identifier of a camera, or a direction (viewpoint) of an image forming a target block.

Figure 5 is an example of a system 100 for delivering a 360 degree image. The system 100 includes an encoder 110 and a decoder 120. The encoder 110 receives a 360 degree image. Although not shown in FIG. 5, a separate computer device or server converts the image acquired by the 360 degree camera 50 into a 360-degree image. In some cases, the encoder 110 may convert a plurality of images received from the 360-degree camera 50 into 360-degree images.

The encoder 110 performs motion estimation on the image. At this time, the encoder 110 performs block matching by adaptively setting the search area according to the position of the target block as described above. Encoder 110 encodes a 360 degree image. At this time, the encoder 110 encodes an image using a motion vector, which is a motion estimation result.

The decoder 120 decodes the video stream in the reverse order of encoding in the bitstream delivered by the encoder 110. [ The decoder 120 decodes the 360-degree image using the image data and the moving vector.

The present embodiment and drawings attached hereto are only a part of the technical idea included in the above-described technology, and it is easy for a person skilled in the art to easily understand the technical idea included in the description of the above- It will be appreciated that variations that may be deduced and specific embodiments are included within the scope of the foregoing description.

50: 360 degree camera
100: System to transmit 360 degree image
110: Encoder
120: decoder

Claims (13)

delete delete delete delete delete delete Determining a target block in a first frame of the 360 degree image;
Determining a search area in a second frame for motion estimation based on the first frame; And
And determining a block having a minimum difference from the target block in the search area by adjusting the number of search points for block matching according to the size of the search area determined according to the identifier of the target block,
Wherein the 360 degree image is generated using a plurality of individual images, the horizontal size of the search area is determined according to the identifier of the target block, and the identifier defines a vertical position of the target block in the first frame And determining a distance between a center of the first frame and the target block in the vertical direction. delete 8. The method of claim 7,
Wherein the identifier is a 360-degree image indicating a direction of an image constituting the target block. 8. The method of claim 7,
Wherein the identifier is a 360-degree image representing a camera that captures an image constituting the target block. delete delete 8. The method of claim 7,
Wherein the size of the search area in the horizontal direction is proportional to the distance between the center of the vertical direction of the first frame and the target block.

Images