TWI686079B - Method and apparatus of processing 360-degree virtual reality images - Google Patents

Abstract

Methods and apparatus of processing 360-degree virtual reality images are disclosed. According to one method, a system that maps a motion vector in a 2D frame into a 3D space and then applies rotation of sphere to adjust motion vector in the 3D space. The rotated motion vector is then mapped back to the 2D frame for processing 360-degree virtual reality images. According to another method, a system that derives motion vector from displacement of viewpoint is disclosed. According to yet another method, a system that applies scaling to adjust motion vector is disclosed. The motion vector scaling can be performed in the 3D space by projecting a motion vector from a 2D frame to a 3D sphere. After motion vector scaling, the scaled motion vector is projected back to the 2D frame. Alternatively, the forward projection, motion vector scaling and inverse projection can be combined into a single function.

Description

Method and device for processing 360-degree virtual reality image

The invention relates to 360-degree virtual reality (VR) image/sequence image/video processing or coding and decoding. Specifically, the present invention relates to deriving motion vectors for three-dimensional (3D) content in different projection formats.

360-degree video, also known as immersive video, is an emerging technology that can provide an "immersive feeling." Surround the user with a wrap-around scene that covers the panoramic view, especially a 360-degree panorama to achieve an immersive feeling. Stereo rendering can further improve the "feeling of being there". Therefore, panoramic video is widely used in virtual reality (Virtual Reality, VR) applications.

Immersive video involves using multiple cameras to capture the scene to cover a panoramic view, for example, a 360-degree field of view. Immersive cameras usually use a panoramic camera or camera set for capturing a 360-degree field of view. Generally, two or more cameras are used for immersive cameras. All videos must be acquired at the same time, and a single segment of the scene (also called a single perspective) is recorded. In addition, camera sets are usually used to capture views horizontally, while other camera designs are possible.

Using a 360-degree spherical panoramic camera or multiple images used to cover all fields of view around 360 degrees, 360-degree VR images can be captured. Using traditional image/video processing equipment, 3D spherical images are difficult to process or store. Therefore, using 3D to 2D projection methods, 360-degree VR images are usually converted into 2D format. For example, equirectangular projection (ERP) and cubemap projection (CMP) systems have generally adopted projection methods. Therefore, 360-degree images can be stored in an isometric projection format. Isometric projection projects the entire surface of the sphere onto a flat image. The vertical axis is latitude and the horizontal axis is longitude. Figure 1 shows an example of projecting a sphere 110 onto a rectangular image 120 according to ERP, where each longitude line is mapped to a vertical line of the ERP image. For ERP projection, the area in the north and south poles of the sphere is stretched more severely than the area near the equator (ie, from a single point to the line). In addition, due to the distortion caused by stretching, especially near two poles, predictive codec tools usually cannot make good predictions, resulting in reduced codec efficiency. Figure 2 shows a cube 210 with 6 faces, in which a 360-degree VR image can be projected onto the 6 faces of the cube according to CMP. There are different ways to take the 6 faces out of the cube and combine them into a rectangular image. The example in Figure 2 divides 6 faces into two parts (ie 220a and 220b), where each part includes 3 connecting faces. These two parts can be expanded into two bands (ie 230a and 230b), where each band corresponds to a continuous surface image. Depending on the selected layout format, the two bands can be combined into a compact rectangular frame.

Such as JVET-F1003 (Y. Ye, et al., "Algorithm descriptions of projection format conversion and video quality metrics in 360Lib", Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/ SC 29/WG 11, 6th Meeting: Hobart, AU, 31 March – 7 April 2017, Document: JVET-F1003), ERP format and CMP format have been included in the projection format conversion, which is being considered for Next-generation video codec. In addition to the ERP format and the CMP format, there are different VR projection formats, for example, Adjusted Cubemap Projection (ACP), Equal-Area Projection (EAP), Octahedron Projection (OHP) ), icosahedron projection (Icosahedron Projection, ISP), segmented sphere projection (Segmented Sphere Projection, SSP) and rotating sphere projection (Rotated Sphere Projection, RSP), which are widely used in this field.

FIG. 3 shows an example of OHP, in which a sphere is projected onto eight faces of the octahedron 310. By cutting the edge of the face between face 1 and face 5, and rotating face 1 and face 5 to connect to face 2 and face 6, respectively, and applying a similar process to face 3 and face 7, removed from octahedron 310 The eight faces 320 can be converted into an intermediate format 330. The intermediate format may be encapsulated into a rectangular image 340.

Figure 4 shows an example of an ISP, in which a sphere is projected onto the 20 faces of the icosahedron 410. The 20 faces 420 from the icosahedron 410 can be encapsulated into a rectangular image 430 (referred to as a projection layout).

JVET-E0025 (Zhang et al., "AHG8: Segmented Sphere Projection for 360-degree video", Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 5th Meeting: Geneva, CH, 12–20 January 2017, Document: JVET-E0025) has disclosed SSP as a method to convert spherical images into SSP format. Figure 5 shows an example of a segmented sphere projection, in which the spherical image 500 is mapped to the north pole image 510, the south pole image 520, and the equatorial segment image 530. The boundary of the three segments corresponds to the latitude 45°N (ie 502) and latitude 45°S (ie 504), where 0° corresponds to the equator (ie 506). The north and south poles are mapped into two circle areas (ie 510 and 520), and the projection of the equatorial segment can be the same as ERP or EAP. The diameter of the circle is equal to the width of the equatorial segment because both the polar segment and the equatorial segment have a latitude span of 90°. The north pole image 510, the south pole image 520, and the equatorial segment image 530 may be packaged into a rectangular image.

FIG. 6 shows an example of RSP, in which the sphere 610 is divided into the middle 270°x90° area 620 and the remaining portion 622. Each RSP part may be further stretched on the top side and the bottom side to generate a deformed part having an elliptical shape. As shown in FIG. 6, these two ellipse-shaped parts can be adapted to the rectangular format 630.

(1)(2)The ACP system is based on CMP. If the two-dimensional coordinates (u', v') of CMP are determined, the two-dimensional coordinates (u, v) of ACP can be calculated by adjusting (u', v') according to the following set of equations:

(1) (2)

Using a table of given positions (u, v) and face index f, 3D coordinates (X, Y, Z) can be derived. For 3D to 2D coordinate conversion, given (X, Y, Z), then (u’, v’) and the surface index f can be calculated according to the CMP table. The 2D coordinates of ACP can be calculated according to the set of equations.

Similarly to ERP, EAP also maps the sphere surface to a surface. In the (u, v) plane, u and v are in the range [0, 1]. For the 2D to 3D coordinate conversion, given the sampling position (m, n), the 2D coordinate (u, v) is first calculated in the same way as the ERP. Subsequently, the longitude and latitude (ϕ, θ) on the sphere can be calculated from (u, v) as: ϕ = (u − 0.5) * (2* π) (3) θ = sin ⁻¹ (1.0 − 2* v) (4)

Finally, using the same equation as used in ERP, (X, Y, Z) can be calculated: X = cos(θ) cos(ϕ) (5) Y = sin(θ) (6) Z = −cos (θ) sin(ϕ) (7)

Conversely, using the following, longitude and latitude (ϕ, θ) can be evaluated from (X, Y, Z) coordinates: ϕ = tan−1(−Z/X) (8) θ = sin−1(Y/( X2+Y2+Z2)1/2) (9)

Since images or videos related to virtual reality may occupy a larger space for storage or a larger bandwidth for transmission, image/video compression is usually used to reduce the required storage space or transmission bandwidth. Inter prediction has become a powerful codec tool to explore inter frame redundancy using motion estimation/motion compensation. If traditional inter-frame prediction is applied to 2D frames converted from 3D space, the use of motion estimation/motion compensation techniques will not work properly, because due to object motion or relative motion between the object and the camera, Objects may become distorted or deformed in 2D frames. In order to improve the inter prediction of 2D frames converted from 3D space, different inter prediction techniques have been developed to improve the accuracy of the inter prediction of 2D frames converted from 3D space.

The invention discloses a method and device for processing 360-degree virtual reality images. According to a method, input data of a current block in a 2D frame is received, where the 2D frame is projected from a 3D sphere. The first motion vector related to the adjacent block in the 2D frame is determined, where the first motion vector points from the first start position in the adjacent block to the first end position in the 2D frame. According to the target projection, the first motion vector is projected onto the 3D sphere. The first motion vector in the 3D sphere is rotated around the rotation axis along the rotation circle on the surface of the 3D sphere to generate the second motion vector in the 3D sphere. According to the inverse target projection, the second motion vector in the 3D sphere is mapped back to the 2D frame. Using the second motion vector, the current block in the 2D frame is encoded or decoded. The second motion vector may be included as a candidate in the merge candidate list or AMVP candidate list for encoding or decoding the current block.

In one embodiment, the rotating circle corresponds to the largest circle on the surface of the 3D sphere. In another embodiment, the rotating circle is smaller than the largest circle on the surface of the 3D sphere. The target projection may correspond to ERP, CMP, ACP, EAP, OHP, ISP, SSP, RSP, or CLP.

In an embodiment, projecting the first motion vector onto the 3D sphere includes: projecting the first start position, the first end position, and the second start position in the 2D frame onto the 3D sphere according to the target projection. The two starting positions are located at the corresponding positions in the current block corresponding to the first starting positions in the adjacent blocks. Rotating the first motion vector in the 3D sphere along the rotation circle includes: determining the target rotation for rotating along the rotation circle on the surface of the 3D sphere from the first starting position to the third in the 3D sphere around the rotation axis Two starting positions; and using target rotation to rotate the first ending position to the second ending position on the 3D sphere. Mapping the second motion vector in the 3D sphere to the 2D frame includes: mapping the second end position on the 3D sphere back to the 2D frame according to the inverse target projection; and according to the second start position and the second end in the 2D frame The position determines the second motion vector in the 2D frame.

According to another method, two 2D frames are received, where two frames are projected from a 3D sphere corresponding to two different viewpoints using target projection, and the current block and neighboring blocks are located in two 2D frames. Based on two 2D frames, the previous point of the camera is determined. Multiple mobile streams are determined in two 2D frames. Based on the first motion vector associated with the neighboring block, the translation of the camera is determined. Based on the camera's translation, a second motion vector related to the current block is derived. Subsequently, using the second motion vector, the current block in the 2D frame is encoded or decoded.

For the above method, multiple moving streams in two 2D frames can be calculated from the tangent direction of each primitive in the two 2D frames. The second motion vector may be included as a candidate in the merge candidate list or AMVP candidate list for encoding or decoding the current block. The target projection may correspond to ERP, CMP, ACP, EAP, OHP, ISP, SSP, RSP, or CLP.

According to yet another method, the input data of the current block in the 2D frame is received, wherein the 2D frame is projected from the 3D sphere according to the target projection. The first motion vector related to the adjacent block in the 2D frame is determined, where the first motion vector points from the first start position in the adjacent block to the first end position in the 2D frame. The first motion vector is scaled to generate a second motion vector. Subsequently, using the second motion vector, the current block in the 2D frame is encoded or decoded.

In an embodiment, the step of scaling the first motion vector to generate the second motion vector includes: projecting the first start position, the first end position, and the second start position in the 2D frame onto the 3D sphere according to the target projection , Where the second starting position is located in the corresponding position in the current block corresponding to the first starting position in the adjacent block. The step of scaling the first motion vector to generate the second motion vector further includes: scaling the longitude component of the first motion vector to generate the scaled longitude component of the first motion vector; scaling the latitude component of the first motion vector to generate the first The scaled latitude component of the motion vector; and based on the scaled longitude component of the first motion vector and the scaled latitude component of the first motion vector, determine a second end position corresponding to the second start position. The step of scaling the first motion vector to generate the second motion vector further includes: mapping the second end position on the 3D sphere back to the 2D frame according to the inverse target projection; and based on the second start position and the second in the 2D frame The end position determines the second motion vector of the 2D frame.

In another embodiment, scaling the first motion vector to generate the second motion vector includes: applying the first combination function to generate the x component of the second motion vector, and applying the second combination function to generate the second motion vector Y component; wherein the first combined function and the second combined function are based on the first starting position, the second starting position in the current block corresponding to the first starting position, the first motion vector and the target projection; and The first combination function and the second combination function combine target projection, scaling and inverse target projection. The target projection is used to project the first data in the 2D frame to the second data in the 3D sphere, and the zoom is to select from the 3D sphere The motion vector is scaled to the scaled motion vector in the 3D sphere, and the inverse target projection is used to project the scaled motion vector into the 2D frame. In an embodiment, when the target projection corresponds to an isometric projection, the first combination function corresponds to ( ), the second combination function corresponds to the identity function, where Corresponds to the first latitude associated with the first starting position, Corresponds to the second latitude associated with the second starting position.

The following description is a preferred way of implementing the invention. The purpose of this description is to explain the general principles of the invention and is not meant to be limiting. The protection scope of the present invention shall be defined as defined in the claims.

For traditional inter prediction in video coding and decoding, motion estimation/motion compensation is widely used to explore correlations in video data in order to reduce transmission information. Traditional video content corresponds to 2D video data, and motion estimation and motion compensation techniques usually assume translational motion. In the next generation of video codecs, more advanced motion models are considered, for example, affine models. However, these techniques derive 3D motion models based on 2D images.

In the present invention, motion information is derived in the 3D domain, so that more accurate motion information can be derived. According to one method, because the blocks in the 2D video content are deformed, the rotation of the sphere is assumed. FIGS. 7A and 7B show examples of the deformation of the 2D image due to the rotation of the sphere. In FIG. 7A, the block 701 on the 3D sphere 700 is moved to become the block 702. When block 701 and moved block 702 are mapped to ERP frame 703, these two corresponding blocks become block 704 and block 705 in the ERP frame. Although the blocks on the 3D sphere (ie, 701 and 702) correspond to the same block, the two blocks (ie, 704 and 705) have different shapes in the ERP frame. In other words, the movement or rotation of the object in 3D space may cause the object in the 2D frame (that is, the ERP frame in this example) to be distorted. In FIG. 7B, the area 711 is located on the surface 710 of the sphere. Due to the rotation of the sphere along the path 713, the area 711 is moved to the area 714. Due to the rotation of the sphere, the motion vector 712 related to the area 711 is mapped to the motion vector 715. The correspondence of the parts 711-715 in the 2D domain 720 is shown as parts 721-725. In another example, the parts corresponding to the trajectories 733 on the surface 730 of the sphere corresponding to 713-725 are shown as parts 733-735, and the parts corresponding to the parts 733-735 in the 2D domain 740 are shown It is 743-745. In yet another example, the parts corresponding to the trajectories 753 on the surface 750 of the sphere corresponding to 713-725 are shown as parts 753-755, respectively, and the parts corresponding to the parts 753-755 in the 2D domain 760 are drawn Shown as 763-765.

FIG. 8 shows an example in which the sphere 800 is rotated 800 from the point a 830 on the larger circle 820 to the point b 840 on the sphere 810, where the larger circle 820 corresponds to the largest circle on the surface of the sphere 810. The rotation axis 850 is indicated by the arrow in FIG. 8. The rotation angle is θ _a .

Figure 9 shows an example of a 900 rotation of a sphere from a point a to a point b on a smaller circle 910 on a sphere 920, where the smaller circle 910 corresponds to a circle smaller than the largest circle on the surface of the sphere 920 (ie circle 930) Circle. The midpoint of rotation is shown as point 912 in Figure 9. Another example is a sphere rotation 950 from the point a on the smaller circle 960 of the sphere 970 to the point b . The larger circle 980 (ie, the largest circle) is shown in FIG. 9. The rotation axis 990 is shown as an arrow in FIG. 9.

、記號

、記號

和記號

為(θ,Φ)域中之三維座標。 Figure 10 shows the use of a sphere rotation model to derive motion vectors for 2D projected images. In FIG. 10, schematic 1010 describes the use of a sphere rotation model to derive a motion vector for a 2D projected image according to an embodiment of the present invention. Position a and position b are two positions in the 2D projection image. The motion vector mv _a points from a to a ′. Find the target system for the motion vector positions b mv _b. The present invention uses 2D to 3D projection to project a 2D projection image onto the sphere 1020. The rotation around the smaller circle or the larger circle is applied to rotate from position a to position b . According to the same sphere rotation selected, position a'is rotated to position b' . Subsequently, the back projection is applied to the 3D sphere to convert the position b′ to the 2D frame 1030. B is a motion vector position can mv _b = _b '- _b is calculated. The motion vector mv _a and the motion vector mv _b are two-dimensional vectors in the (x, y) domain. Position a, position a ', the position b and the position b' two-dimensional coordinates (x, y) of the domain. mark

,mark

,mark

And mark

Is the three-dimensional coordinate in the (θ,Φ) domain.

The derivation MV can be used as a candidate for video encoding and decoding using merge mode or AMVP mode, where, as disclosed in High Efficiency Video Coding (HEVC), both merge mode and AMVP mode are predictive Ground codec block or block motion information technology. When encoding and decoding blocks in the merge mode, the current block uses the motion information of the block indicated by the merge index, which points to the candidate selected in the merge candidate list. When encoding and decoding a block in the AMVP mode, the motion vector of the current block is predicted by the predictor indicated by the AMVP index, and the AMVP index points to the selected candidate in the AMVP candidate list. Figure 11 shows the use of the MV derived based on the rotation of the sphere as a merge candidate or AMVP candidate. In Fig. 11, the neighboring blocks for deriving the layout of the combined candidate 1110 is shown in a block 1112, wherein the block includes a spatially adjacent neighboring blocks A _0, spatially neighboring blocks A _1, spatially adjacent blocks B ₀ , spatially adjacent block B ₁ and spatially adjacent block B ₂ , and temporally adjacent block Col ₀ or temporally adjacent block Col ₁ . For the current block 1120 in the 2D frame, the motion vectors from block A ₀ and block B ₀ can be used to derive motion candidates for the current block. As disclosed above, according to the rotation of the sphere, of adjacent motion blocks A _{0 A} vector mv may be used to derive a respective motion vector mv of the current block _{a '.} Similarly, according to the rotation of the sphere, the motion vector mv _{b of the} adjacent block B ₀ can be used to derive the corresponding motion vector mv _{b′ of the} current block. The motion vector mv _{a 'and} a motion vector mv _b' may be included in the candidate list or merge AMVP candidate list.

In the present invention, the motion vector derivation based on the translation of the viewpoint is disclosed. Figure 12 shows an example in which objects (ie trees) are projected onto the surface of a sphere at different camera positions. At camera position A, the tree is projected onto the sphere 1210 to form an image 1240 of the tree. At the leading position B, the tree is projected onto the sphere 1220 to form an image 1250 of the tree. The image 1241 of the tree corresponding to camera position A is also shown on the sphere 1220 for comparison. At yet another preposition C, the tree is projected onto the sphere 1230 to form an image 1260 of the tree. The image 1242 of the tree corresponding to camera position A and the image 1251 of the tree corresponding to camera position B are also shown on the sphere 1220 for comparison. In Figure 12, for the video captured by a camera moving along a straight line, the direction of movement of the camera in 3D space (as indicated by arrows 1212, 1222, and 1232 at three different camera positions in Figure 12) can be It is represented by latitude and longitude coordinates (θ, φ), where (θ, φ) corresponds to the intersection of the motion vector and the 3D sphere. The point (θ, φ) is projected onto the 2D target projection plane, and this point becomes the leading point.

Figure 13 is an example of an ERP frame covered with a moving flow model, where if the camera leading point is known, the flow of the background (ie, static objects) can be determined. These flows are shown by arrows. The moving stream corresponds to the moving direction of the video content based on the camera moved in one direction. The movement of the camera causes the relative movement of the static background object. The movement direction of the background object on the 2D frame captured by the camera can be expressed as a moving stream. FIG. 14 shows an exemplary flow of MV derivation based on viewpoint translation. In Figure 14, an object 1410 is projected onto the sphere 1420 to form an image 1422 on the surface of the sphere corresponding to the camera position 1424. At camera position 1434, object 1410 is projected onto sphere 1430 to form image 1432 on the surface of the sphere. On the surface of the sphere 1430, the corresponding position of the object is shown at position 1423. The process of deriving the motion vector based on the viewpoint translation is as follows: · Find the camera's previous point 1440; · Calculate the moving stream 1450 in the 2D frame (that is, the tangent direction at each primitive); · By using adjacent blocks The MV determines the motion vector of the current block. a. As indicated by arrow 1460, the MV of the adjacent block can be used to determine the camera's translation. b. As indicated by arrow 1470, the camera's translation can be used to determine the MV of the current block.

MV derivation based on viewpoint translation can be applied to different projection methods. The moving stream in the 2D frame can be mapped to the 3D sphere. As shown in Figure 15, it is a moving flow on the 3D sphere 1510, where the leading point and two different moving flow lines (ie, 1512 and 1514) are shown. The moving flow on the 3D sphere related to the ERP 1520 is shown in Figure 15, where the moving flow is shown for the ERP frame 1526. The moving flow on the 3D sphere associated with the CMP 1530 is shown in Figure 15, where the moving flow is shown for the six faces of the CMP frame 1536 in the 2x3 layout format. The moving flow on the 3D sphere associated with OHP 1540 is shown in Figure 15, where the moving flow is shown for the eight faces of OHP frame 1546. The moving flow on the 3D sphere associated with the ISP 1550 is shown in Figure 15, where the moving flow is shown for the twenty faces of the ISP frame 1556. The moving flow on the 3D sphere related to SSP 1560 is shown in Figure 15, where the moving flow is shown for the segmentation plane of SSP frame 1566.

As mentioned earlier, there are different mappings, such as ERP, CMP, SSP, OHP, and ISP, to project from a 3D sphere to a 2D frame. When an object in a 3D sphere moves, the corresponding object in a 2D frame may be deformed. Deformation is based on the type of projection. The mapping positions of the length, angle and area in the 2D frame can no longer be consistent. The MV scaling technique on the 3D sphere is disclosed to perform the deformation by scaling the MV on the 3D sphere to minimize the influence of the projection type on the deformation. The MV scaling technique on 2D frames is also disclosed, which can be directly applied to 2D frames by combining projection, scaling MV, and back projection into a single function.

Fig. 16 shows an example of deformation related to the motion in the ERP frame. In Figure 16, three motion vectors along three latitude lines (ie, 1612, 1614, and 1616) are shown on the surface of the 3D sphere 1610, where the three motion vectors have approximately the same length. If the surface of the 3D sphere is expanded into a 2D plane 1620, the three motion vectors (1622, 1624, and 1626) maintain their characteristics of equal length. For ERP frames, the expanded image needs to be stretched, where more stretching is needed at higher latitudes. Therefore, as shown in Figure 16, the three motion vectors (1632, 1634, and 1636) have different lengths. If the neighboring block with motion vector 1634 is used for the current block (ie, the motion vector at position 1632), the motion vector 1634 must be scaled appropriately before being used for codec, such as merge index or AMVP index. The example in Figure 16 shows the need for MV scaling.

The following discloses an exemplary process of MV scaling technology in a 3D sphere. Suppose the mv _a starting point for _a, mv a the end of a '. We can predict the motion vector mv _b at the point b by the following process:

As shown in Figure 17, a, a'and b are mapped from the 2D frame 1710 to the 3D sphere 1720. i. Let the mapping function be ( ) = Pprojection type(x, y), which maps the primitives located at (x, y) in the 2D frame to ( ). 2. Calculate Δ And Δ i. (Δ , Δ ) = . 3. Scale function () and scale () Applied to Δ And Δ . i. Δ = scale (Δ , Δ ); ii. Δ '= scale (Δ , Δ ). 4. Calculate according to the following formula : I. = + (Δ , Δ ') 5. Will The 3D sphere 1720 is mapped to the 2D frame 1730 to generate b'. i. Let the inverse function be (x, y) = IP _{projection type} ( ), which will be on the 3D sphere ( ) Is mapped to a 2D frame. 6. According to mv _b = b ' -b , MV mv _b can be determined.

和Δ

之縮放函數分別為scale

(Δ

, Δ

)和scale

(Δ

, Δ

)。如下所示，為縮放函數之一些示例。In the above process, Δ

And Δ

The scaling functions are scale

(Δ

, Δ

) And scale

(Δ

, Δ

). Some examples of scaling functions are shown below.

(Δ

, Δ

) = Δ

; scale

(Δ

, Δ

) = Δ

因此，Δ

= Δ

; Δ

’ = Δ

Example 1: scale

(Δ

, Δ

) = Δ

; scale

(Δ

, Δ

) = Δ

Therefore, Δ

= Δ

; Δ

'= Δ

(Δ

, Δ

) =

; scale

(Δ

, Δ

) = Δ

Δ

=

Δ

’ = Δ

Example 2: scale

(Δ

, Δ

) =

; scale

(Δ

, Δ

) = Δ

Δ

=

Δ

'= Δ

= scale

(Δ

, Δ

,

1,

,

2,

) Δ

’ = scale

(Δ

, Δ

,

1,

,

2,

)Example 3: General form of scaling function-based on projection Δ

= scale

(Δ

, Δ

,

1,

,

2,

) Δ

'= scale

(Δ

, Δ

,

1,

,

2,

)

= scale

(Δ

, Δ

,

1,

,

2,

, 投影類型) Δ

’ = scale

(Δ

, Δ

,

1,

,

2,, 投影類型)Example 4: General form of scaling function-based on projection Δ

= scale

(Δ

, Δ

,

1,

,

2,

, Projection type) Δ

'= scale

(Δ

, Δ

,

1,

,

2, , Projection type)

By combining the projection function, the inverse scaling function and the projection function into a single function _{f (a, b, mv a} , projection type), MV 3D domain of scaling may also be performed in a 2D domain. Therefore, the mapping between 3D and 2D can be skipped. mv _b = (x _mvb , y _mvb ) = f(a, b, mv _a , projection type) x _mvb = f _x (a, b, mv _a , projection type) y _mvb = f _y (a, b, mv _a , projection type)

In the above equation, f _x (a, b, mv _a , projection type) is a single function that generates x _mvb , and f _y (a, b, mv _a , projection type) is a single function that generates y _mvb . Figure 18 shows an exemplary flow of MV scaling in 2D frames. Two blocks (ie A and B) in the 2D frame 1810 are shown. The motion vector at position a in block A is marked. In Figure 18, the end position of position a related to mv _a is marked as a' . The motion vector at position b in block A is marked. In Figure 18, the end position of position b related to mv _b is marked as b' . The steps of the projection function (ie, front projection 1830), the scaling function in the 3D sphere 1820, and the back projection function (ie, back projection 1840) can be combined into a single function according to the above equation.

As shown in Figure 19, it is an example of MV scaling in an ERP frame. In FIG. 19, the surface of the 3D sphere 1910 is expanded into a 2D plane 1920. For the ERP frame 1930, the expanded image needs to be stretched so that all latitude lines have the same length. Based on the characteristics of ERP, when it is closer to the North Pole or the South Pole, the length of the level line is expanded more. In the ERP frame, the neighboring block 1932 has a motion vector mv ₁ . The motion vector mv ₂ needs to be derived from mv ₁ for the current block 1934. The derived motion vector can be used to encode and decode the current block. For example, the derivation block may be used as a merge candidate or AMVP candidate. Since the neighboring blocks are located at different positions on the 3D sphere, the motion vectors related to the neighboring blocks need to be scaled before they are used to encode and decode the current block. Specifically, since the neighboring block is located at a higher latitude, the motion vector of the neighboring block is stretched more in the x direction than the motion vector of the current block. Therefore, the x component of mv ₁ needs to be scaled down before it is used in the current block. The MV scaling process 1940 is shown in FIG. 19, in which the motion vector at the position a of the adjacent block is scaled to mv ₂ and used to encode and decode the current block. In one embodiment of the present invention, a scaling function for maintaining the movement distance is disclosed, where mv ₂ is a function of mv ₁ , φ1, φ1, θ2, φ2: mv ₂ = f(mv ₁ , θ1, φ1, θ2, φ2 , ERP) mv _x2 = mv _x1 * ( ) mv _y2 = mv _y1

In the above equation, (θ1, j1) corresponds to the longitude and latitude of position a in the adjacent block, and (θ2, j2) corresponds to the longitude and latitude of position b in the current block. Position a and position b may correspond to the centers of the respective blocks, or other positions of the respective blocks. The y component of the derived motion vector is the same as the y component of mv ₁ . In other words, the scaling function of the y component is an identity function.

MV scaling of other projections has also been made public. In FIG. 20, MV scaling of ERP is disclosed, in which the current block 2012 and the neighboring block 2014 in the ERP image 2010 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the present invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , ERP), using the scaling function, the motion vector mv ₁ from the adjacent block needs to be scaled, where (x ₁ , y ₁ ) corresponds to the coordinates (x, y) of position a in the adjacent block, (x ₂ , y ₂ ) corresponds to the coordinates (x, y) of position b in the current block.

In FIG. 21, the MV scaling of CMP is disclosed, in which the current block 2112 and the adjacent block 2114 in the CMP image 2110 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the present invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , CMP), using the scaling function, the motion vector mv ₁ from the adjacent block needs to be scaled.

In FIG. 22, the MV scaling of SSP is disclosed, in which the current block 2212 and the neighboring block 2214 in the SSP image 2210 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , SSP), using the scaling function, the motion vector mv ₁ from the neighboring block needs to be scaled.

In FIG. 23, the MV scaling of OHP is disclosed, in which the current block 2312 and the adjacent block 2314 in the OHP image 2310 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the present invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , OHP), using the scaling function, the motion vector mv ₁ from the adjacent block needs to be scaled.

In FIG. 24, the MV scaling of the ISP is disclosed, in which the current block 2412 and the adjacent block 2414 in the ISP image 2410 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the present invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , ISP), using the scaling function, the motion vector mv ₁ from the adjacent block needs to be scaled.

In FIG. 25, the MV scaling of EAP is disclosed, in which the current block 2512 and the neighboring block 2514 in the EAP image 2510 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the present invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , EAP), using the scaling function, the motion vector mv ₁ from the adjacent block needs to be scaled.

In FIG. 26, the MV scaling of ACP is disclosed, in which the current block 2612 and the adjacent block 2614 in the ACP image 2610 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the present invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , ACP), using the scaling function, the motion vector mv ₁ from the adjacent block needs to be scaled.

In FIG. 27, the MV scaling of RSP is disclosed, in which the current block 2712 and the adjacent block 2714 in the RSP image 2710 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , RSP), using the scaling function, the motion vector mv ₁ from the neighboring block needs to be scaled.

In addition to the above projections, cylinder projections are also used to project 3D spheres into 2D frames. As shown in FIG. 28, conceptually, by wrapping a cylinder 2820 around a sphere 2830, and projecting light onto the cylinder through the sphere, a cylinder projection is created. Cylindrical projections represent the meridian as a straight, evenly spaced vertical line, and the parallel as a straight horizontal line. As it is on the ball, the meridian and parallel lines cross at right angles. According to the position of the light source, different cylindrical projections (Cylindrical Projection, CLP) are generated. In FIG. 28, the MV scaling of the CLP is disclosed, in which the current block 2812 and the adjacent block 2814 in the CLP image 2810 are shown. The neighboring block has a motion vector mv ₁ . The mv ₁ from the neighboring block is used to derive the motion vector mv _{2 of the} current block. According to the invention, according to mv ₂ = f(mv ₁ , x ₁ , y ₁ , x ₂ , y ₂ , CLP), using the scaling function, the motion vector mv ₁ from the neighboring block needs to be scaled.

FIG. 29 shows an exemplary flowchart of a system for applying sphere rotation to adjust motion vectors for processing 360-degree virtual reality images according to an embodiment of the present invention. The steps shown in the flow may be implemented as program code executable on one or more processors (eg, one or more CPUs) at the encoder side. The steps shown in the flowchart can also be implemented based on hardware, for example, one or more electronic devices or processors for performing the steps in the flowchart. According to this method, in step 2910, the input data of the current block in the 2D frame is received, wherein the 2D frame is projected from the 3D sphere. The input data may correspond to the primitive data in the 2D frame to be encoded. In step 2920, a first motion vector related to the adjacent block in the 2D frame is determined, where the first motion vector points from the first start position in the adjacent block to the first end position in the 2D frame. In step 2930, according to the target projection, the first motion vector is projected onto the 3D sphere. In step 2940, the first motion vector in the 3D sphere is rotated around the rotation axis along the rotation circle on the surface of the 3D sphere to generate the second motion vector in the 3D sphere. In step 2950, according to the inverse target projection, the second motion vector in the 3D sphere is mapped back to the 2D frame. In step 2960, the second motion vector is used to encode or decode the current block in the 2D frame.

FIG. 30 shows an exemplary flowchart of a system for deriving a motion vector from translation of a viewpoint according to an embodiment of the present invention for processing a 360-degree virtual reality image. According to another method, in step 3010, two 2D frames are received, where the two frames are projected from a 3D sphere corresponding to two different viewpoints using target projection, and the current block and adjacent blocks are located in these two 2D frame. In step 3020, based on the two 2D frames, the camera front point is determined. In step 3030, the mobile stream is determined in these two 2D frames. In step 3040, based on the first motion vector associated with the neighboring block, the translation of the camera is determined. In step 3050, based on the translation of the camera, a second motion vector related to the current block is derived. In step 3060, the second motion vector is used to encode or decode the current block in the 2D frame.

FIG. 31 shows an exemplary flowchart of a system for applying zoom to adjust motion vectors for processing 360-degree virtual reality images according to an embodiment of the present invention. According to the method, in step 3110, the input data of the current block in the 2D frame is received, wherein the 2D frame is projected from the 3D sphere according to the target projection. In step 3120, a first motion vector related to the adjacent block in the 2D frame is determined, where the first motion vector points from the first start position in the adjacent block to the first end position in the 2D frame. In step 3130, the first motion vector is scaled to generate a second motion vector. In step 3140, the second motion vector is used to encode or decode the current block in the 2D frame.

The flowchart shown in the present invention is used to show an example of video according to the present invention. Without departing from the spirit of the present invention, those skilled in the art can modify each step, reorganize these steps, separate one step, or combine these steps to implement the present invention.

The above description is presented to enable those of ordinary skill in the art to implement the present invention in the context of specific applications and their needs. It will be obvious to those skilled in the art that various modifications of the described embodiments will be apparent, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not limited to the specific embodiments shown and described, but is to be accorded the maximum scope consistent with the principles and novel features disclosed herein. In the foregoing detailed description, various specific details are described in order to thoroughly understand the present invention. Nevertheless, it will be understood by those skilled in the art that the present invention can be practiced.

The embodiments of the present invention as described above can be implemented in various hardware, software codes, or a combination of both. For example, the embodiments of the present invention may be a circuit integrated in a video compression chip, or a program code integrated in a video compression software to perform the processing described herein. An embodiment of the present invention may also be a program code executed on a digital signal processor (Digital Signal Processor, DSP) to perform the processing described herein. The invention may also include several functions performed by a computer processor, digital signal processor, microprocessor, or field programmable gate array (FPGA). According to the present invention, these processors may be configured to perform specific tasks by executing machine-readable software codes or firmware codes that define specific methods implemented by the present invention. The software code or firmware code can be developed by different programming languages and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, software code formats and languages, and other forms of configuration codes for performing the tasks of the present invention will not depart from the spirit and scope of the present invention.

The present invention is implemented in other specific forms without departing from its spirit or essential characteristics. The examples described are merely illustrative in all respects, not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than the foregoing description. The meaning of the claims and all changes within the same scope should be included in the scope.

110, 610, 700, 810, 920, 970, 1020, 1210, 1220, 1230, 1420, 1430, 1510, 1610, 1720, 1910, 2830 ‧ ‧ ‧ sphere 120, 340, 430, 630 ‧ ‧ ‧ rectangular image 210‧‧‧Cube 220a, 220b, 620, 622‧‧‧ part 230a, 230b ‧‧‧ with 310‧‧‧ octahedron 320, 420‧‧‧ face 330‧‧‧ intermediate format 410‧‧‧ twenty faces Volume 500 ‧‧‧ Spherical image 510 ‧‧‧ Arctic image 520 ‧ ‧ ‧ Antarctic image 530 ‧ ‧ ‧ Equatorial image 502 ‧ ‧ ‧ Latitude 45°N504‧‧‧ Latitude 45° S506 ‧‧‧Equatorial 701 , 702, 704, 705, 1112, 1120, 1932, 1934, 2012, 2014, 2112, 2114, 2212, 2214, 2312, 2314, 2412, 2414, 2512, 2514, 2612, 2614, 2712, 2714, 2814, 2812 ‧‧‧Blocks 703, 1526, 1930, 2010 ‧‧‧ ERP frame 711, 714 ‧ ‧ ‧ region 710 ‧ ‧ ‧ surface 713 ‧ ‧ ‧ path 712, 715, 1612, 1614, 1616, 1622, 1624, 1626, 1632 , 1634, 1636‧‧‧Motion vectors 720, 740, 760‧‧‧ 2D domain 721-725‧‧‧‧ 2D domain 720 part 711-715 corresponds to 733, 753‧‧‧ locus 733-735‧‧‧ locus 733 corresponds to the part 713-725 743-745 ‧‧‧ 2D domain 740 corresponds to the part 733-735 part 750 ‧‧‧ the surface of the sphere 753-755 ‧‧‧ track 753 corresponds to the part 713-725 763-765 ‧‧‧ 2D domain 760 corresponding to the parts 753-755 820, 910, 930, 960, 980 ‧ ‧ ‧ circle 830, 840, 912 ‧ ‧ ‧ point 800, 900, 950 ‧ ‧ sphere Rotation 850, 990‧‧‧ Rotation axis 1010, 1030‧‧‧ Sketch 1100‧‧‧ Layout 1240, 1250, 1241, 1260, 1242, 1251, 1422, 1432‧‧‧ Image 1212, 1222, 1232, 1460, 1470 ‧‧‧Arrow 1410‧‧‧object 1424, 1434, 1423 ‧‧‧ position 1440 ‧‧‧ leading point 1450 ‧‧‧ mobile flow 1512, 1514‧‧‧ mobile flow line 1520‧‧‧ERP1530‧‧‧CMP1536, 2110‧‧‧CMP frame 1540‧‧‧OHP1546, 2310‧‧‧OHP frame 1550‧‧‧ISP1 556, 2410‧‧‧ISP frame 1560‧‧‧SSP1566, 2210‧‧‧SSP frame 1620, 1920‧‧‧2D plane 1710, 1730, 1810‧‧‧2D frame 1830‧‧‧front projection 1840‧‧‧ reverse Projection 1940‧‧‧MV zoom process 2510‧‧‧EAP image 2610‧‧‧ACP image 2710‧‧‧RSP image 2820‧‧‧Cylinder 2810‧‧‧CLP image 2910-2960, 3010-3060‧ ‧‧step

Figure 1 is an example of projecting a sphere onto a rectangular image based on an isometric projection, where each longitude line is mapped to the vertical line of the ERP image. Figure 2 is a cube with 6 faces, where a 360-degree VR image can be projected onto the 6 faces of the cube according to the projection of the cube. Figure 3 is an example of octahedron projection, in which a sphere is projected onto an octahedron of 8 faces. Figure 4 is an example of icosahedral projection, in which a sphere is projected onto an icosahedron of 20 faces. Figure 5 is an example of segmented sphere projection (SSP), in which the sphere image is mapped to the North Pole image, the South Pole image, and the equatorial segment image. Figure 6 is an example of a rotated sphere projection (RSP), in which the sphere is divided into the middle 270°x90° area and the remaining part. The two parts of the RSP can also be stretched on the top and bottom sides to produce deformed parts with elliptical borders on the top and bottom ends.

Figures 7A and 7B are examples of deformation of 2D images due to the rotation of the sphere.

Figure 8 is an example of the rotation of the sphere from point a to point b on the larger circle of the sphere, where the larger circle corresponds to the largest circle on the surface of the sphere.

Figure 9 is an example of the rotation of the sphere from point a to point b on the smaller circle of the sphere, where the larger circle corresponds to a circle smaller than the largest circle on the surface of the sphere.

Figure 10 uses the sphere rotation model to derive the vector of the motion vector used for the 2D projection image.

Figure 11 is an example of using motion vector (MV) derived based on the rotation of the sphere as a candidate for merging or advanced motion vector prediction (AMVP).

Figure 12 is an example of objects (ie trees) being projected onto the surface of a sphere at different camera positions.

Figure 13 is an example of an ERP frame covered by a model of a moving stream, where if the camera forward point is known, the flow of the background (ie, static objects) can be determined.

Figure 14 is an exemplary flow of MV derivation based on viewpoint translation.

Figure 15 is an exemplary MV derivation based on viewpoint translation for different projection methods.

Figure 16 is an example of the deformation related to the motion in the ERP frame.

Figure 17 is an exemplary flow of MV scaling technology used in a 3D sphere.

Figure 18 is an exemplary flow of the MV scaling technique in 2D frames. Figure 19 is an exemplary flow of the MV scaling technique in the ERP frame. Figure 20 is an exemplary flow of the MV scaling technique in the ERP frame, where the current block and neighboring blocks are shown in the ERP image. Figure 21 is an exemplary flow of the MV scaling technique in the CMP frame, where the current block and the neighboring blocks are shown in the CMP image. Figure 22 is an exemplary flow of the MV scaling technique in the SSP frame, where the current block and neighboring blocks are shown in the SSP image. Figure 23 is an exemplary flow of the MV scaling technique in an Octahedron Projection (OHP) frame, where the current block and neighboring blocks are shown in the OHP image. Figure 24 is an exemplary flow of the MV scaling technique in the icosahedron projection (Icosahedron Projection, ISP) frame, where the current block and the neighboring blocks are shown in the ISP image. Figure 25 is an exemplary flow of the MV scaling technique in the Equal-Area Projection (EAP) frame, where the current block and neighboring blocks are shown in the EAP image. Figure 26 is an exemplary flow of the MV scaling technique in the Adjusted Cubemap Projection (ACP) frame, where the current block and neighboring blocks are shown in the ACP image. Figure 27 is an exemplary flow of the MV scaling technique in a Rotated Sphere Projection (Rotated Sphere Projection, RSP) frame, where the current block and neighboring blocks are shown in the RSP image. Figure 28 is an exemplary flow of the MV scaling technique in a Cylindrical Projection (CLP) frame, where the current block and neighboring blocks are shown in the CLP image. FIG. 29 is an exemplary flowchart of a system for processing 360-degree VR images by applying rotation of a sphere to adjust motion vectors according to an embodiment of the present invention. FIG. 30 is an exemplary flowchart of a system for deriving a motion vector from a translation of a viewpoint to process a 360-degree VR image according to an embodiment of the present invention. FIG. 31 is an exemplary flowchart of a system for processing 360-degree VR images by applying scaling to adjust motion vectors according to an embodiment of the present invention.

703‧‧‧ERP frame

701, 702, 704, 705‧‧‧ block

700‧‧‧Sphere

Claims (20)

A method for processing a 360-degree virtual reality image, characterized in that the method includes: receiving input data of a current block in a two-dimensional frame, wherein the two-dimensional frame is projected from a three-dimensional sphere; The first motion vector related to the adjacent block in the two-dimensional frame, wherein the first motion vector points from the first starting position in the adjacent block to the first ending position in the two-dimensional frame; according to the target projection , Project the first motion vector onto the three-dimensional sphere; rotate the first motion vector in the three-dimensional sphere around a rotation axis along a rotating circle on the surface of the three-dimensional sphere to generate A second motion vector in the three-dimensional sphere; mapping the second motion vector in the three-dimensional sphere back to the two-dimensional frame according to inverse target projection; and encoding or decoding using the second motion vector The current block in the two-dimensional frame. The method for processing a 360-degree virtual reality image as described in item 1 of the scope of the patent application law, wherein the second motion vector is included as a candidate in the merge candidate list or the advanced motion vector prediction candidate list for use in Encode or decode the current block. The method for processing a 360-degree virtual reality image as described in item 1 of the scope of the patent application law, wherein the rotating circle corresponds to the largest circle on the surface of the three-dimensional sphere. Process 360-degree virtual reality maps as described in Item 1 of the Patent Application Scope Like the method, wherein the rotating circle is smaller than the largest circle on the surface of the three-dimensional sphere. The method for processing a 360-degree virtual reality image as described in item 1 of the scope of the patent application law, wherein the target projection corresponds to isometric projection, cube projection, adjusted cube projection, isoregion projection, octahedral projection , Icosahedron projection, segmented sphere projection, rotating sphere projection or cylinder projection. The method for processing a 360-degree virtual reality image as described in item 1 of the scope of the patent application law, wherein projecting the first motion vector onto the three-dimensional sphere includes: according to the target projection, projecting the two The first start position, first end position, and second start position in the dimensional frame are projected onto the three-dimensional sphere, where the second start position is located at a corresponding position in the current block, so The corresponding position corresponds to the first starting position in the adjacent block. The method for processing a 360-degree virtual reality image as described in item 6 of the scope of the patent application law, wherein rotating the first motion vector in the three-dimensional sphere along the rotation circle includes: determining a target rotation , For rotating along the rotation circle on the surface of the three-dimensional sphere from the first starting position to the second starting position in the three-dimensional sphere around the axis of rotation; and using the target rotation, The first end position is rotated to a second end position on the three-dimensional sphere. The method for processing a 360-degree virtual reality image as described in item 1 of the scope of the patent application law, wherein mapping the second motion vector in the three-dimensional sphere to the two-dimensional frame includes: Mapping the second end position on the three-dimensional sphere back to the two-dimensional frame according to the inverse target projection; and according to the second starting position and the second in the two-dimensional frame The ending position determines the second motion vector in the two-dimensional frame. An apparatus for processing 360-degree virtual reality images, characterized in that the apparatus includes one or more electronic devices or processors for: receiving input data of a current block in a two-dimensional frame, wherein the two-dimensional The frame is projected from a three-dimensional sphere; the first motion vector related to the adjacent block in the two-dimensional frame is determined, wherein the first motion vector points from the first starting position in the adjacent block to the two The first end position in the dimension frame; project the first motion vector onto the three-dimensional sphere according to the target projection; project the first motion vector in the three-dimensional sphere along the surface of the three-dimensional sphere The rotating circle rotates around the rotation axis to generate the second motion vector in the three-dimensional sphere; according to the inverse target projection, the second motion vector in the three-dimensional sphere is mapped back to the two-dimensional frame; And using the second motion vector to encode or decode the current block in the two-dimensional frame. A method for processing a 360-degree virtual reality image, wherein the method includes: receiving two two-dimensional frames, wherein the two frames are projected from a three-dimensional sphere corresponding to two different viewpoints using target projection, and The current block and the adjacent block are located in the two two-dimensional frames; Based on the two two-dimensional frames, determine the previous point of the camera; determine multiple moving streams in the two two-dimensional frames; determine the translation of the camera based on the first motion vector related to the adjacent block; Based on the translation of the camera, derive a second motion vector related to the current block; and use the second motion vector to encode or decode the current block in the two-dimensional frame. The method for processing a 360-degree virtual reality image as described in item 10 of the scope of the patent application law, wherein the multiple mobile streams in the two two-dimensional frames are derived from the two two-dimensional frames The tangent direction of each primitive is calculated. The method for processing a 360-degree virtual reality image as described in item 10 of the scope of the patent application law, wherein the second motion vector is included as a candidate in the merge candidate list or the advanced motion vector prediction candidate list for use in Encode or decode the current block. The method for processing a 360-degree virtual reality image as described in item 10 of the scope of the patent application law, wherein the target projection corresponds to isometric projection, cube projection, adjusted cube projection, isoregion projection, octahedral projection , Icosahedron projection, segmented sphere projection, rotating sphere projection or cylinder projection. A method for processing a 360-degree virtual reality image, characterized in that the method includes: receiving input data of a current block in a two-dimensional frame, wherein the two-dimensional frame is projected from a three-dimensional sphere according to a target projection; A first motion vector related to an adjacent block in the two-dimensional frame, where the first motion vector points from a first starting position in the adjacent block to a first ending position in the two-dimensional frame; Scaling the first motion vector to generate a second motion vector; and using the second motion vector to encode or decode the current block in the two-dimensional frame; wherein, the scaling the first motion The step of generating a second motion vector by using a vector includes: projecting the first start position, the first end position, and the second start position in the two-dimensional frame to the three-dimensional sphere according to the target projection Above, wherein the second starting position is located at a corresponding position in the current block, and the corresponding position corresponds to the first starting position in the neighboring block. The method for processing a 360-degree virtual reality image as described in item 14 of the scope of the patent application law, wherein the step of scaling the first motion vector to generate a second motion vector further includes: scaling the first motion The longitude component of the vector to generate the scaled longitude component of the first motion vector; scale the latitude component of the first motion vector to generate the scaled latitude component of the first motion vector; and based on the first The scaled longitude component of the motion vector and the scaled latitude component of the first motion vector determine a second end position corresponding to the second start position. The method for processing a 360-degree virtual reality image as described in item 15 of the scope of the patent application law, wherein the step of scaling the first motion vector to generate a second motion vector further includes: according to the inverse target projection, the Mapping the second end position on the three-dimensional sphere back to the two-dimensional frame; and based on the second start position and the second end position in the two-dimensional frame, determining the position of the two-dimensional frame The second motion vector. The method for processing a 360-degree virtual reality image as described in item 14 of the scope of the patent application law, wherein the step of scaling the first motion vector to generate a second motion vector includes: applying a first combination function, Generating an x component of the second motion vector and applying a second combination function to generate a y component of the second motion vector; wherein the first combination function and the second combination function are based on the A starting position, a second starting position in the current block corresponding to the first starting position, the first motion vector and the target projection; and wherein the first combined function and the second combined function Combining the target projection, the scaling and the inverse target projection, the target projection is used to project the first data in the two-dimensional frame to the second data in the three-dimensional sphere, the scaling is to The selected motion vector in the three-dimensional sphere is scaled to the scaled motion vector in the three-dimensional sphere, and the inverse target projection is used to project the scaled motion vector into the two-dimensional frame. The method for processing a 360-degree virtual reality image as described in Item 17 of the scope of the patent application law, wherein the target projection corresponds to an isometric projection; the first combination function corresponds to (cosφ1/cosφ2), where cosφ1 Corresponding to the first latitude related to the first starting position, cosφ2 corresponds to the second latitude related to the second starting position; and the second combined function corresponds to the identity function. The method for processing a 360-degree virtual reality image as described in item 14 of the scope of the patent application law, wherein the second motion vector is included as a candidate in the merge candidate list or the advanced motion vector prediction candidate list for Encode or decode the current block. The method for processing 360-degree virtual reality images as described in item 14 of the scope of the patent application law, wherein the target projection corresponds to isometric projection, cubic projection, adjusted cubic projection, isoregion projection, octahedral projection , Icosahedron projection, segmented sphere projection, rotating sphere projection or cylinder projection.

Images