CN116912129A

CN116912129A - Method for processing a projection-based frame comprising at least one projection surface encapsulated in a 360 degree virtual reality projection layout

Info

Publication number: CN116912129A
Application number: CN202310978612.7A
Authority: CN
Inventors: 林建良; 王鹏; 刘琳; 李亚璇; 林鸿志; 张胜凯
Original assignee: MediaTek Inc
Current assignee: MediaTek Inc
Priority date: 2017-09-27
Filing date: 2018-09-25
Publication date: 2023-10-20
Also published as: CN109906468B; DE112018002436T5; CN109906468A; TW201921035A; TWI702567B; WO2019062714A1

Abstract

A video processing method, comprising: acquiring a plurality of projection surfaces from the omnidirectional content of the sphere, wherein the omnidirectional content of the sphere is mapped onto the plurality of projection surfaces through cube projection, and the plurality of projection surfaces comprise a first projection surface; resampling at least a portion of the first projection surface using a non-uniform mapping by a resampling circuit to obtain a first resampled projection surface; generating a projection-based frame from a projection layout of the cube projection, wherein the projection-based frame includes a first resampled projection surface encapsulated in the projection layout; and encodes the projection-based frame to generate a portion of a bitstream.

Description

Method for processing a projection-based frame comprising at least one projection surface encapsulated in a 360 degree virtual reality projection layout

Cross Reference to Related Applications

The present application is a continuation of part of U.S. patent application Ser. No. 15/917,844, filed on 12 months of 2008, and claims priority from U.S. provisional application Ser. No. 62/563,787, filed on 27 months of 2017, U.S. provisional application Ser. No. 62/583,078, filed on 11 months of 2017, and U.S. provisional application Ser. No. 62/583,573, filed on 11 months of 2017, wherein U.S. patent application Ser. No. 15/917,844 claims priority from U.S. provisional application Ser. No. 62/470,425, filed on 13 months of 2017.

The entire contents of the related applications, including U.S. patent application Ser. No. 15/917,844, U.S. provisional application Ser. No. 62/563,787, U.S. provisional application Ser. No. 62/583,078, U.S. provisional application Ser. No. 62/583,573, and U.S. provisional application Ser. No. 62/470,425, are incorporated herein by reference.

Technical Field

The present invention relates to processing omni-directional image/video content. More particularly, the present invention relates to a method for processing a projection-based frame comprising at least one projection surface encapsulated in a 360 degree virtual reality (360-degree virtual reality,360 VR) projection layout.

Background

Virtual Reality (VR) with head-mounted displays (HMDs) are associated with various applications. The ability to display wide-field content to a user may be used to provide an immersive visual experience. The real world environment must be captured in all directions in order to produce an omni-directional image/video content corresponding to the sphere. With advances in camera equipment and HMDs, the delivery of VR content may quickly become a bottleneck due to the high bit rate required to represent such 360 degree image/video content. When the resolution of omni-directional video is 4K or higher, data compression/encoding is critical to reduce the bit rate.

Typically, the omnidirectional image/video content corresponding to a sphere is converted into a sequence of images, each image being a projection-based frame having 360 degrees of image/video content represented by one or more projection planes arranged in a 360 degree virtual reality (360 VR) projection layout, and then the sequence of projection-based frames is encoded into a bitstream for transmission. Projection-based frames may have image content discontinuities at layout boundaries and/or face boundaries. The image quality around the layout boundary and/or the plane boundary may be poor after compression. Furthermore, projection layout conversion by decoded projection-based frames may introduce artifacts, resulting in degradation of the image quality of the converted projection-based frames.

Disclosure of Invention

It is an object of the present invention to provide a method for processing a projection-based frame comprising at least one projection surface encapsulated in a 360 degree virtual reality (360 VR) projection layout.

According to a first aspect of the present invention, an exemplary video processing method is disclosed. The exemplary video processing method includes: obtaining a plurality of projection surfaces from the omnidirectional content of the sphere, wherein the omnidirectional content of the sphere is mapped onto the projection surfaces by cube projection, and the plurality of projection surfaces includes a first projection surface; resampling at least a portion of the first projection surface by a resampling circuit, the resampling circuit obtaining a first resampled projection surface, wherein the first projection surface has a first source region and a second source region, the first resampled projection surface has a first resampling region and a second resampling region, the first resampling region is resampled from the first source region at a first sampling density, the second resampling region is resampled from the second source region at a second resampling density, the first sampling density is different from the second sampling density; generating a projection-based frame from a projection layout of the cube projection, wherein the projection-based frame includes a first resampled projection surface encapsulated in the projection layout; and encoding the projection-based frame to generate a portion of the bitstream.

According to a second aspect of the present invention, an exemplary video processing method is disclosed. The exemplary video processing method includes: obtaining a plurality of projection surfaces from the omnidirectional content of the sphere according to the cube projection; generating at least one fill area by a fill circuit; generating a projection-based frame by encapsulating the plurality of projection facets and the at least one fill region in a projection layout of a cube projection, wherein the plurality of projection facets encapsulated in the projection layout comprises a first projection facet; the at least one fill area encapsulated in the projection layout includes a first fill area; the first filling area is connected with at least the first projection surface and forms at least one part of one boundary of the projection layout; and encoding the projection-based frame to generate a portion of the bitstream.

According to a third aspect of the present invention, an exemplary video processing method is disclosed. The exemplary video processing method includes: a portion of the bitstream is received and the portion of the bitstream is decoded to generate a decoded projection-based frame, wherein the projection-based frame has at least one projection surface and at least one fill region encapsulated in a 360 degree virtual reality (360 VR) projection layout. The step of decoding a portion of the bitstream to generate a decoded projection-based frame comprises: reconstructing the first pixels included in the at least one fill region by mixing decoded pixel values obtained by the first pixels included in the at least one fill region and decoded pixel values obtained by the second pixels included in the at least one projection plane.

These and other objects of the present invention will no doubt become obvious to those of ordinary skill in the art after having read the preferred embodiments which are illustrated in the various drawing figures and drawings which are described in the following detailed description.

Drawings

Fig. 1 is a schematic diagram of a first 360 degree virtual reality (360 VR) system in accordance with an embodiment of the invention.

Fig. 2 is a schematic representation of a triangular projection surface in an octahedral projection-based format obtained from the projection of spheres onto an irrotational octahedron (octahedron).

Fig. 3 is a schematic diagram of a first compact octahedral projection layout (compact octahedron projection layout) according to an embodiment of the present invention.

Fig. 4 is a schematic illustration of a triangular projection surface in an octahedral projection-based format obtained from the projection of spheres onto rotated octahedrons.

FIG. 5 is a schematic diagram of a second compact octahedral projection layout according to an embodiment of the present invention.

FIG. 6 is a schematic diagram of a first compact octahedral projection layout with filling according to an embodiment of the present invention.

FIG. 7 is a schematic illustration of a second compact octahedral projection layout with filling according to an embodiment of the present invention.

Fig. 8 is a schematic diagram of interpolation performed by the fill circuit shown in fig. 1.

Fig. 9 is a schematic diagram of geometric filling performed by the fill circuit shown in fig. 1.

Fig. 10 is a schematic diagram of a compact cube (cube) projection layout with fill in, according to an embodiment of the invention.

FIG. 11 is a schematic illustration of a third compact octahedral projection layout with filling according to an embodiment of the present invention.

FIG. 12 is a schematic diagram of a first ERP/EAP layout with padding, according to an embodiment of the invention.

FIG. 13 is a schematic diagram of a second ERP/EAP layout with padding, according to an embodiment of the invention.

FIG. 14 is a schematic diagram of a third ERP/EAP layout with padding, according to an embodiment of the invention.

FIG. 15 is a schematic diagram of an octahedral projection layout with filling according to an embodiment of the present invention.

FIG. 16 is a schematic diagram of a cube projection layout with fill in accordance with an embodiment of the invention.

FIG. 17 is a schematic diagram of a fourth compact octahedral projection layout with filling according to an embodiment of the present invention.

FIG. 18 is a schematic diagram of a compact cube projection layout with fill in accordance with an embodiment of the invention.

Fig. 19 is a schematic diagram of a second 360VR system in accordance with an embodiment of the invention.

Fig. 20 is a schematic view of six square projection planes in a cube projection layout obtained from a sphere through a cube projection (cubemap projection, CMP).

FIG. 21 is a schematic diagram of resampling a square projection surface obtained by a cube projection by uniform mapping, in accordance with an embodiment of the invention.

FIG. 22 is a schematic diagram of a uniform mapping function curve according to an embodiment of the invention.

FIG. 23 is a schematic diagram of a first example of resampling a square projection surface obtained by cube projection by non-uniform mapping, in accordance with an embodiment of the invention.

FIG. 24 is a schematic diagram of a second example of resampling a square projection surface obtained by cube projection by non-uniform mapping, in accordance with an embodiment of the invention.

FIG. 25 is a schematic diagram of a first non-uniform mapping function curve according to an embodiment of the invention.

FIG. 26 is a schematic diagram of a second non-uniform mapping function curve according to an embodiment of the invention.

Fig. 27 is a schematic diagram of a third 360VR system in accordance with an embodiment of the invention.

FIG. 28 is a schematic diagram of internal boundary filling (edge packing) of a cube projection layout, according to an embodiment of the invention.

Fig. 29 is a schematic diagram of external boundary padding (boundary padding) and internal boundary padding for a cube projection layout, according to an embodiment of the invention.

FIG. 30 is a schematic diagram of external boundary filling and internal boundary filling for other cube projection layouts, according to an embodiment of the present invention.

FIG. 31 is a schematic illustration of a fill design for creating a fill area of one projection surface by copying a partial area in another projection surface in accordance with an embodiment of the present invention.

Fig. 32 is a schematic diagram of a fourth 360VR system in accordance with an embodiment of the invention.

Fig. 33 is a schematic diagram of a fifth 360VR system in accordance with an embodiment of the invention.

Fig. 34 is a schematic diagram of a decoder-side mixing operation according to an embodiment of the present invention.

Fig. 35 is a schematic diagram of a relationship between a weight value of a pixel and an index value of the pixel, which is involved in updating a pixel value of the pixel in a projection plane, according to an embodiment of the present invention.

Fig. 36 is a schematic diagram of another relationship between a weight value of a pixel and an index value of the pixel, which is involved in updating a pixel value of the pixel in the projection plane, according to an embodiment of the present invention.

Fig. 37 is a schematic diagram of a sixth 360VR system in accordance with an embodiment of the invention.

Fig. 38 is a schematic diagram of a seventh 360VR system in accordance with an embodiment of the invention.

Fig. 39 is a schematic diagram of updating the pixel values of pixels in the projection plane and the pixel values of the filled pixels in the filled region, according to an embodiment of the present invention, the relation between the weight values of the pixels concerned and the index values of the pixels.

Fig. 40 is a schematic diagram of another relationship between the weight value of the pixel concerned and the index value of the pixel, according to an embodiment of the present invention, of updating the pixel value of the pixel in the projection plane and the pixel value of the filling pixel in the filling area.

Detailed Description

Certain terms are used throughout the description and claims to refer to particular components. Those skilled in the art will appreciate that electronic device manufacturers may refer to a component by different names. The description and claims do not take the form of an element differentiated by name, but rather by functional differences. In the following description and in the claims, the terms "include" and "comprise" are used in an open-ended fashion, and thus should be interpreted to mean "include, but not limited to. In addition, the term "coupled" as used herein includes any direct or indirect electrical connection. Thus, if a first device couples to a second device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.

Fig. 1 is a schematic diagram of a first 360 degree virtual reality (360 VR) system in accordance with an embodiment of the invention. The 360VR system 100 includes two video processing apparatuses (e.g., a source electronic device 102 and a target electronic device 104). The source electronic device 102 includes a video capture device 112, a conversion circuit 114, and a video encoder 116. For example, video capture device 112 may be a set of cameras for providing omnidirectional image/video content (e.g., multiple images covering the entire surrounding environment) s_in corresponding to a sphere. The conversion circuit 114 is coupled between the video capture device 112 and the video encoder 116. The conversion circuit 114 generates a projection-based frame IMG having a 360 degree virtual reality projection layout l_vr from the omnidirectional image/video content s_in. For example, the projection-based frame IMG may be one frame included in the sequence of projection-based frames generated from the conversion circuit 114. The video encoder 116 is an encoding circuit for encoding/compressing projection-based frames IMG to generate a portion of the bitstream BS. Further, the video encoder 116 outputs the bit stream BS to the target electronic device 104 via the transmission means 103. For example, the projection-based sequence of frames may be encoded into the bit stream BS and the transmission means 103 may be a wired/wireless communication link or a storage medium.

The target electronic device 104 may be a Head Mounted Display (HMD) device. As shown in fig. 1, the target electronic device 104 includes a decoding circuit 122, a graphics rendering circuit 124, and a display screen 126. The decoding circuit 122 receives the bit stream BS from the transmission device 103 (e.g., a wired/wireless communication link or memory) and then performs a video decoder function for decoding a portion of the received bit stream BS to generate decoded frames IMG'. For example, the decoding circuit 122 generates a sequence of decoded frames by decoding the received bit stream BS, wherein the decoded frame IMG' is one frame included in the sequence of decoded frames. In this embodiment, the projection-based frame IMG encoded by encoder-side video encoder 116 has a 360VR projection format and projection layout. Thus, after decoding the bit stream BS by the decoder-side decoding circuit 122, the decoded frame IMG' is a decoded projection-based frame having the same 360VR projection format and the same projection layout. Graphics rendering circuitry 124 is coupled between decoding circuitry 122 and display screen 126. Graphics rendering circuit 124 renders and displays output image material on display screen 126 in accordance with decoded frame IMG'. For example, a viewport region associated with a portion of 360-degree image/video content carried by the decoded frame IMG' may be displayed on the display screen 126 via the graphics rendering circuit 124.

As described above, the conversion circuit 114 generates the projection-based frame IMG from the 360VR projection layout l_vr and the omnidirectional image/video content s_in. If the 360VR projection layout L _ VR is a compact projection layout without padding, the packaging of the projection surfaces may result in the existence of image content discontinuity boundaries between adjacent projection surfaces.

Fig. 2 is a schematic illustration of a triangular projection surface in an octahedral projection-based format obtained from a sphere to an irrotational octahedral projection. The omnidirectional image/video content of sphere 202 is mapped to eight triangular projection planes (labeled "1", "2", "3", "4", "5", "6", "7", and "8") of non-rotated octahedron 204. As shown in fig. 2, triangular projection planes "1" - "8" are arranged in an octahedral projection layout 206. The shape of each triangular projection plane "1" - "8" is an equilateral triangle. For a triangular projection plane "K" (k=1-8), the plane has three sides, denoted SK1, SK2 and SK3. Sphere 202 is comprised of a top hemisphere (e.g., a northern hemisphere) and a bottom hemisphere (e.g., a southern hemisphere). Since the triangular projection planes "1", "2", "3", and "4" are all from the top hemisphere, the triangular projection planes "5", "6", "7", and "8" are all from the bottom hemisphere, and the equator 208 of the sphere 202 is mapped along the sides S13, S23, S33, S43, S53, S63, S73, and S83 of the triangular projection planes "1" - "8", as indicated by the dashed lines in fig. 2, based on the octahedral projections of the non-rotated octahedrons 204.

The projection-based frame IMG to be encoded needs to be rectangular. If the octahedral projection layout 206 is used directly to create a projection-based frame IMG, the projection-based frame IMG cannot have a compact frame layout, and many empty regions (e.g., black, gray, or white filled regions) need to be filled in the projection-based frame IMG. Thus, there is a need for a compact octahedral projection layout that can avoid empty areas (e.g., areas filled in black, gray, or white).

Please refer to fig. 2 and fig. 3 in combination. Fig. 3 is a schematic diagram of a first compact octahedral projection layout according to an embodiment of the present invention. The equator 208 of the sphere 202 is mapped along the sides of the triangular projection plane "1" - "8", as shown by the dashed lines in fig. 3. The compact octahedral projection layout 310 is derived from the octahedral projection layout 206 by triangle projection plane rotation and triangle projection plane segmentation. As shown in the middle portion of fig. 3, the triangular projection plane "1" in the octahedral projection layout 206 is rotated 60 ° clockwise, the triangular projection plane "3" in the octahedral projection layout 206 is rotated 60 ° counterclockwise, the triangular projection plane "5" in the octahedral projection layout 206 is rotated 60 ° counterclockwise, and the triangular projection plane "7" in the octahedral projection layout 206 is rotated 60 ° clockwise. Therefore, the side S21 of the triangular projection plane "2" is connected to the side S12 of the triangular projection plane "1", and the side S22 of the triangular projection plane "2" is connected to the side S31 of the triangular projection plane "3". The side S62 of the triangular projection plane "6" is connected to the side S51 of the triangular projection plane "5", and the side S61 of the triangular projection plane "6" is connected to the side S72 of the triangular projection plane "7".

As shown in the middle part of fig. 3, there is an image content continuity boundary between the side S21 of the triangle projection plane "2" and the side S12 of the triangle projection plane "1" (i.e., the content is continuously represented in the triangle projection planes "1" and "2"), an image content continuity boundary between the side S22 of the triangle projection plane "2" and the side S31 of the triangle projection plane "3" (i.e., the content is continuously represented in the triangle projection planes "2" and "3"), an image content continuity boundary between the side S23 of the triangle projection plane "2" and the side S63 of the triangle projection plane "6" (i.e., the content is continuously represented in the triangle projection planes "2" and "6"), an image content continuity boundary between the side S62 of the triangle projection plane "6" and the side S51 of the triangle projection plane "5" (i.e., the content is continuously represented in the triangle projection planes "5" and "6"), and an image content continuity boundary between the side S61 of the triangle projection plane "6" and the side S72 of the triangle projection plane "7" (i.e., the content is continuously represented in the triangle projection plane "7").

In addition, the triangular projection plane "8" in the octahedral projection layout 206 is divided into two right triangle parts 302 and 304, and the triangular projection plane "4" in the octahedral projection layout 206 is divided into two right triangle parts 306 and 308. As shown in the bottom of fig. 3, the right triangle portion 304 of the triangular projection plane "8" and the right triangle portion 308 of the triangular projection plane "4" are connected to the triangular projection planes "7" and "3", respectively; the right triangle portion 302 of triangle projection plane "8" and the right triangle portion 306 of triangle projection plane "4" are repositioned and connected to triangle projection planes "5" and "1", respectively.

The right triangle portion 302 of the triangle projection plane "8" has three sides S811, S812 and s83_1, where side S811 is side S81 of the triangle projection plane "8" and side s83_1 is the first portion of side S83 of the triangle projection plane "8". The right triangle portion 304 of the triangle projection plane "8" has three sides S821, S822 and s83_2, wherein side S821 is side S82 of the triangle projection plane "8" and side s83_2 is the second portion of side S83 of the triangle projection plane "8".

The right triangle portion 306 of the triangle projection plane "4" has three sides S421, S422 and s43_1, wherein side S421 is side S42 of the triangle projection plane "4" and side s43_1 is the first portion of side S43 of the triangle projection plane "4". The right triangle portion 308 of the triangular projection plane "4" has three sides S411, S412 and s43_2, where side S411 is side S41 of the triangular projection plane "4" and side s43_2 is the second portion of side S43 of the triangular projection plane "4".

According to the compact octahedral projection layout 310, the side S821 of the right triangle part 304 of the triangle projection plane "8" is connected with the side S73 of the triangle projection plane "7", the side s83_2 of the right triangle part 304 of the triangle projection plane "8" is connected with the side s43_2 of the right triangle part 308 of the triangle projection plane "4", the side S411 of the right triangle part 308 of the triangle projection plane "4" is connected with the side S33 of the triangle projection plane "3", the side S811 of the right triangle part 302 of the triangle projection plane "8" is connected with the side S53 of the triangle projection plane "5", the side s83_1 of the right triangle part 302 of the triangle projection plane "8" is connected with the side s43_1 of the right triangle part 306 of the triangle projection plane "4", and the side S421 of the triangle part 306 of the triangle projection plane "4" is connected with the side S13 of the triangle projection plane "1".

There is an image content continuity boundary between the side s83_2 of the right triangle section 304 of triangle projection plane "8" and the side s43_2 of the right triangle section 308 of triangle projection plane "4". There is an image content continuity boundary between the side s83_1 of the right triangle portion 302 of the triangle projection plane "8" and the side s43_1 of the right triangle portion 306 of the triangle projection plane "4". That is, the contents are continuously represented in the triangular projection planes "4" and "8". Further, there is an image content discontinuity boundary between the side S821 of the right triangle portion 304 of the triangle projection plane "8" and the side S73 of the triangle projection plane "7", an image content discontinuity boundary between the side S411 of the right triangle portion 308 of the triangle projection plane "4" and the side S33 of the triangle projection plane "3", an image content discontinuity boundary between the side S811 of the right triangle portion 302 of the triangle projection plane "8" and the side S53 of the triangle projection plane "5", and an image content discontinuity boundary between the side S421 of the right triangle portion 306 of the triangle projection plane "4" and the side S13 of the triangle projection plane "1".

As shown at the bottom of fig. 3, the 360VR projection layout l_vr set by the compact octahedral projection layout 310 is a rectangle without any dummy area (e.g., black area or white area). Further, the portions of the 360-degree image/video content are continuously represented in the triangular projection planes "1", "2", "3", "5", "6", "7", and there is no discontinuity in the image content. However, some image content discontinuity boundaries are still inevitably present in the compact octahedral projection layout 310. Thus, if the 360VR projection layout l_vr is set by the compact octahedral projection layout 310, the image quality near the discontinuous boundary of the image content after compression may be poor.

When triangular projection planes "1" - "8" as shown in fig. 2 are rearranged and packaged in a compact octahedral projection layout 310, some of the triangular projection planes must be segmented and repositioned, resulting in image content discontinuities based on the equator 208 in the projected frame IMG. In general, the top and bottom regions of sphere 202 generally represent the "sky" and "ground," respectively, and moving objects in the surrounding environment are mostly located at the equator 208 of sphere 202. If the equator 208 represented in the projection-based frame IMG has image content discontinuities, the coding efficiency and image quality may be significantly reduced. Coding efficiency and image quality may be improved if the equator 208 of the sphere 202 is mapped along the middle of the triangular projection plane or anywhere other than the sides of the triangular projection plane.

Fig. 4 is a schematic illustration of a triangular projection surface in an octahedral projection-based format obtained from a sphere to a rotated octahedral projection. The omnidirectional image/video content of sphere 202 is mapped to eight triangular projection planes (labeled "1", "2", "3", "4", "5", "6", "7" and "8") of rotated octahedron 404. The rotated octahedron 404 shown in fig. 4 may be obtained by applying a 90 degree rotation to the octahedron 204 shown in fig. 2. As shown in fig. 4, triangular projection planes "1" - "8" are encapsulated in an octahedral projection layout 406. The shape of each triangular projection plane "1" - "8" is an equilateral triangle. For a triangular projection plane "K" (k=1-8), the plane has three sides, denoted SK1, SK2 and SK3. Sphere 202 is comprised of a left hemisphere and a right hemisphere. Triangular projection planes "1", "2", "3" and "4" are all from the right hemisphere, and triangular projection planes "5", "6", "7" and "8" are all from the left hemisphere. Due to the octahedral projections on the rotated octahedron 404, the equator 208 of the sphere 202 is not mapped along either side of each triangular projection plane. In this embodiment, the equator 208 of the sphere 202 is mapped along the middle of the triangular projection planes "2", "4", "6", and "8", as shown by the dashed lines in fig. 4. As described above, the projection-based frame IMG to be encoded needs to be rectangular. Therefore, projection-based frame IMG should use a compact octahedral projection layout.

Please refer to fig. 4 and fig. 5 in combination. FIG. 5 is a schematic diagram of a second compact octahedral projection layout according to an embodiment of the present invention. The equator 208 of the sphere 202 is mapped along the middle of the triangular projection planes "2", "4", "6" and "8", as shown by the dashed lines in fig. 5. The compact octahedral projection layout 510 is derived from the octahedral projection layout 406 by triangle projection plane rotation and triangle projection plane segmentation. As shown in the middle portion of fig. 5, the triangular projection plane "7" in the octahedral projection layout 406 rotates 60 ° clockwise, the triangular projection plane "5" in the octahedral projection layout 406 rotates 60 ° counterclockwise, the triangular projection plane "3" in the octahedral projection layout 406 rotates 60 ° counterclockwise, and the triangular projection plane "1" in the octahedral projection layout 406 rotates 60 ° clockwise. Therefore, the side S72 of the triangular projection plane "7" is connected to the side S61 of the triangular projection plane "6", and the side S51 of the triangular projection plane "5" is connected to the side S62 of the triangular projection plane "6". The side S31 of the "triangle projection plane" 3 "is connected to the side S22 of the triangle projection plane" 2", and the side S12 of the triangle projection plane" 1 "is connected to the side S21 of the triangle projection plane" 2 ".

As shown in the middle part of fig. 5, there is an image content continuity boundary between the side S72 of the triangle projection plane "7" and the side S61 of the triangle projection plane "6" (i.e., the content is continuously represented in the triangle projection planes "6" and "7"), an image content continuity boundary between the side S51 of the triangle projection plane "5" and the side S62 of the triangle projection plane "6" (i.e., the content is continuously represented in the triangle projection planes "5" and "6"), an image content continuity boundary between the side S31 of the triangle projection plane "3" and the side S22 of the triangle projection plane "2" (i.e., the content is continuously represented in the triangle projection planes "2" and "3"), an image content continuity boundary between the side S12 of the triangle projection plane "1" and the side S21 of the triangle projection plane "2" (i.e., the content is continuously represented in the triangle projection planes "1" and "2"), and an image content continuity boundary between the side S23 of the triangle projection plane "2" and the side S63 of the triangle projection plane "2" (i.e., the content is continuously represented in the triangle projection plane "2").

In addition, the triangular projection plane "4" in the octahedral projection layout 406 is divided into two right triangle parts 502 and 504, and the triangular projection plane "8" in the octahedral projection layout 406 is divided into two right triangle parts 506 and 508. As shown in the right part of fig. 5, the right triangle portion 504 of the triangle projection plane "4" and the right triangle portion 508 of the triangle projection plane "8" are connected to the triangle projection planes "1" and "5", respectively; the right triangle portion 502 of triangle projection plane "4" and the right triangle portion 506 of triangle projection plane "8" are repositioned and connected to triangle projection planes "3" and "7", respectively.

The right triangle portion 502 of the triangular projection plane "4" has three sides S411, S412 and s43_1, where side S411 is side S41 of the triangular projection plane "4" and side s43_1 is the first portion of side S43 of the triangular projection plane "4". The right triangle portion 504 of the triangle projection plane "4" has three sides S421, S422 and s43_2, wherein side S421 is side S42 of the triangle projection plane "4" and side s43_2 is the second portion of side S43 of the triangle projection plane "4".

The right triangle portion 506 of the triangle projection plane "8" has three sides S821, S822 and s83_1, where side S821 is side S82 of the triangle projection plane "8" and side s83_1 is the first portion of side S83 of the triangle projection plane "8". The right triangle portion 508 of the triangle projection plane "8" has three sides S811, S812 and s83_2, where side S811 is side S81 of the triangle projection plane "8" and side s83_2 is the second portion of side S83 of the triangle projection plane "8".

According to the compact octahedral projection layout 510, the side S421 of the right triangle portion 504 of the triangle projection plane "4" is connected with the side S13 of the triangle projection plane "1", the side s43_2 of the right triangle portion 504 of the triangle projection plane "4" is connected with the side s83_2 of the right triangle portion 508 of the triangle projection plane "8", the side S811 of the right triangle portion 508 of the triangle projection plane "8" is connected with the side S53 of the triangle projection plane "5", the side S411 of the right triangle portion 502 of the triangle projection plane "4" is connected with the side S33 of the triangle projection plane "3", the side s43_1 of the right triangle portion 502 of the triangle projection plane "4" is connected with the side s83_1 of the right triangle portion 506 of the triangle projection plane "8", and the side S821 of the triangle portion 506 of the triangle projection plane "8" is connected with the side S73 of the triangle projection plane "7".

There is an image content continuity boundary between side s43_2 of right triangle portion 504 of triangle projection plane "4" and side s83_2 of right triangle portion 508 of triangle projection plane "8". There is an image content continuity boundary between the side s43_1 of the right triangle portion 502 of the triangle projection plane "4" and the side s83_1 of the right triangle portion 506 of the triangle projection plane "8". That is, the contents are continuously represented in the triangular projection planes "4" and "8". Further, there is an image content discontinuity boundary between the side S421 of the right triangle portion 504 of the triangle projection plane "4" and the side S13 of the triangle projection plane "1", an image content discontinuity boundary between the side S811 of the right triangle portion 508 of the triangle projection plane "8" and the side S53 of the triangle projection plane "5", an image content discontinuity boundary between the side S411 of the right triangle portion 502 of the triangle projection plane "4" and the side S33 of the triangle projection plane "3", and an image content discontinuity boundary between the side S821 of the right triangle portion 506 of the triangle projection plane "8" and the side S73 of the triangle projection plane "7".

In the right-hand portion of fig. 5, the 360VR projection layout l_vr arranged by the shape of the compact octahedral projection layout 510 is a rectangle without any empty areas (e.g., areas filled in black, gray, or white). Further, the portions of the 360-degree image/video content are continuously represented in the triangular projection planes "1", "2", "3", "5", "6", "7", and there is no discontinuity in the image content. Furthermore, the equator 208 represented by the triangular projection planes "2", "4", "6" and "8" in the projection-based image IMG (which uses the compact octahedral projection layout 510) is free of image content discontinuities due to triangular projection plane segmentation. However, some image content discontinuity boundaries are still inevitably present in the compact octahedral projection layout 510. Thus, if a 360VR projection layout l_vr is set by the compact octahedral projection layout 510, the image quality near the discontinuous boundary of the image content after compression may be poor.

To solve the above-described image quality degradation problem, the present invention proposes an innovative 360VR projection layout design with padding, which can improve the image quality at the projection plane boundary after compression. For example, a 360VR projection layout l_vr may be set by a compact cube layout with padding or a compact octahedral layout with padding. Specifically, the conversion circuit 114 receives the omnidirectional image/video content of the sphere 202 from the video capture device 112 and obtains a plurality of projection planes from the omnidirectional image/video content of the sphere 202, wherein the omnidirectional image/video content of the sphere 202 is mapped onto the projection planes by a selected 360VR projection (e.g., a cube projection or an octahedral projection). As shown in fig. 1, the conversion circuit 114 has a fill circuit 115 arranged to generate at least one fill region. The conversion circuit 114 creates a projection-based frame IMG by encapsulating multiple projection facets and at least one fill region in a 360VR projection layout l_vr (e.g., a compact cube layout with fill or a compact octahedral layout with fill).

For example, a projection surface encapsulated in a 360VR projection layout l_vr includes a first projection surface and a second projection surface, wherein if a first edge of the first projection surface is connected to a first edge of the second projection surface, an image content discontinuity boundary exists between the first edge of the first projection surface and the first edge of the second projection surface. The at least one fill area encapsulated in the 360VR projection layout l_vr includes a first fill area, wherein the first fill area is connected to the first side of the first projection surface and the first side of the second projection surface for isolating the first side of the first projection surface and the first side of the second projection surface in the 360VR projection layout l_vr. The first padding area is intentionally inserted to provide more information for the compression process. Thus, the image quality of the first side of the compressed first projection surface and the first side of the second projection surface can be improved.

FIG. 6 is a schematic diagram of a first compact octahedral projection layout with filling according to an embodiment of the present invention. The 360VR projection layout L _ VR employed by the conversion circuit 114 may be set by the compact octahedral projection layout 310' shown in fig. 6. The compact octahedral projection layout 310' may be derived from the compact octahedral projection layout 310 shown in fig. 3. With respect to the compact octahedral projection layout 310 shown in fig. 3, there is an image content discontinuous boundary between the side S821 of the right triangle portion 304 of the triangle projection plane "8" and the side S73 of the triangle projection plane "7", an image content discontinuous boundary between the side S411 of the right triangle portion 308 of the triangle projection plane "4" and the side S33 of the triangle projection plane "3", an image content discontinuous boundary between the side S811 of the right triangle portion 302 of the triangle projection plane "8" and the side S53 of the triangle projection plane "5", and an image content discontinuous boundary between the side S421 of the right triangle portion 306 of the triangle projection plane "4" and the side S13 of the triangle projection plane "1". As shown in fig. 6, the first filling region pr_1 is inserted to connect with a side S421 of the right triangle portion 306 of the triangle-shaped projection plane "4" (which is also a side S42 of the projection plane "4") and a side S13 of the projection plane "1"; the second filling region pr_2 is inserted to connect with the side S411 of the right triangle section 308 of the triangle-shaped projection plane "4" (which is also the side S41 of the projection plane "4") and the side S33 of the projection plane "3"; the third filling region pr_3 is inserted to connect with the side S811 of the right triangle portion 302 of the triangle-shaped projection plane "8" (which is also the side S81 of the projection plane "8") and the side S53 of the projection plane "5"; and inserting the fourth padding region pr_4 to connect with the side S821 of the right triangle portion 304 of the triangle projection "8" (which is also the side S82 of the projection plane "8") and the side S73 of the projection plane "7". Assuming that the width of each fill region is D, and the compact octahedral projection layout 310 shown in fig. 3 has a width W and a height H, the compact octahedral projection layout 310' shown in fig. 6 has a width w+2d and a height H. For example, the width D of each fill region may be 16 pixels.

FIG. 7 is a schematic illustration of a second compact octahedral projection layout with filling according to an embodiment of the present invention. The 360VR projection layout L _ VR employed by the conversion circuit 114 may be set by the compact octahedral projection layout 510' shown in fig. 7. The compact octahedral projection layout 510' may be derived from the compact octahedral projection layout 510 shown in fig. 5. With respect to the compact octahedral projection layout 510 shown in fig. 5, there is an image content discontinuous boundary between the side S821 of the right triangle portion 506 of the triangle projection plane "8" and the side S73 of the triangle projection plane "7", an image content discontinuous boundary between the side S411 of the right triangle portion 502 of the triangle projection plane "4" and the side S33 of the triangle projection plane "3", an image content discontinuous boundary between the side S811 of the right triangle portion 508 of the triangle projection plane "8" and the side S53 of the triangle projection plane "5", and an image content discontinuous boundary between the side S421 of the right triangle portion 504 of the triangle projection plane "4" and the side S13 of the triangle projection plane "1". As shown in fig. 7, the first filling region pr_1 is inserted to connect with a side S421 of the right triangle portion 504 of the triangle-shaped projection plane "4" (which is also a side S42 of the projection plane "4") and a side S13 of the projection plane "1"; the second filling region pr_2 is inserted to connect with the side S411 of the right triangle part 502 of the triangle projection plane "4" (which is also the side S41 of the projection plane "4") and the side S33 of the projection plane "3"; the third filling region pr_3 is inserted to connect with the side S811 of the right triangle portion 508 of the triangle-shaped projection plane "8" (the side S81 of the projection plane "8" as well) and the side S53 of the projection plane "5"; and inserting the fourth padding region pr_4 to connect with the side S821 of the right triangle portion 506 of the triangle-shaped projection plane "8" (which is also the side S82 of the projection plane "8") and the side S73 of the projection plane "7". Assuming that the height of each filled region is D and the compact octahedral projection layout 510 shown in fig. 5 has a width W and a height H, the compact octahedral projection layout 510' shown in fig. 7 has a width W and a height h+2d. For example, the height D of each fill area may be 16 pixels.

In one exemplary fill implementation, fill circuit 115 sets pixel values for pixels included in the fill region by performing interpolation operations based on pixel values for pixels included in adjacent projection planes connected to the fill region. Regarding each of the compact octahedral projection layout 310 'shown in fig. 6 and the compact octahedral projection layout 510' shown in fig. 7, the pixel value of the pixel included in the first filling region pr_1 is obtained by performing an interpolation operation based on the pixel values of the pixels included in the adjacent projection planes "1" and "4"; obtaining pixel values of pixels included in the second filling region pr_2 by performing interpolation operation based on pixel values of pixels included in adjacent projection planes "3" and "4"; obtaining a pixel value of a pixel included in the third filling region pr_3 by performing interpolation operation based on pixel values of pixels included in adjacent projection planes "8" and "5"; the pixel value of the pixel included in the fourth padding region pr_4 is obtained by interpolation based on the pixel values of the pixels included in the adjacent projection planes "7" and "8".

The interpolation employed may be nearest neighbor interpolation (nearest neighbor interpolation), linear interpolation, bilinear interpolation, or other suitable interpolation algorithm. The sampling points used for the interpolation employed may be obtained from a single direction or from different directions. Fig. 8 is a schematic diagram of interpolation performed by the fill circuit 115 shown in fig. 1. The fill region PR needs to be interposed between adjacent projection planes A1 and A2, said projection planes A1 and A2 being obtained from a selected 360VR projection of the sphere, wherein if the projection plane A1 is connected with the projection plane A2, there is an image content discontinuity boundary between the adjacent projection planes A1 and A2. As shown in sub-graph (a) of fig. 8, interpolation is performed on sampling points (i.e., pixels) P1 and P2 obtained from adjacent projection planes A1 and A2 in the vertical direction. Therefore, the interpolation sampling point (i.e., interpolation pixel) S is determined from the sampling values of the sampling points P1 and P2, the distance between the sampling point P1 and the interpolation sampling point S, and the distance between the sampling point P2 and the interpolation sampling point S.

As shown in sub-graph (B) of fig. 8, sampling points (i.e., pixels) Q1 and Q2 obtained from adjacent projection planes A1 and A2 in the horizontal direction are interpolated. Therefore, the interpolation sampling point (i.e., interpolation pixel) S is determined from the sampling values of the sampling points Q1 and Q2, the distance between the sampling point Q1 and the interpolation sampling point S, and the distance between the sampling point Q2 and the interpolation sampling point S.

As shown in sub-graph (C) of fig. 8, interpolation is performed on sampling points (i.e., pixels) P1 and P2 obtained from adjacent projection planes A1 and A2 in the vertical direction and sampling points (i.e., pixels) Q1 and Q2 obtained from adjacent projection planes A1 and A2 in the horizontal direction. Thus, the interpolation sampling point (i.e., interpolation pixel) S is determined from the sampling values of the sampling points P1, P2, Q1, and Q2, the distance between the sampling point P1 and the interpolation sampling point S, the distance between the sampling point P2 and the interpolation sampling point S, the distance between the sampling point Q1 and the interpolation sampling point S, and the distance between the sampling point Q2 and the interpolation sampling point S.

In another exemplary fill implementation, fill circuit 115 applies a first geometric fill to one adjacent projection surface to determine a first pixel value of a pixel included in the fill area, applies a second geometric fill to another adjacent projection surface to determine a second pixel value of a pixel included in the fill area, and sets the pixel value of the pixel included in the fill area by mixing the first pixel value derived from the first geometric fill and the second pixel value derived from the second geometric fill. Fig. 9 is a schematic diagram of geometric filling performed by the fill circuit 115 shown in fig. 1. The fill region PR needs to be interposed between adjacent projection planes A1 and A2, which projection planes A1 and A2 are obtained by a selected 360VR projection of the sphere, wherein if the projection plane A1 is connected with the projection plane A2, there is an image content discontinuity boundary between the adjacent projection planes A1 and A2. The first geometric fill applied to projection plane A1 determines a geometric map area a1_gp, wherein geometric map area a1_gp is obtained by mapping the content of an area on a sphere (e.g., sphere 202 shown in fig. 2/4) onto a fill area PR, wherein the area on the sphere is adjacent to the area where projection plane A1 is obtained. Therefore, there is an image content continuity boundary between the projection plane A1 and the geometric map area a1_gp extending from the projection plane A1 (i.e., the content is continuously represented in the projection plane A1 and the geometric map area a1_gp).

The second geometrical filling applied to the projection surface A2 determines a further geometrical mapping region a2_gp, wherein the geometrical mapping region a2_gp is obtained by mapping the content of the region on a sphere (e.g. sphere 202 shown in fig. 2/4) onto the filling region PR, wherein the region on the sphere is adjacent to the region where the projection surface A2 is obtained. Therefore, there is an image content continuity boundary between the projection plane A2 and the geometric map area a2_gp extending from the projection plane A2 (i.e., the content is continuously represented in the projection plane A2 and the geometric map area a2_gp).

After obtaining the geometric map areas a1_gp and a2_gp associated with the same fill area PR, the fill circuit 115 mixes the geometric map areas a1_gp and a2_gp to determine pixel values of pixels included in the fill area PR. That is, pr=f (a1_gp, a2_gp), where f () is a mixing function. For example, the mixing function f () may be an average function. With respect to each pixel in the filling region PR, the pixel value of the pixel in the filling region PR is set by an average value of the first pixel value of the pixel in the geometric map region a1_gp and the second pixel value of the pixel in the geometric map region a2_gp.

In yet another exemplary fill implementation, fill circuit 115 sets the pixel values of the pixels included in the fill area by copying the pixel values of the pixels included in the adjacent projection plane obtained from the selected 360VR projection of the sphere. For example, the boundary pixels of one side of projection plane A1 are copied to create filler pixels extending from one side of projection plane A1, and the boundary pixels of one side of projection plane A2 are copied to create filler pixels extending from one side of projection plane A2. In other words, the first portion of the fill region PR includes fill pixels, each of which is a replica of one boundary pixel of the projection plane A1, and the second portion of the fill region PR includes fill pixels, each of which is a replica of one boundary pixel of the projection plane A2.

If the first side of the first projection surface is connected to the first side of the second projection surface and has an image content discontinuity boundary between the first side of the first projection surface and the first side of the second projection surface, the image quality of the first side of the first projection surface after compression and the image quality of the first side of the second projection surface after compression can be improved by inserting a fill region between the first projection surface and the second projection surface. The projection plane included in the decoded frame IMG' generated by the decoding circuit 122 may have better image quality. As described above, the graphics rendering circuit 124 renders and displays output image material on the display screen 126 according to the decoded frame IMG'. Since the padding areas in the decoded frame IMG ' are additionally added and may not be displayed, the graphics rendering circuit 124 may discard/ignore the padding areas in the decoded frame IMG ' after generating the decoded frame IMG ' from the decoding circuit 122.

As shown in fig. 6 and 7, padding is added to the compact octahedral projection layout for improved image quality for compression at the discontinuous boundaries of image content. However, these are for illustrative purposes only and are not meant to be a limitation of the present invention. In practice, padding may be added to other 360VR projection layouts to improve the quality of the compressed image at the discontinuous boundaries of the image content. These alternative projection designs with padding are within the scope of the invention.

FIG. 10 is a schematic diagram of a compact cube projection layout with fill in accordance with an embodiment of the invention. The omnidirectional image/video content of the sphere is mapped onto six square projection planes by cube projection, wherein the square projection planes include a left projection plane labeled "L", a front projection plane labeled "FR", a right projection plane labeled "R", a top projection plane labeled "T", a rear projection plane labeled "BK", and a bottom projection plane labeled "B". If the bottom edge of the left projection plane "L" is connected to the top edge of the bottom projection plane "B" in a compact cube projection layout without padding, there is a discontinuous boundary of image content between the left projection plane "L" and the bottom projection plane "B". If the bottom side of the front projection surface FR is connected with the top side of the rear projection surface BK in a compact cube projection layout without padding, there is an image content discontinuity boundary between the front projection surface FR and the rear projection surface BK. If the bottom side of the right projection plane R in a compact cube projection layout without padding is connected to the top side of the top projection plane T, there is an image content discontinuity boundary between the right projection plane R and the top projection plane T. According to the compact cube projection arrangement 1002 shown in fig. 10, a first fill region pr_1 is interposed between the left side projection plane L and the bottom projection plane B, a second fill region pr_2 is interposed between the front projection plane FR and the rear projection plane BK, and a third fill region pr_3 is interposed between the right side projection plane R and the top projection plane T. Each of the padding areas pr_1-pr_3 may be generated by selecting one of the geometric padding and the replication methods using the interpolation method described above.

For example, the replication approach used by the proposed fill technique may extend the boundary pixels of the projection surface. Thus, a first portion of the fill area interposed between the first projection surface and the second projection surface includes fill pixels, wherein each fill pixel is a replica of one boundary pixel of the first projection surface, and a second portion of the fill area interposed between the first projection surface and the second projection surface includes fill pixels, wherein each fill pixel is a replica of one boundary pixel of the second projection surface.

For another example, the pixel values of pixels included in the filling area may be set by copying pixel values of pixels included in the first projection plane and the second projection plane but not connected to the filling area between the first projection plane and the second projection plane in a copy manner used by the proposed filling technique. In case the proposed filling technique uses a replication approach to obtain a replica of the partial area in the projection surface. The first portion of the fill area interposed between the first projection surface and the second projection surface is a replica of a partial area of the first projection surface, and the second portion of the fill area interposed between the first projection surface and the second projection surface is a replica of a partial area of the second projection surface, wherein neither the partial area of the first projection surface nor the partial area of the second projection surface is connected to the fill area interposed between the first projection surface and the second projection surface.

For another example, a copy manner used by the proposed filling technique may set pixel values of pixels included in a filling area interposed between the first projection surface and the second projection surface by copying pixel values of pixels included in at least one projection surface different from the first projection surface and the second projection surface. Taking the compact cube projection arrangement 1002 shown in fig. 10 as an example, a first fill region pr_1 interposed between the left side projection plane L and the bottom projection plane B may be set by copying pixels (e.g., pixels of a partial region) in at least one projection plane (e.g., FR, BK, R, and/or T, which is not any of the left side projection plane L and the bottom projection plane B). The second filling region pr_2 interposed between the front projection surface FR and the rear projection surface BK may be set by copying pixels (e.g., pixels of a partial region) in at least one projection surface (e.g., L, B, R, and/or T, which is not any one of the front projection surface FR and the rear projection surface BK), and/or the third filling region pr_3 interposed between the right projection surface R and the top projection surface T may be set by copying pixels (e.g., pixels of a partial region) in at least one projection surface (e.g., L, B, FR, and/or BK, which is not any one of the right projection surface R and the top projection surface T).

FIG. 11 is a schematic illustration of a third compact octahedral projection layout with filling according to an embodiment of the present invention. By octahedral projection, the omnidirectional image/video content of the sphere is mapped to eight triangular projection surfaces (labeled with reference numerals "1", "2", "3", "4", "5", "6", "7" and "8"). The triangular projection plane "8" is divided into two right triangle sections. If one side of one right triangle portion of the triangular projection plane "8" is connected with one side of the triangular projection plane "1" in a compact octahedral projection layout without filling, there is an image content discontinuity boundary between one right triangle portion of the triangular projection plane "8" and the triangular projection plane "1". If the other side of the triangular projection plane "1" is connected to one side of the triangular projection plane "5" in a compact octahedral projection layout without padding, there is an image content discontinuity boundary between the triangular projection planes "1" and "5". If the other side of the triangular projection plane "5" is connected to one side of the triangular projection plane "2" in a compact octahedral projection layout without padding, there is an image content discontinuity boundary between the triangular projection planes "5" and "2". If the other side of the triangular projection plane "2" is connected to one side of the triangular projection plane "6" in a compact octahedral projection layout without padding, there is an image content discontinuity boundary between the triangular projection planes "2" and "6". If the other side of projection plane "6" in a compact octahedral projection layout without padding is connected to one side of triangular projection plane "3", there is an image content discontinuity boundary between triangular projection planes "6" and "3". If the other side of the triangular projection plane "3" is connected to one side of the triangular projection plane "7" in a compact octahedral projection layout without padding, there is an image content discontinuity boundary between the triangular projection planes "3" and "7". If the other side of projection plane "7" in a compact octahedral projection layout without padding is connected to one side of triangular projection plane "4", there is an image content discontinuity boundary between triangular projection planes "7" and "4". If one side of the other right triangle portion of the triangular projection plane "8" is connected with the other side of the triangular projection plane "4" in the compact octahedral projection layout without filling, there is an image content discontinuity boundary between the other right triangle portion of the triangular projection plane "8" and the triangular projection plane "4".

According to the compact octahedral projection layout 1102 shown in fig. 11, a first filling region pr_1 is inserted between triangle projection planes "1" and one right triangle portion of triangle projection plane "8", a second filling region pr_2 is inserted between triangle projection planes "1" and "5", a third filling region pr_3 is inserted between triangle projection planes "5" and "2", a fourth filling region pr_4 is inserted between triangle projection planes "2" and "6", a fifth filling region pr_5 is inserted between triangle projection planes "6" and "3", a sixth filling region pr_6 is inserted between triangle projection planes "3" and "7", a seventh filling region pr_7 is inserted between triangle projection planes "7" and "4", and an eighth filling region pr_8 is inserted between triangle projection planes "4" and another right triangle portion of triangle projection planes "8". Each of the padding regions pr_1 to pr_8 may be generated by selecting one of the geometric padding mode and the replication mode using the interpolation mode described above.

In addition to the image quality of the discontinuous boundaries of the image content after compression, the image quality of the layout boundaries after compression can be improved by the proposed filling technique. For example, when mapping the omnidirectional image/video content of a sphere by equidistant rectangular projection (equirectangular projection, ERP) or Equal Area Projection (EAP), only a single projection plane is generated and arranged in the ERP/EAP layout. If the view of the viewport is 180 degrees, the decoded partial region at the left boundary of the ERP/EAP layout and the decoded partial region at the right boundary of the ERP/EAP layout are selected to combine to form the viewport region to be displayed. Since the blocks at the left and right boundaries of a typical ERP/EAP layout are independently encoded, a viewport region is created from a combination of decoded partial regions located at the left boundary and decoded partial regions located at the right boundary of the ERP/EAP layout, and thus artifacts may be created in the viewport region along the location of the layout boundary. To solve this problem, the present invention further proposes to add a filler region to the layout boundary to provide more information for the compression process.

FIG. 12 is a schematic diagram of a first ERP/EAP layout with padding, according to an embodiment of the invention. A single projection plane A with top S_T, bottom S_B, left S_L and right S_R is arranged in ERP/EAP layout 1202 without padding. Top S_T, bottom S_B, left S_L and right S_R are four boundaries of ERP/EAP layout 1202. In addition, the projection plane A in the ERP/EAP layout 1202 has a first partial region P_L that includes boundary pixels at the left side S_L and a second partial region P_R that includes boundary pixels at the right side S_R. The 360VR projection layout L _ VR employed by the translation circuit 114 may be set by the ERP/EAP layout 1202' shown in fig. 12. ERP/EAP layout 1202' may be derived from ERP/EAP layout 1202. Projection plane a is obtained by equidistant rectangular projection/equal area projection of the sphere. The projection plane a, the first fill region pr_l and the second fill region pr_r are encapsulated in the ERP/EAP layout 1202'. As shown in fig. 12, the first fill region pr_l is connected to the left side s_l of the projection plane a and forms the left boundary of the ERP/EAP layout 1202', and the second fill region pr_r is connected to the right side s_r of the projection plane a and forms the right boundary of the ERP/EAP layout 1202'. For example, the width of the first filling region pr_l may be 8 pixels and the width of the second filling region pr_r may be 8 pixels. Since the left side s_l and the right side s_r of the projection plane a are opposite sides, the first filling region pr_l is not connected to the right side s_r of the projection plane a, and the second filling region pr_r is not connected to the left side s_l of the projection plane a.

In this embodiment, the first filling region pr_l is a replica of the second partial region p_r of the projection plane a, and the second filling region pr_r is a replica of the first partial region p_l of the projection plane a. Therefore, the pixels of the first filling region pr_l include boundary pixels at the right side s_r of the projection plane a, but do not include boundary pixels at the left side s_l of the projection plane a; the pixels of the second filling region pr_r include boundary pixels at the left side s_l of the projection plane a, but do not include boundary pixels at the right side s_r of the projection plane a. Due to the inherent nature of equidistant rectangular projection/equal area projection, there is an image content continuity boundary between the first fill area PR_L and the projection plane A enclosed in the ERP/EAP layout 1202 '(i.e., the content is continuously represented in the projection plane A and the first fill area PR_L), and an image content continuity boundary between the projection plane A and the second fill area PR_R enclosed in the ERP/EAP layout 1202' (i.e., the content is continuously represented in the projection plane A and the second fill area PR_R).

In addition to adding fill areas to the left and right sides of the projection surface obtained by equidistant rectangular projection/equal area projection, fill areas can also be added to the top and bottom sides of the projection surface, providing more information for the compression process.

FIG. 13 is a schematic diagram of a second ERP/EAP layout with padding, according to an embodiment of the invention. A single projection plane A with top S_T, bottom S_B, left S_L and right S_R is arranged in ERP/EAP layout 1302 without padding. Top S_T, bottom S_B, left S_L and right S_R are four boundaries of ERP/EAP layout 1302. In addition, projection plane a in ERP/EAP layout 1302 has a plurality of image areas (labeled "1", "2", "3", "4", "5", "6", "7", and "8"). The image areas "1" - "3" form a partial area and include boundary pixels at the top edge s_t. The image areas "3" - "5" form one partial area, and include boundary pixels at the right side s_r. The image areas "5" - "7" form a partial area and include boundary pixels at the bottom side s_b. The image areas "1", "8" and "7" form one partial area, and include boundary pixels at the left side s_l.

The 360VR projection layout L _ VR employed by the translation circuit 114 may be set by the ERP/EAP layout 1302' shown in fig. 13. ERP/EAP layout 1302' may be derived from ERP/EAP layout 1302. The projection plane a is obtained by equidistant rectangular projection/equal area projection of the sphere. As shown in fig. 13, a first filled region connected to the left side s_l of the projection plane a is generated by copying the image regions "3" - "5" of the projection plane a; generating a second filling area connected to the right side s_r of the projection plane a by copying the image areas "1", "8" and "7" of the projection plane a; a third filling area connected to the top side s_t of the projection plane a is generated by copying the image areas "1" - "3" of the projection plane a to obtain a copied partial area, and then flipping (flipping) the copied partial area; and generating a fourth filling area connected to the bottom side s_b of the projection plane a by copying the image area "5" - "7" of the projection plane a to obtain a copied partial area and then inverting the copied partial area.

To change the shape of the ERP/EAP layout 1302' to rectangular, the top left fill area is generated by copying image area "3" to obtain a copied fill area, and then flipping the copied fill area; obtaining a copied filled region by copying the image region "1", and then flipping the copied filled region to generate an upper right corner filled region; obtaining a copied filled region by copying the image region "5", and then flipping the copied filled region to generate a left lower corner filled region; and obtaining a copied filled region by copying the image region "7", and then flipping the copied filled region to generate a bottom right filled region.

Due to the inherent characteristics of equidistant rectangular projection/equal area projection, there is an image content continuity boundary between the upper left fill area and the first fill area, an image content continuity boundary between the upper left fill area and the third fill area, an image content continuity boundary between the upper right fill area and the second fill area, an image content continuity boundary between the upper right fill area and the third fill area, an image content continuity boundary between the lower left fill area and the first fill area, an image content continuity boundary between the lower left fill area and the fourth fill area, an image content continuity boundary between the lower right fill area and the second fill area, and an image content continuity boundary between the lower right fill area and the fourth fill area.

As shown in fig. 13, a first fill area connected to the left side s_l of projection plane a forms part of the left boundary of the ERP/EAP layout 1302', a second fill area connected to the right side s_r of projection plane a forms part of the right boundary of the ERP/EAP layout 1302', a third fill area connected to the top side s_t of projection plane a forms part of the top boundary of the ERP/EAP layout 1302', and a fourth fill area connected to the bottom side s_b of projection plane a forms part of the bottom boundary of the ERP/EAP layout 1302'. Due to the inherent nature of equidistant rectangular projection/equal area projection, there is an image content continuity boundary between the first fill area and projection plane a encapsulated in the ERP/EAP layout 1302 '(i.e., content is continuously represented in the first fill area and projection plane a), an image content continuity boundary between the second fill area and projection plane a encapsulated in the ERP/EAP layout 1302' (i.e., content is continuously represented in the second fill area and projection plane a), an image content continuity boundary between the third fill area and projection plane a encapsulated in the ERP/EAP layout 1302 '(i.e., content is continuously represented in the third fill area and projection plane a), and an image content continuity boundary between the fourth fill area and projection plane a encapsulated in the ERP/EAP layout 1302' (i.e., content is continuously represented in the fourth fill area and projection plane a).

FIG. 14 is a schematic diagram of a third ERP/EAP layout with padding, according to an embodiment of the invention. A single projection plane A with top S_T, bottom S_B, left S_L and right S_R is arranged in ERP/EAP layout 1402 without padding. Top S_T, bottom S_B, left S_L and right S_R are four boundaries of ERP/EAP layout 1402. In addition, the projection plane a in the ERP/EAP layout 1402 has a plurality of image areas (labeled "1", "2", "3", "4", "5", "6", "7", and "8"). The image areas "1" - "3" form a partial area and include boundary pixels at the top edge s_t. The image areas "3" - "5" form one partial area, and include boundary pixels at the right side s_r. The image areas "5" - "7" form a partial area and include boundary pixels at the bottom side s_b. The image areas "7" - "8" and "1" form one partial area, and include boundary pixels at the left side s_l.

The 360VR projection layout L _ VR employed by the translation circuit 114 may be set by the ERP/EAP layout 1402' shown in fig. 14. ERP/EAP layout 1402' may be derived from a typical ERP/EAP layout 1402. Projection plane a is obtained by equidistant rectangular projection/equal area projection of the sphere. As shown in fig. 14, by copying the image areas "3" - "5" of the projection plane a, a first filled area connected to the left side s_l of the projection plane a is generated; generating a second filling area connected to the right side s_r of the projection plane a by copying the image areas "1", "8" and "7" of the projection plane a; a third filling area connected to the top side s_t of the projection plane a is generated by copying the image area "1" - "3" of the projection plane a to obtain a copied partial area, and then rotating (rotating) the copied partial area by 180 °; and generating a fourth filled region connected to the bottom side s_b of the projection plane a by copying the image region "5" - "7" of the projection plane a to obtain a copied partial region, and then rotating the copied partial region by 180 °.

To change the shape of the ERP/EAP layout 1402' to rectangular, a copied fill area is obtained by copying image area "1" and then rotating the copied fill area 180 to generate an upper left fill area; obtaining a copied filled region by copying the image region "3", and then rotating the copied filled region by 180 °, to generate an upper right corner filled region; obtaining a copied filled region by copying the image region "7", and then rotating the copied filled region by 180 °, to generate a left lower corner filled region; and obtaining a copied filled region by copying the image region "5", and then rotating the copied filled region by 180 °, to generate a right-bottom corner filled region.

As shown in fig. 14, a first fill area connected to the left side s_l of projection plane a forms part of the left side boundary of ERP/EAP layout 1402', a second fill area connected to the right side s_r of projection plane a forms part of the right side boundary of ERP/EAP layout 1402', a third fill area connected to the top side s_t of projection plane a forms part of the top boundary of ERP/EAP layout 1402', and a fourth fill area connected to the bottom side s_b of projection plane a forms part of the bottom boundary of ERP/EAP layout 1402'. Due to the inherent nature of equidistant rectangular projection/equal area projection, there is an image content continuity boundary between the first fill area and projection plane a encapsulated in the ERP/EAP layout 1402 '(i.e., the content is continuously represented in the first fill area and projection plane a), an image content continuity boundary between the second fill area and projection plane a encapsulated in the ERP/EAP layout 1402' (i.e., the content is continuously represented in the second fill area and projection plane a), an image content continuity boundary between the third fill area and projection plane a encapsulated in the ERP/EAP layout 1402 '(i.e., the content is continuously represented in the third fill area and projection plane a), and an image content continuity boundary between the fourth fill area and projection plane a encapsulated in the ERP/EAP layout 1402' (i.e., the content is continuously represented in the fourth fill area and projection plane a).

As shown in fig. 12-14, padding is added to the ERP/EAP layout for improving the compressed image quality at the layout boundaries. However, these are for illustrative purposes only and are not meant to be a limitation of the present invention. In practice, padding may be added to other 360VR projection layouts to improve the compressed image quality at layout boundaries. These alternative projection designs with padding are within the scope of the invention.

FIG. 15 is a schematic diagram of an octahedral projection layout with filling according to an embodiment of the present invention. FIG. 16 is a schematic diagram of a cube projection layout with fill in accordance with an embodiment of the invention. FIG. 17 is a schematic diagram of a fourth compact octahedral projection layout with filling according to an embodiment of the present invention. FIG. 18 is a schematic diagram of a compact cube projection layout with fill in accordance with an embodiment of the invention. The fill area encapsulated in the projection layout 1502/1602/1702/1802 can be generated by the geometric fill method described above that applies geometric fill to the projection surface to determine pixel values for pixels included in the fill area connected to the projection surface; or may be generated by the above-described copy manner of setting the pixel values of the pixels included in the filling area connected to the projection surface by copying the pixel values of the boundary pixels of the projection surface, or by copying the pixel values of the pixels included in the projection surface but not connected to the filling area, or by copying the pixel values of the pixels not included in the projection surface.

It should be noted that the foregoing layout examples are for illustrative purposes only and are not meant to limit the present invention. In other embodiments of the present invention, a 360VR projection layout with padding may be obtained by adding padding areas to a layout of other projection formats, such as a pyramid (pyramid) projection layout, a tetrahedron (tetrahedron) projection layout, a quadrilateral quartz-based projection layout (tetrahedron-based projection layout), an icosahedron (icosahedron) projection layout, or a hexagonal quartz-based projection layout (hexagon-based projection layout).

By means of the filling area added to the boundary of the projection layout, the image quality of the boundary after compression can be improved. As described above, the graphics rendering circuit 124 renders and displays output image material on the display screen 126 according to the decoded frame IMG'. Since the padding areas in the decoded frame IMG ' are additionally added and may not be displayed, the graphics rendering circuit 124 may discard/ignore the padding areas in the decoded frame IMG ' after generating the decoded frame IMG ' from the decoding circuit 122.

Fig. 19 is a schematic diagram of a second 360VR system in accordance with an embodiment of the invention. The main difference between the 360VR systems 100 and 1900 is that the conversion circuit 1914 of the source electronic device 1902 has a resampling circuit 1915 arranged to perform the proposed encoder-side projection surface resampling function to adjust the sampling density (or sampling rate). For example, the encoder-side projection surface resampling function may apply downsampling (down-sampling) to the projection surface prior to encoding. For another example, the encoder-side projection surface resampling function may apply upsampling (up-sampling) to the projection surface prior to encoding. For yet another example, the encoder-side projection surface resampling function may apply resampling without a change in size to the projection surface prior to encoding.

In some embodiments of the invention, the 360VR projection layout L_VR is a cube projection layout, which is a non-viewport based projection layout (non-viewport based projection layout). Thus, the conversion circuit 1914 obtains a plurality of square projection surfaces from the omnidirectional image/video content of the sphere, wherein the omnidirectional image/video content of the sphere is mapped onto the plurality of square projection surfaces via the cube projection (cubemap projection, CMP). Fig. 20 is a schematic view of six square projection surfaces according to a cube projection layout obtained from a cube projection of a sphere. The omnidirectional image/video content of sphere 2002 is mapped onto six square projection planes (labeled "L", "F", "R", "BK", "T" and "B") of cube 2004. The square projection plane "L" represents the left side of the cube 2004. The square projection plane "F" represents the front face of the cube 2004. The square projection plane "R" represents the right side of the cube 2004. The square projection plane "BK" represents the back face of the cube 2004. The square projection plane "T" represents the top surface of the cube 2004. The square projection plane "B" represents the bottom surface of the cube 2004. As shown in fig. 20, square projection planes "L", "F", "R", "BK", "T", and "B" are arranged in the CMP layout 2006 corresponding to the expanded cubes. The projection-based frame IMG to be encoded needs to be rectangular. If the CMP layout 2006 is directly used to create a projection-based frame IMG, the projection-based frame IMG must fill in empty regions (e.g., regions filled in black, gray, or white) to form a rectangular frame for encoding. Thus, the square projection planes "L", "F", "R", "BK", "T" and "B" may be packaged in another CMP layout, such as a 1x6 cube layout, a 6x1 cube layout, a 3x2 cube layout, or a 2x3 cube layout. In this way, the encoding efficiency can be improved.

One or more of the square projection planes "L", "F", "R", "BK", "T", and "B" may be processed by the resampling circuit 1915 before being packaged into a 360VR projection layout l_vr, which is a CMP layout (e.g., a 1x6 cube layout, a 6x1 cube layout, a 3x2 cube layout, or a 2x3 cube layout). For example, the resampling circuit 1915 resamples at least a portion (i.e., part or all) of a square projection surface to obtain a resampled projection surface by the proposed encoder-side projection surface resampling function. The conversion circuit 1914 generates a projection-based frame IMG according to the employed CMP layout (e.g., a 1x6 cube layout, a 6x1 cube layout, a 3x2 cube layout, or a 2x3 cube layout), where the projection-based frame IMG has one or more resampled projection surfaces encapsulated in the employed CMP layout. The encoder-side projection surface resampling function may be a resampling function with non-uniform mapping or a resampling function with uniform mapping, depending on practical design considerations. Further details of uniform and non-uniform mapping are described below.

Please refer to fig. 21 and 22 in combination. FIG. 21 is a schematic diagram of resampling a square projection surface obtained from a cube projection by uniform mapping, in accordance with an embodiment of the invention. FIG. 22 is a schematic diagram of a uniform mapping function curve according to an embodiment of the invention. The square projection surface 2102 to be resampled may be any one of the square projection surfaces "L", "F", "R", "BK", "T", "B" shown in fig. 20. In this example, the square projection surface 2104 is downsampled from the square projection surface 2102 in the height direction and the width direction thereof, wherein the square projection surface 2102 has a width W and a height H (h=w), and the square projection surface 2104 has a width W and a height H (w=h < H). That is, the downsampling from the height H to the height H is performed by the uniform mapping, and the downsampling from the width W to the width W is performed by the uniform mapping. By way of example and not limitation, the same uniform mapping function is applied to the width direction (i.e., x-axis direction) and the height direction (i.e., y-axis direction). For example, the following formulas may be used to represent uniform mapping functions in different directions.

Thus, with integer pixel locations located at Y-axis coordinate Y in square projection plane 2104, a corresponding sample point located at Y-axis coordinate Y in square projection plane 2102 can be determined according to the uniform mapping function represented in equation (1). Due to the uniform mapping in the height direction, two vertically adjacent sampling points in the square projection surface 2102 are uniformly distributed at a constant distance D. Similarly, with integer pixel locations at X-axis coordinate X in square projection plane 2104, the corresponding sample points at X-axis coordinate X in square projection plane 2102 can be determined according to the uniform mapping function represented in equation (2). Due to the uniform mapping in the width direction, two horizontally adjacent sampling points in the rectangular projection surface 2102 are uniformly distributed at a constant distance D'. According to equations (1) and (2), the pixel value of position P in square projection surface 2104 is derived using the pixel value of the corresponding sampling position P' found in square projection surface 2102.

The sampling points (i.e., the obtained pixel positions p') in the square projection surface 2102 may not be at integer positions. If at least one of the x-axis coordinates x and y-axis coordinates y of the sampling points in the square projection surface 2102 is a non-integer position, an interpolation filter (not shown) in the conversion circuit 1914 (specifically, the resampling circuit 1915) may be applied to integer pixels around the sampling points in the square projection surface 2102 to derive pixel values of the sampling points.

To preserve more detail of a specific region within the projection plane, the present invention also proposes to resample the square projection plane obtained from the cube projection by means of a non-uniform mapping. FIG. 23 is a schematic diagram of a first example of resampling a square projection surface obtained from a cube projection by non-uniform mapping, in accordance with an embodiment of the invention. The square projection plane 2302 to be resampled may be any one of the square projection planes "L", "F", "R", "BK", "T", "B" shown in fig. 20. In this example, the square projection surface 2304 is obtained from downsampling the square projection surface 2302 in its height direction (i.e., y-axis direction) and width direction (i.e., x-axis direction), wherein the square projection surface 2302 has a width W and a height H (h=w), and the square projection surface 2304 has a width W and a height H (w=h < H).

With integer pixel locations at the y-axis coordinates in square projection plane 2304, the corresponding sampling points at the y-axis coordinates in square projection plane 2302 may be determined according to a non-uniform mapping function. As shown in fig. 23, the interval between two vertically adjacent sampling points is not constant. For example, the spacing between two vertically adjacent sampling points may be one of D1, D2, D3, and D4, where D4> D3> D2> D1. Specifically, the sampling points are unevenly distributed in the height direction of the square projection surface 2302. For example, pixels in the first resampled area 2322 of square projection surface 2304 are obtained by resampling the first source area 2312 of square projection surface 2302; and obtaining pixels in the second resampled area 2324 of the square projection surface 2304 by resampling the second source area 2314 of the square projection surface 2302. The density of sampling points obtained from the first source region 2312 is different from the density of sampling points obtained from the second source region 2314 due to the non-uniform mapping in the height direction. In other words, different sampling rates are used in the height direction of the square projection surface 2302. The first resampling area 2322 is derived from resampling the first source area 2312 in the elevation direction at a first sampling rate (or first sampling density) and the second resampling area 2324 is derived from resampling the second source area 2314 in the elevation direction at a second sampling rate (or second sampling density), wherein the second sampling rate (or second sampling density) is different from the first sampling rate (or first sampling density).

Similarly, with integer pixel locations at the x-axis coordinates in square rectangular projection plane 2304, the corresponding sampling points at the x-axis coordinates in square projection plane 2302 may be determined by a non-uniform mapping function. As shown in fig. 23, the interval between two horizontally adjacent sampling points is not constant. For example, the spacing between two horizontally adjacent sampling points may be one of D1', D2', D3', and D4', where D4'> D3' > D2'> D1'. Specifically, the sampling points are unevenly distributed in the width direction of the square projection surface 2302. For example, pixels in the first resampled area 2326 of square projection surface 2304 are obtained by resampling first source area 2316; and obtaining pixels in the second resampled area 2328 of square projection surface 2304 by resampling the second source area 2318 of square projection surface 2302. Due to the non-uniform mapping in the width direction, the density of sampling points obtained from the first source region 2316 is different from the density of sampling points obtained from the second source region 2318. In other words, different sampling rates are used in the width direction of the square projection surface 2302. The first resampling area 2326 is derived from resampling the first source area 2316 in the width direction at a first sampling rate (or first sampling density), and the second resampling area 2326 is derived from resampling the second source area 2318 in the width direction at a second sampling rate (or second sampling density), wherein the second sampling rate (or second sampling density) is different from the first sampling rate (or first sampling density).

The pixel value of position P in square projection plane 2304 is derived from the non-uniform mapping function used in the x-axis direction and the y-axis direction using the pixel value of the corresponding sampling position P' found in square projection plane 2302. The sampling point (i.e., the obtained pixel position p') in the square projection plane 2302 may not be at an integer position. If at least one of the x-axis coordinates x and y-axis coordinates y of the sampling points in the square projection plane 2302 is a non-integer position, an interpolation filter (not shown) in the conversion circuit 1914 (specifically, the resampling circuit 1915) may be applied to integer pixels around the sampling points in the square projection plane 2302 to derive pixel values of the sampling points.

In this example, the non-uniform mapping applied to square projection surface 2302 includes a first non-uniform mapping function for resampling at least a portion (i.e., a portion or all) of square projection surface 2302 in a first direction (e.g., one of a width direction and a height direction) and a second non-uniform mapping function for resampling at least a portion (i.e., a portion or all) of square projection surface 2302 in a second direction (e.g., the other of a width direction and a height direction). In one exemplary design, the first non-uniform mapping function may be the same as the second non-uniform mapping function. That is, the first direction and the second direction (e.g., the width direction and the height direction) may use the same non-uniform mapping curve. In another exemplary design, the first non-uniform mapping function may be different from the second non-uniform mapping function. That is, the first direction and the second direction (e.g., the width direction and the height direction) may use different non-uniform mapping curves.

In the example shown in fig. 23, the resampled square projection surface is derived by applying downsampling to the original square projection surface by non-uniform mapping. However, this is for illustrative purposes only and is not meant to be a limitation of the present invention. Alternatively, the resampled square projection surface may be derived by applying upsampling to the original square projection surface by non-uniform mapping, or the resampled square projection surface may be derived by applying resampling, which is not subject to a change in size, to the original square projection surface by non-uniform mapping. FIG. 24 is a schematic diagram of a second example of resampling a square projection surface obtained from a cube projection by non-uniform mapping, in accordance with an embodiment of the invention. The square projection surface 2402 to be resampled may be any one of the square projection surfaces "L", "F", "R", "BK", "T", "B" shown in fig. 20. In this example, resampling is employed without changing size. Thus, the square projection surface 2404 is obtained by resampling the square projection surface 2402 in its height direction (i.e., y-axis direction) and width direction (i.e., x-axis direction), wherein the square projection surface 2402 has a width W and a height H (h=w), and the square projection surface 2404 has a width W and a height H (w=h=h). With integer pixel locations at the y-axis coordinates in the square rectangular projection surface 2404, corresponding sample points at the y-axis coordinates in the square projection surface 2402 may be determined from the non-uniform mapping function in the y-axis direction. With integer pixel locations at x-axis coordinates in square rectangular projection surface 2404, corresponding sample points at x-axis coordinates in square projection surface 2402 may be determined in the x-axis direction from a non-uniform mapping function, where the non-uniform mapping function used in the x-axis direction may be the same as or different from the non-uniform mapping function used in the y-axis direction.

In practice, the non-uniform mapping function may be implemented using any non-decreasing function (non-decreasing function) that passes through coordinate points (0, 0) and (1, 1). That is, the curve of the non-decreasing non-uniform mapping function (non-decreasing non-uniform mapping function) starts at coordinate point (0, 0) and ends at coordinate point (1, 1). For example, the non-uniform mapping function may be a piecewise linear function (piecewise linear function), an exponential function (exponential function), a quadratic function (quadratic equation function), or other functions. FIG. 25 is a schematic diagram of a first non-uniform mapping function curve according to an embodiment of the invention. FIG. 26 is a schematic diagram of a second non-uniform mapping function curve according to an embodiment of the invention. The non-uniform mapping curves shown in fig. 25-26 are for illustration purposes only and are not meant to limit the invention.

Consider the case where the non-uniform mapping function is set by a quadratic function. The quadratic function can be defined as f (p) =a×p ² +b×p, wherein a+b=1, p represents the sum of the values of%For example, the x-axis direction or the y-axis direction) and f (p) represents the pixel position within the resampled square projection plane in the selected direction. According to the experimental result, A can be set to-0.385 and B can be set to 1.385, so that the non-uniform mapping function has the best BD-rate # rate)。

Furthermore, the non-uniform mapping function applied to different square projection planes obtained by cube projection is not necessarily the same. For example, the resampling circuit 1915 resamples at least a portion (i.e., part or all) of a first square projection surface (e.g., one of the square projection surfaces "L", "F", "R", "BK", "T", and "B" shown in fig. 20) by non-uniform mapping to obtain a first resampled projection surface; at least a portion (i.e., part or all) of a second projection plane (e.g., another one of square projection planes "L", "F", "R", "BK", "T", and "B" shown in fig. 20) is resampled by non-uniform mapping to obtain a second resampled projection plane, wherein the first resampled projection plane and the second resampled projection plane are contained in a projection-based frame IMG packaged into a 360VR projection layout l_vr (which is a CMP layout), and at least one non-uniform mapping function (e.g., a non-uniform mapping function in a width direction and/or a non-uniform mapping function in a height direction) for resampling the first projection plane is different from at least one non-uniform mapping function (e.g., a non-uniform mapping function in a width direction and/or a non-uniform mapping function in a height direction) for resampling the second projection plane.

Fig. 27 is a schematic diagram of a third 360VR system in accordance with an embodiment of the invention. The main difference between the 360VR systems 1900 and 2700 is that: the conversion circuit 2714 of the source electronic device 2702 has a resampling circuit 2715 and a fill circuit 2716. Similar to the resampling circuit 1915 shown in fig. 19, the resampling circuit 2715 is arranged to perform the proposed encoder-side projection surface resampling function to adjust the sampling density (or sampling rate). Similar to the filler circuit 115 shown in fig. 1, the filler circuit 2716 is arranged to generate at least one filler region for reducing artifacts. The conversion circuit 2714 creates a projection-based frame IMG by encapsulating the resampled projection surface and at least one fill area in a projection layout with fill. For example, the 360VR projection layout l_vr is a CMP layout with padding.

With respect to the embodiment shown in fig. 19, the resampled square projection surface is packaged in a CMP layout without padding, such as a 1x6 cube layout, a 6x1 cube layout, a 3x2 cube layout, or a 2x3 cube layout. However, the projection-based frame IMG after encoding may have artifacts due to layout boundaries of CMP and/or discontinuous boundaries of CMP layout. For example, a CMP layout without padding has a top layout boundary, a bottom layout boundary, a left side layout boundary, and a right side layout boundary. In addition, there is at least one image content discontinuity boundary between two adjacent resampled square projection surfaces of the package in a CMP layout without padding. Around layout boundaries, discontinuous boundaries and/or transitions in sampling rate, additional guard bands (additional guard band) generated by pixel filling may be inserted to reduce seam artifacts (seam artifacts).

In a first exemplary guard band design, pixel fills may be added only at discontinuous boundaries. FIG. 28 is a schematic diagram of a cube projection layout with internal boundary filling in accordance with an embodiment of the invention. Subplot (a) of fig. 2 shows the proposed 3 x 2 cube layout 2802 with internal boundary filling. The resampled square projection surfaces are marked by "0", "1", "2", "3", "4" and "5". For example, resampled square projection plane "0" is generated by applying the non-uniform map to square projection plane "F" shown in fig. 20, resampled square projection plane "1" is generated by applying the non-uniform map to square projection plane "L" shown in fig. 20, resampled square projection plane "2" is generated by applying the non-uniform map to square projection plane "R" shown in fig. 20, resampled square projection plane "3" is generated by applying the non-uniform map to square projection plane "BK" shown in fig. 20, resampled square projection plane "4" is generated by applying the non-uniform map to square projection plane "T" shown in fig. 20, and resampled square projection plane "5" is generated by applying the non-uniform map to square projection plane "B" shown in fig. 20.

In a typical 3x2 cube projection layout without padding, if the bottom side of the resampled square projection surface "1" is connected to the top side of the resampled square projection surface "4", there is a discontinuous boundary of image content between the resampled square projection surfaces "1" and "4". In a typical 3x2 cube projection layout without padding, if the bottom side of the resampled square projection surface "0" is connected to the top side of the resampled square projection surface "3", there is an image content discontinuity boundary between the resampled square projection surfaces "0" and "3". In a typical 3x2 cube projection layout without padding, if the bottom side of the resampled square projection surface "2" is connected to the top side of the resampled square projection surface "5", there is a discontinuous boundary of image content between the resampled square projection surfaces "2" and "5". According to the proposed 3x2 cube projection arrangement 2802 with padding, a padding region pr_de1 is inserted between resampled square projection planes "1" and "4", a padding region pr_de2 is inserted between resampled square projection planes "0" and "3", and a padding region pr_de3 is inserted between resampled square projection planes "2" and "5".

The first fill region PR_DE1 includes a guard band for the resampled square projection plane "1" and a guard band for the resampled square projection plane "4", thus isolating the bottom side of the resampled square projection plane "1" from the top side of the resampled square projection plane "4" in the projection layout 2802. The second fill region PR_DE2 includes a guard band for the resampled square projection plane "0" and a guard band for the resampled square projection plane "3", thus isolating the bottom side of the resampled square projection plane "0" from the top side of the resampled square projection plane "3" in the projection layout 2802. The third fill region PR_de3 includes a guard band of resampled square projection plane "2" andthe guard band of resampled square projection plane "5" thus isolates the bottom side of resampled square projection "2" from the top side of resampled square projection plane "5" in projection layout 2802. Each guard band has a guard band size S _GB . Thus, the width of each of the fill areas PR_De1/PR_De2/PR_De3 is equal to 2*S _GB . For example, guard band size S _GB May be 8 pixels in width. Note that the guard band size S _GB Is adjustable. As an option, the guard band size S _GB May be 4 pixels, 16 pixels, or any number of pixels.

Fig. 28, panel (B), shows the proposed 6x1 cube layout 2804 with internal boundary filling. In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "2" is connected to the left side of the resampled square projection surface "4", there is an image content discontinuity boundary between the resampled square projection surfaces "2" and "4". According to the proposed 6x1 cube projection layout 2804 with padding, a padding region pr_de is inserted between resampled square projection planes "2" and "4". The fill region PR_DE includes a guard band for the resampled square projection plane "2" and a guard band for the resampled square projection plane "4", thus isolating the right side of the resampled square projection plane "2" from the left side of the resampled square projection plane "4" in the projection layout 2804. Each guard band has a guard band size S _GB . Therefore, the width of the filling region PR_DE is equal to 2*S _GB . For example, guard band size S _GB May be 8 pixels in width. Note that the guard band size S _GB Is adjustable. As an option, the guard band size S _GB May be 4 pixels, 16 pixels, or any number of pixels.

In a second exemplary guard band design, padding may be added at layout boundaries and discontinuous boundaries. FIG. 29 is a schematic diagram of a cube projection layout with outer boundary fill and inner boundary fill in accordance with an embodiment of the invention. Fig. 29, panel (a), shows a proposed 3x2 cube layout 2902 with outer boundary fill and inner boundary fill. If resampled square projection planes "0", "1", "2", "3", "4", and "5" are packaged in a typical 3x2 cube projection layout without padding, the top edges of the resampled square projection planes "1", "0", and "2" form a top layout boundary, the bottom edges of the resampled square projection planes "4", "3", and "5" form a bottom layout boundary, the left edges of the resampled square projection planes "1" and "4" form a left layout boundary, and the right edges of the resampled square projection planes "2" and "5" form a right layout boundary. The proposed 3x2 cube layout 2902 with external and internal boundary fills may be derived by adding external boundary fills to the proposed 3x2 cube layout 2802 with internal boundary fills. The proposed 3x2 cube layout 2902 with outer and inner boundary filling therefore has, in addition to the fill areas pr_de1, pr_de2, pr_de3 at the discontinuous boundary: top fill region pr_t connected to the top edges of resampled square projection planes "1", "0" and "2", bottom fill region pr_b connected to the bottom edges of resampled square projection planes "4", "3" and "5", left fill region pr_l connected to the left edges of resampled square projection planes "1" and "4", and right fill region pr_r connected to the right edges of resampled square projection planes "2" and "5".

The top fill region PR_T includes a guard band of resampled square projection plane "1", a guard band of resampled square projection plane "0", and a guard band of resampled square projection plane "2". The underfill region PR_B includes a guard band of resampled square projection plane "4", a guard band of resampled square projection plane "3", and a guard band of resampled square projection plane "5". The left side fill region pr_l includes a guard band of resampled square projection plane "1" and a guard band of resampled square projection plane "4". The right side fill region pr_r includes a guard band of resampled square projection plane "2" and a guard band of resampled square projection plane "5". Each guard band has a guard band size SGB. Therefore, the width of each of the outer boundary fill areas pr_t/pr_b/pr_l/pr_r is equal to SGB. For example, the width of the guard band size SGB may be 8 pixels. It should be noted that the guard band size SGB is adjustable. Alternatively, the guard band size SGB may be 4 pixels wide, 16 pixels wide, or any number of pixels.

Fig. 29, panel (B), shows a proposed 6x1 cube layout 2904 with outer boundary fill and inner boundary fill. If resampled square projection surfaces "0", "1", "2", "3", "4", and "5" are packaged in a typical 6x1 cube projection layout without padding, the top edges of the resampled square projection surfaces "1", "0", "2", "4", "3", and "5" form a top layout boundary, the bottom edges of the resampled square projection surfaces "1", "0", "2", "4", "3", and "5" form a bottom layout boundary, the left edge of the resampled square projection surface "1" forms a left side layout boundary, and the right edge of the resampled square projection surface "5" forms a right side layout boundary. The proposed 6x1 cube layout 2904 with external and internal boundary fills may be derived by adding external boundary fills to the proposed 6x1 cube layout 2804 with internal boundary fills. Thus, in addition to the fill region pr_de at the discontinuous boundary, the proposed 6x1 cube layout with outer boundary fill and inner boundary fill has: top fill region pr_t connected to the top edges of resampled square projection planes "1", "0", "2", "4", "3", and "5", bottom fill region pr_b connected to the bottom edges of resampled square projection planes "1", "0", "2", "4", "3", and "5", left fill region pr_l connected to the left edge of resampled square projection plane "1", and right fill region pr_r connected to the right edge of resampled square projection plane "5".

The top fill region PR_T includes a guard band of resampled square projection plane "1", a guard band of resampled square projection plane "0", a guard band of resampled square projection plane "2Guard bands, guard bands for resampled square projection plane "4", guard bands for resampled square projection plane "3", and guard bands for resampled square projection plane "5". The underfill region PR_B includes a guard band of resampled square projection plane "1", a guard band of resampled square projection plane "0", a guard band of resampled square projection plane "2", a guard band of resampled square projection plane "4", a guard band of resampled square projection plane "3", and a guard band of resampled square projection plane "5". The left side fill region pr_l includes a guard band of resampled square projection plane "1". The right side fill region PR R includes a guard band of resampled square projection plane "5". Each guard band has a guard band size S _GB . Therefore, the width of each of the outer boundary fill areas PR_T/PR_B/PR_L/PR_R is equal to S _GB . For example, guard band size S _GB May be 8 pixels in width. Note that the guard band size S _GB Is adjustable. As an option, the guard band size S _GB May be 4 pixels, 16 pixels, or any number of pixels.

In a third exemplary guard band design, padding may be added at layout boundaries, discontinuous boundaries and continuous boundaries. FIG. 30 is a schematic diagram of other cube projection layouts with outer boundary fill and inner boundary fill in accordance with an embodiment of the present invention. Fig. 30, subplot (a), shows another proposed 3x2 cube layout 3002 with outer boundary fill and inner boundary fill. In a typical 3x2 cube projection layout without padding, if the right side of the resampled square projection surface "1" is connected to the left side of the resampled square projection surface "0", then there is an image content continuity boundary between the resampled square projection surfaces "1" and "0". In a typical 3x2 cube projection layout without padding, if the right side of the resampled square projection surface "0" is connected to the left side of the resampled square projection surface "2", then there is an image content continuity boundary between the resampled square projection surfaces "0" and "2". In a typical 3x2 cube projection layout without padding, if the right side of the resampled square projection surface "4" is connected to the left side of the resampled square projection surface "3", then there is an image content continuity boundary between the resampled square projection surfaces "4" and "3". In a typical 3x2 cube projection layout without padding, if the right side of the resampled square projection surface "3" is connected to the left side of the resampled square projection surface "5", then there is an image content continuity boundary between the resampled square projection surfaces "3" and "5". The proposed 3x2 cube layout 3002 with outer boundary fill and inner boundary fill may be derived by adding more fill to the proposed 3x2 cube layout 2902 with outer boundary fill and inner boundary fill. Thus, in addition to the fill areas pr_de1, pr_de2, pr_de3 at the discontinuous boundary and the fill areas pr_t, pr_b, pr_l, pr_r at the layout boundary, the proposed cube layout 3002 with outer and inner boundary filling has: a fill region pr_ce1 connected to the right of the resampled square projection plane "1" and the left of the resampled square projection plane "0", a fill region pr_ce2 connected to the right of the resampled square projection plane "0" and the left of the resampled square projection plane "2", a fill region pr_ce3 connected to the right of the resampled square projection plane "4" and the left of the resampled square projection plane "3", and a fill region pr_ce4 connected to the right of the resampled square projection plane "3" and the left of the resampled square projection plane "5".

The fill region PR_CE1 includes a guard band for the resampled square projection plane "1" and a guard band for the resampled square projection plane "0", thus isolating the right side of the resampled square projection plane "1" from the left side of the resampled square projection plane "0" in the projection layout 3002. The fill region PR_CE2 includes a guard band for the resampled square projection plane "0" and a guard band for the resampled square projection plane "2", thus isolating the right side of the resampled square projection plane "0" from the left side of the resampled square projection plane "1" in the projection layout 3002. The fill region PR_CE3 includes a guard band for the resampled square projection surface "4" and a guard band for the resampled square projection surface "3", thus isolating the right side of the resampled square projection surface "4" from the left side of the resampled square projection surface "3" in the projection layout 3002. The fill region PR_CE4 includes a guard band for the resampled square projection plane "3" and a guard band for the resampled square projection plane "5", thus isolating the right side of the resampled square projection plane "3" from the left side of the resampled square projection plane "5" in the projection layout 3002. Each guard band has a guard band size SGB. Thus, the width of each of the fill areas pr_ce1/pr_ce2/pr_ce3/pr_ce4 is equal to 2×sgb. For example, the width of the guard band size SGB may be 8 pixels. It should be noted that the guard band size SGB is adjustable. Alternatively, the guard band size SGB may be 4 pixels wide, 16 pixels wide, or any number of pixels.

Fig. 30, panel (B), shows another proposed 6x1 cube layout 3004 with outer boundary fill and inner boundary fill. In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "1" is connected to the left side of the resampled square projection surface "0", then there is an image content continuity boundary between the resampled square projection surfaces "1" and "0". In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "0" is connected to the left side of the resampled square projection surface "2", then there is an image content continuity boundary between the resampled square projection surfaces "0" and "2". In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "4" is connected to the left side of the resampled square projection surface "3", then there is an image content continuity boundary between the resampled square projection surfaces "4" and "3". In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "3" is connected to the left side of the resampled square projection surface "5", then there is an image content continuity boundary between the resampled square projection surfaces "3" and "5".

The proposed 6 x 1 cube layout 3004 with outer boundary fill and inner boundary fill may be derived by adding more fill to the proposed 6 x 1 cube layout 2904 with outer boundary fill and inner boundary fill. Thus, in addition to the fill region pr_de at the discontinuous boundary and the fill regions pr_t, pr_b, pr_l, pr_r at the layout boundary, the proposed cube layout 3004 with outer boundary fill and inner boundary fill has: a fill region pr_ce1 connected to the right of the resampled square projection plane "1" and the left of the resampled square projection plane "0", a fill region pr_ce2 connected to the right of the resampled square projection plane "0" and the left of the resampled square projection plane "2", a fill region pr_ce3 connected to the right of the resampled square projection plane "4" and the left of the resampled square projection plane "3", and a fill region pr_ce4 connected to the right of the resampled square projection plane "3" and the left of the resampled square projection plane "5".

The fill region PR_CE1 includes a guard band of resampled square projection plane "1" and a guard band of resampled square projection plane "0" separating the right side of the resampled square projection plane "1" from the left side of the resampled square projection plane "0" in the projection layout 3004. The fill region PR_CE2 includes a guard band of resampled square projection plane "0" and a guard band of resampled square projection plane "2" separating the right side of the resampled square projection plane "0" from the left side of the resampled square projection plane "2" in the projection layout 3004. The fill region PR_CE3 includes a guard band of resampled square projection plane "4" and a guard band of resampled square projection plane "3" separating the right side of the resampled square projection plane "4" from the left side of the resampled square projection plane "3" in the projection layout 3004. The fill region PR_CE4 includes a guard band for the resampled square projection surface "3" and a guard band for the resampled square projection surface "5" separating the right side of the resampled square projection surface "3" from the left side of the resampled square projection surface "5" in the projection layout 3004. Each guard band has a guard band size SGB. Thus, the width of each of the fill areas pr_ce1/pr_ce2/pr_ce3/pr_ce4 is equal to 2×sgb. For example, the width of the guard band size SGB may be 8 pixels. It should be noted that the guard band size SGB is adjustable. Alternatively, the guard band size SGB may be 4 pixels wide, 16 pixels wide, or any number of pixels.

In a first exemplary fill design, fill circuit 2716 applies geometric fill to the projection surface to determine pixel values for pixels included in a fill area (e.g., one of PR_DE, PR_DE1-PR_DE3, PR_T, PR_B, PR_L, PR_R, and PR_CE1-PR_CE4) connected to the projection surface. Taking the fill region pr_t shown in sub-graph (a) of fig. 29 as an example, which includes a left geometric map region serving as a guard band of the resampled square projection plane "1", a middle geometric map region serving as a guard band of the resampled square projection plane "0", and a right geometric map region serving as a guard band of the resampled square projection plane "2". The contents of the region on the sphere (e.g., sphere 2002 shown in fig. 20) is mapped to the left geometric mapping region of the fill region pr_t, where the region on the sphere is adjacent to the region where the square projection plane "L" is obtained, and the resampled square projection plane "1" is obtained by applying the non-uniform mapping to the square projection plane "L". The contents of the region on the sphere (e.g., sphere 2002 shown in fig. 20) is mapped to an intermediate geometrically mapped region of the fill region pr_t, where the region on the sphere is adjacent to the region where the square projection plane "F" is obtained, and the resampled square projection plane "0" is obtained by applying a non-uniform mapping to the square projection plane "F". The contents of the region on the sphere (e.g., sphere 2002 shown in fig. 20) is mapped to the right geometric mapping region of the fill region pr_t, where the region on the sphere is adjacent to the region where the square projection plane "R" is obtained, and the resampled square projection plane "2" is obtained by applying a non-uniform mapping to the square projection plane "R". Thus, there is image content continuity between the resampled square projection plane "1" and the left geometric map area of the fill area pr_t, there is image content continuity between the resampled square projection plane "0" and the middle geometric map area of the fill area pr_t, and there is image content continuity between the resampled square projection plane "2" and the right geometric map area of the fill area pr_t. That is, the contents are continuously represented in several mapped areas on the left side of the resampled projection plane "1" and the fill area pr_t, the contents are continuously represented in the middle geometry mapped area of the resampled square projection plane "0" and the fill area pr_t, and the contents are continuously represented in the right geometry mapped area of the resampled square projection plane "2" and the fill area pr_t.

In a second exemplary fill design, fill circuit 2716 sets the pixel values of pixels included in a fill area (e.g., one of pr_de, pr_de1-pr_de3, pr_t, pr_b, pr_l, pr_r, and pr_ce1-pr_ce4) by copying the pixel values of pixels included in a projection plane connected to the fill area. For example, boundary pixels of the projection surface are duplicated to create filler pixels of a filler region connected to the projection surface. Taking the fill region pr_t shown in sub-picture (a) of fig. 29 as an example, it includes a left side copy region serving as a guard band of the resampled square projection plane "1", a middle copy region serving as a guard band "0" of the resampled square projection plane, and a right side copy region serving as a guard band of the resampled square projection plane "2". Since the left copy region is connected to the top side of the resampled square projection plane "1", the boundary pixels located at the top side of the resampled square projection plane "1" are directly copied to set the pixel values of the pixels of the left copy region of the fill region pr_t. Since the middle copy region is connected to the top side of the resampled square projection plane "0", the boundary pixels located at the top side of the resampled square projection plane "0" are directly copied to set the pixel values of the pixels of the middle copy region of the fill region pr_t. Since the right copy region is connected to the top side of the resampled square projection plane "2", the boundary pixels located at the top side of the resampled square projection plane "2" are directly copied to set the pixel values of the pixels of the right copy region of the fill region pr_t.

In a third exemplary fill design, fill circuit 2716 sets pixel values of pixels included in a fill area (e.g., one of pr_de, pr_de1-pr_de3, pr_t, pr_b, pr_l, pr_r, and pr_ce1-pr_ce4) of a first projection plane by copying pixel values of pixels included in a second projection plane that is not connected to the fill area. For example, the first projection surface and the second projection surface correspond to adjacent surfaces of a cube in 3D space (e.g., adjacent surfaces of cube 2004 shown in fig. 20). FIG. 31 is a schematic illustration of a fill design for creating a fill area of one projection surface by copying a partial area in another projection surface in accordance with an embodiment of the present invention. In this example, a resampled square projection plane "0" is generated by applying a non-uniform map to a square projection plane "F" shown in fig. 20, a resampled square projection plane "1" is generated by applying a non-uniform map to a square projection plane "L" shown in fig. 20, a resampled square projection plane "2" is generated by applying a non-uniform map to a square projection plane "R" shown in fig. 20, a resampled square projection plane "3" is generated by applying a non-uniform map to a square projection plane "BK" shown in fig. 20, a resampled square projection plane "4" is generated by applying a non-uniform map to a square projection plane "T" shown in fig. 20, and a resampled square projection plane "5" is generated by applying a non-uniform map to a square projection plane "B" shown in fig. 20. Furthermore, the proposed 3 x 2 cube layout 2902 with outer boundary fill and inner boundary fill is employed as shown in sub-graph (a) of fig. 29.

Based on the image content continuity feature, the fill region pr_de1 interposed between the resampled square projection planes "1" and "4" includes a replica of the partial region PK in the resampled square projection plane "5" and a replica of the partial region PA in the resampled square projection plane "1"; the fill region pr_de2 interposed between the resampled square projection planes "0" and "3" includes a replica of the partial region PM of the resampled square projection plane "5" and a replica of the partial region PD in the resampled square projection plane "1"; the fill region pr_de3 interposed between the resampled square projection planes "2" and "5" includes a replica of the partial region PP in the resampled square projection plane "5" and a replica of the partial region PE in the resampled square projection plane "1".

Further, based on the image content continuity feature, the top-fill region pr_t connected to the resampled square projection surface "1", "0", and "2" includes a replica of the partial region PI of the resampled square projection surface "4", a replica of the partial region PL in the resampled square projection surface "4", and a replica of the partial region PN in the resampled square projection surface "4"; the underfill region pr_b connected to the resampled square projection surfaces "4", "3", and "5" includes a replica of the partial region PC in the resampled square projection surface "2", a replica of the partial region PH in the resampled square projection surface "2", and a replica of the partial region PG in the resampled square projection surface "2"; the left side fill region pr_l connected to the resampled square projection planes "1" and "4" includes a replica of the partial region PJ in the resampled square projection plane "3" and a replica of the partial region PB of the resampled square projection plane "0"; the right side fill region pr_r connected to the resampled square projection planes "2" and "5" includes a replica of the partial region PO in the resampled square projection plane "3" and a replica of the partial region PF of the resampled square projection plane "0".

In addition, corner filling is performed, i.e. corner filling areas are provided around certain corners of the resampled projection surfaces "1", "2", "4" and "5". Specifically, the pixel value of each fill pixel in the corner fill region is derived by performing interpolation using the boundary pixels of the adjacent fill region. Taking corner-fill region 3102 as an example, by a boundary pixel C for replicated partial region PB _y And boundary pixels C of the copied partial region PC _x Performing interpolation to set the pixel value of the filler pixel CMiddle-fill pixel C and boundary pixel C _y With the same y-axis coordinates, fill pixel C and boundary pixel C _x With the same x-axis coordinates. Fill pixel C and boundary pixel C _x The horizontal distance between them is denoted by i. Fill pixel C and boundary pixel C _y The vertical distance between them is denoted by j. Interpolation may be represented using the following formula.

With respect to the embodiment shown in fig. 27, conversion circuit 2714 has resampling circuit 2715 and padding circuit 2716, which is set by the proposed CMP layout with padding according to a 360VR projection layout l_vr, such that the resampled square projection surface resulting from resampling circuit 2715 and the padding area resulting from padding circuit 2716 are encapsulated in a projection-based frame IMG. However, this is for illustrative purposes only and is not meant to be a limitation of the present invention. For example, the proposed CMP layout with fill may be used by the conversion circuit 114 shown in fig. 1. Thus, the fill area shown in fig. 28-31 may be created by the fill circuit 115, and the projection planes "0", "1", "2", "3", "4", "5" shown in fig. 28-31 may be square projection planes "L", "F", "R", "BK", "T", "B" shown in fig. 20. According to the 360VR projection layout l_vr, the layout is provided by the proposed CMP layout with filling, the square projection planes "L", "F", "R", "BK", "T", "B", obtained directly by cube projection without resampling, and the filling areas resulting from the filling circuit 114 are encapsulated in projection-based frames IMG.

The guard bands may be provided according to one guard band design selected from the first exemplary guard band design, the second exemplary guard band design, and the third exemplary guard band design as described above. In addition, the width of the guard band size SGB may be 4 pixels, 8 pixels, 16 pixels, or any number of pixels. And guard band information may be transmitted in the bitstream BS for further reconstruction/rendering at the decoder side.

According to the proposed syntax signaling method, the following syntax table may be employed.

/>

It should be noted that the descriptors in the above exemplary syntax table specify the parsing process of each syntax element. For example, the descriptor u (n) describes an unsigned integer (unsigned integer) using n bits.

The syntax element guard_band_width specifies the width of the guard band in units of luminance samples on the top/left/right/bottom size of each projection plane. When the decoded image has 4:2:0 or 4:2: in the 2 chroma format, the guard_band_width should be even.

The syntax element guard_band_type specifies the pixel filling method on the guard band. When a guard band surrounding the projection plane is not specified, the syntax element guard_band_type may be set to 0 (i.e., guard_band_type= 0). When deriving pixels in the guard band from copying boundary pixels of the projection plane, the syntax element guard_band_type may be set to 1 (i.e., guard_band_type= 1). When pixels in the guard band are derived from copying neighboring planes in the 3D space, the syntax element guard_band_type may be set to 2 (i.e., guard_band_type= 2). When deriving pixels in a guard band of the projection plane by applying geometric padding to the projection plane, the syntax element guard_band_type may be set to 3 (i.e., guard_band_type= 3).

As described above, the decoding circuit 122 of the target electronic device 104 receives the bit stream BS from the transmission apparatus 103 (e.g., a wired/wireless communication link or storage medium) and performs a decoder function to decode a portion of the received bit stream BS to generate a decoded frame IMG', which is a projection-based frame having the same 360VR projection layout l_vr employed by the conversion circuit 114/2714 of the source electronic device 102/2702. In the case of setting the 360VR projection layout l_vr by a projection layout with padding (e.g., a projection layout with outer boundary padding, a projection layout with inner boundary padding, or a projection layout with outer boundary padding and inner boundary padding), the decoded frame IMG' has a padded region located at the layout boundary and/or the face boundary of the projection layout. In one embodiment, the decoding circuitry 122 may crop the filled region such that only the non-filled region is reconstructed (e.g., the omnidirectional image/video content represented in a projection plane obtained from a 360VR projection, or the omnidirectional image/video content represented in a resampled projection plane derived from a 360VR projection and resampled projection). In an alternative design, the decoding circuit 122 may be modified to perform blending based on the pixels in the filled region and the pixels in the non-filled region. For example, the pixel values of the pixels in the projection plane may be updated by mixing the original pixel values of the pixels in the projection plane with the pixel values of the corresponding filler pixels in the filler region. For another example, the pixel values of the fill pixels in the fill region may be updated by mixing the original pixel values of the fill pixels in the fill region with the pixel values of the corresponding pixels in the projection plane.

Fig. 32 is a schematic diagram of a fourth 360VR system in accordance with an embodiment of the invention. The primary difference between the 360VR systems 2700 and 3200 is that the decoding circuitry 3222 of the target electronic device 3204 has mixing circuitry 3224 configured to perform mixing after encoding. Fig. 33 is a schematic diagram of a fifth 360VR system in accordance with an embodiment of the invention. The main difference between the 360VR systems 100 and 3300 is that the decoding circuit 3322 of the target electronic device 3304 has a blending circuit 3324 configured to perform blending after encoding.

The decoding circuitry 3222/3322 is arranged to decode a portion of the bitstream BS to generate a decoded frame (i.e., a decoded projection-based frame) IMG' having at least one projection surface and at least one padded region encapsulated in a 360VR projection layout l_vr (e.g., a projection layout with outer boundary padding, a projection layout with inner boundary padding, or a projection layout with outer boundary padding and inner boundary padding). During the decoding process of generating the decoded frame IMG', the decoding circuit 3222/3322 mixes the decoded pixel values obtained for the first pixels included in the projection plane and the decoded pixel values obtained for the second pixels included in the filled region by using the mixing circuit 3224/3324 to reconstruct the first pixels in the projection plane. For example, by using the third exemplary fill design described above, a fill region is generated by fill circuit 2716/115 on the encoder side (i.e., source electronic device 2702/102). The pixel values of the second pixels thus encapsulated in the fill area in the projection-based frame IMG are generated by a copy of the pixels of the partial area comprised in the projection plane. As another example, by using the first exemplary fill design described above, a fill region is generated by fill circuit 2716/115 on the encoder side (i.e., source electronic device 2702/102). The pixel value of the second pixel thus encapsulated in the fill area in the projection-based frame IMG is generated by the geometrical mapping of the pixels comprised in the projection plane. In some embodiments of the invention, hybrid 3224/3324 may employ a distance-based weighting scheme.

With respect to the mixing circuit 3224, it is possible to update the pixel values of the pixels in the projection plane obtained from the resampling processing. With respect to the hybrid circuit 3324, it may be used to update the pixel values of pixels in the projection plane that have not undergone the resampling process. Hybrid circuits 3224 and 3324 may employ the same distance-based weighting scheme. In the following description of the distance-based weighting scheme, the term "projection plane" may denote a projection plane obtained from the resampling process or a projection plane not subjected to the resampling process.

Fig. 34 is a schematic diagram of a decoder-side mixing operation according to an embodiment of the present invention. Assume that fill circuit 2716/115 employs the exemplary fill design shown in fig. 31. Thus, according to the third exemplary fill design described above, the fill area of one projection surface is obtained by copying a partial area in another projection surface. Based on the image content continuity feature, the fill area added to the left side of the projection plane "4" in fig. 34 is set by copying the partial area PB in the square projection plane "0". However, the encoding result of the partial region PB in the square projection plane "0" and the encoding result of the filler region added to the left side of the projection plane "4" are not necessarily the same. Thus, on the decoder side (e.g., target electronic device 3204/3304), decoded pixels obtained from decoding of the partial region PB in the square projection plane "0" may be mixed with decoded pixels obtained from decoding of the filler region added to the left side of the projection plane "4". If the projection surface has different widths and/or heights due to filling, the filled region needs to be resampled (e.g., interpolated filled pixels) for blending according to the ratio of the different widths and/or heights.

The reconstructed pixel value of the target pixel in the projection plane (i.e., the source pixel having the pixel value to be updated) is the updated pixel value, which can be calculated by using the following formula.

In the above formula (4), S _REC A reconstructed pixel value (updated pixel value) representing a target pixel in the projection plane (e.g., pixel a in square projection plane "0"), S representing a decoded pixel value (original pixel value) obtained for the target pixel, and T representing a decoded pixel value obtained for a corresponding filler pixel in the filler region (e.g., filler pixel a' added to the filler region to the left side of square projection plane "4"). M represents the filling width of the filling area, and N represents the distance between the target pixel and one side of the projection plane. In fig. 34, the distance between the pixel a and the top edge of the square projection surface "0" is denoted by d (n=d), and the distance between the filler pixel a 'and the left edge of the square projection surface "4" is denoted by d'. According to the third exemplary fill design described above, since the fill area of one projection plane is obtained by copying a partial area in another projection plane, in the fill area, the fill pixel a 'is located at an integer position (i.e., (x, y), where x and y are integer positions), and the value of d is equal to the value of d'.

However, if a fill area of the projection surface is obtained by applying geometric fill to the projection surface according to the aforementioned first exemplary fill design, the fill pixel a 'may be located at a non-integer position (i.e., (x, y), where x is not an integer position, and/or y is not an integer position), and the value of d may be different from the value of d'. Specifically, due to the geometric mapping, the 2D coordinates of the filler pixel a' are transformed from the 2D coordinates of the pixel a. That is, pixel a located at an integer position (i.e., (X, Y) in the square projection plane "0", where X and Y are integer positions) may be mapped to a fill pixel a' located at a non-integer position (i.e., (X, Y), where X is not an integer position, and/or Y is not an integer position) in the fill region. Since the pixel value of the filler pixel a 'located at the non-integer position is not directly available in the filler region, the mixing circuit 3224/3324 may determine the pixel value of the filler pixel a' located at the non-integer position in the filler region by processing the filler pixel located at the integer position in the filler region using an interpolation filter. After determining the pixel value of the filler pixel a' at the non-integer position, the updated pixel value of pixel a in square projection plane "0" is calculated using equation (4) above.

In the above formula (4), N represents the distance between the target pixel and one side of the projection surface. In a first exemplary design, N is set by a positive integer value. For example, n=i+1, where i is an index (distance) counted from one side of the projection surface, and 0.ltoreq.i < M. Fig. 35 is a schematic diagram of a relationship between a weight value of a pixel and an index value of the pixel, which is involved in updating a pixel value of the pixel in a projection plane, according to an embodiment of the present invention. Let m=4 and n=i+1. The above formula (4) can be rewritten as follows.

In the above formula (5), A _i Representing decoded pixel values obtained for a target pixel having an index i in the projection plane, A _i,updated Reconstructed pixel value (updated pixel value) representing target pixel in projection plane, A' _i Representing decoded pixel values obtained for corresponding filler pixels in the filler region. As shown in fig. 35, the target imageElement A ₃ (i.e., a of i=3 _i ) Is equal to 8 (i.e., 4+3+1), and the corresponding filler pixel a' ₃ (i.e., a 'of i=3' _i ) The weight of (2) is equal to 0 (i.e., 4-3-1); target pixel A ₂ (i.e., i=2 a _i ) Is equal to 7 (i.e. 4+2+ 1) and the corresponding filler pixel a' ₂ (i.e., a 'of i=2' _i ) The weight of (2) is equal to 1 (i.e., 4-2-1); target pixel A ₁ (i.e., i=1 a _i ) Is equal to 6 (i.e. 4+ 1) and the corresponding fill pixel a' ₁ (i.e., a 'of i=1' _i ) The weight of (2) (i.e., 4-1-1); target pixel a ₀ (i.e., i=0 a _i ) Is equal to 5 (i.e., 4+0+ 1) and fills in pixel a' ₀ (i.e., a 'where i=0' _i ) The weight of (2) is equal to 3 (i.e., 4-0-1).

As shown in fig. 35, since the difference between adjacent weight values "5" and "3" is equal to 2, the weight does not decrease from 8 to 0 in constant steps (constant step size) with the index (distance). In order to solve this problem, the present invention proposes another setting of the distance between the target pixel and one side of the projection surface. In a second exemplary design, N is set by a positive non-integer value. For example, n=i+0.5, where i is an index (distance) counted from one side of the projection surface, and 0.ltoreq.i < M. Fig. 36 is a schematic diagram of another relationship between a weight value of a pixel and an index value of the pixel, which is involved in updating a pixel value of the pixel in the projection plane, according to an embodiment of the present invention. Let m=4 and n=i+0.5. The above formula (4) can be rewritten as follows.

In the above formula (6), A _i Representing decoded pixel values obtained for a target pixel having an index i in the projection plane, A _i,updated Reconstructed pixel value (updated pixel value) representing target pixel in projection plane, A' _i Representing decoded pixel values obtained for corresponding filler pixels in the filler region. As shown in fig. 36, a target pixel a ₃ (i.e., a of i=3 _i ) The weight of (c) is equal to 7.5 (i.e., 4+3+ 0.5),and corresponding fill pixel A' ₃ (i.e., a 'with i=3' _i ) The weight of (a) is equal to 0.5 (i.e., 4-3-0.5); target pixel A ₂ (i.e., i=2 a _i ) Is equal to 6.5 (i.e. 4+2+ 0.5) and the corresponding fill pixel a' ₂ (i.e., a 'of i=2' _i ) The weight of (a) is equal to 1.5 (i.e., 4-2-0.5); target pixel A ₁ (i.e., i=1 a _i ) Is equal to 5.5 (i.e. 4+1+ 0.5) and the corresponding fill pixel a' ₁ (i.e., a 'of i=1' _i ) The weight of (2) is equal to 2.5 (i.e., 4-1-0.5); target pixel a ₀ (i.e., i=0 a _i ) Is equal to 4.5 (i.e. 4+0+ 0.5) and fills in pixel a' ₀ (i.e., a 'where i=0' _i ) The weight of (a) is equal to 3.5 (i.e., 4-0-0.5). The weight decreases from 7.5 to 0.5 in constant steps.

For some applications, the conversion circuitry may be implemented in the target electronic device to convert a decoded frame of a projection layout having a first 360VR projection format to a converted frame of a projection layout having a second 360VR projection format different from the first 360VR projection format. For example, the decoded frame generated from the decoding circuit may be a projection-based frame having a projection surface and a fill region encapsulated in a cube projection layout having fill; and the converted frames (converted frames) generated by the conversion circuitry and used by the subsequent graphics rendering circuitry may be projection-based frames having projection surfaces encapsulated in a typical equidistant rectangular projection layout without padding. Pixels located at integer positions (i.e., (x, y), where x and y are integer positions) in the transformed frame may be mapped to pixels located at non-integer positions (i.e., (x ', y'), where x 'is not an integer position and/or y' is not an integer position) in the decoded frame. That is, when performing projection layout conversion, the conversion circuit may map pixel values of pixels located at integer positions in the converted frame to pixel values of pixels located at non-integer positions in the decoded frame. Since the pixel values of the pixels located at the non-integer positions are not directly available in the decoded frame, the conversion circuit may determine the pixel values of the pixels located at the non-integer positions in the decoded frame by processing the pixels located at the integer positions in the decoded frame using an interpolation filter. In the case where the pixel having the non-integer position is located at the layout boundary of the projection layout in the decoded frame or near the discontinuous boundary, the pixel used by the interpolation filter may include at least one pixel selected from the projection plane and at least one pixel selected from the corresponding fill region. The pixel values of the pixels in the projection surface are then updated by blending (e.g., distance-based weights). However, if the pixel values of the filled pixels in the respective filled regions are not updated by blending (e.g., distance-based weights). As interpolation is performed using updated pixel values of pixels in the projection plane and original (non-updated) pixel values of filled pixels in the corresponding filled region, artifacts may be introduced as a result. To solve this problem, the present invention proposes another blending scheme of updating pixel values by blending both pixels in the projection plane and filler pixels in the filler region.

Fig. 37 is a schematic diagram of a sixth 360VR system in accordance with an embodiment of the invention. The main difference between the 360VR systems 3200 and 3700 is: the blending circuit 3724 in the decoding circuit 3722 of the target electronic device 3704 is arranged to update pixel values by blending for both pixels in the projection plane and for filler pixels in the filler region, and the target electronic device 3704 further comprises a conversion circuit 3726 arranged to convert a decoded frame (i.e. decoded projection-based frame) IMG 'having one 360VR projection layout into a converted frame (i.e. converted projection-based frame) IMG' having a different 360VR projection layout. In one embodiment of the invention, the 360VR projection layout l_vr may be a CMP layout with padding and the converted 360VR projection layout may be an ERP layout without padding. However, this is for illustrative purposes only and is not meant to be a limitation of the present invention.

Fig. 38 is a schematic diagram of a seventh 360VR system in accordance with an embodiment of the invention. The main difference between the 360VR systems 3300 and 3800 is that: the blending circuit 3824 in the decoding circuit 3822 of the target electronic device 3804 is arranged to update pixel values by blending for both pixels in the projection surface and for filler pixels in the filler region, and the target electronic device 3804 further comprises a conversion circuit 3826, the conversion circuit 3826 being arranged to convert a decoded frame IMG' having one 360VR projection layout into a converted frame IMG having a different 360VR projection layout. In one embodiment of the invention, the 360VR projection layout L_VR may be a CMP layout and the converted 360VR projection layout may be an ERP layout. However, this is for illustrative purposes only and is not meant to be a limitation of the present invention.

With respect to the embodiments shown in fig. 37 and 38, the 360VR projection layout l_vr used based on the projected frame IMG and decoded frame IMG' may be a projection layout of one 360VR projection format that is different from another 360VR projection format associated with the converted frame IMG ". For example, the different projection layouts used for decoding frame IMG' and converting frame img″ may be selected from a set of projection layouts including ERP layouts, a plurality of cube-based projection layouts (e.g., CMP layouts, pyramid projection layouts, truncated square pyramid projection layouts, and view-port-based cube projection layouts), a plurality of triangle-based projection layouts (e.g., octahedral projection layouts, icosahedral projection layouts, tetrahedral projection layouts, quadrilateral quartz-based projection layouts, and hexagonal quartz-based projection layouts), segment sphere projection (segmented sphere projection, SSP) layouts, equatorial cylinder projection layouts, rotary sphere projection layouts, and so forth.

In some embodiments of the invention, the hybrid circuits 3724/3824 may employ a distance-based weighting scheme. Regarding the mixing circuit 3724, it can be used to update the pixel values of pixels in the projection plane obtained from the resampling process, and can also be used to update the pixel values of the filled pixels in the filled region. Regarding the hybrid circuit 3824, it can be used to update the pixel values of pixels in the projection plane that have not undergone the resampling process, and can also be used to update the pixel values of the fill pixels in the fill area. Hybrid circuits 3724 and 3824 may employ the same distance-based weighting scheme. In the following description of the distance-based weighting scheme, the term "projection plane" may denote a projection plane obtained from the resampling process or a projection plane not subjected to the resampling process.

Please refer to fig. 34 again. Assume that fill circuit 2716/115 employs the exemplary fill design shown in fig. 31. Thus, according to the third exemplary fill design described above, the fill area of one projection surface is obtained by copying a partial area in another projection surface. Based on the image content continuity feature, the fill area added to the left of the projection plane "4" is set by a replica of the partial area PB in the square projection plane "0". However, the encoding result of the partial region PB in the square projection plane "0" and the encoding result of the filler region added to the left of the projection plane "4" are not necessarily the same. Thus, on the decoder side (e.g., target electronics 3704/3804), decoded pixels obtained from decoding of the partial region PB in the square projection plane "0" may be mixed with decoded pixels obtained from decoding of the filler region added to the left of the projection plane "4". In the present embodiment, the mixing circuit 3724/3824 is configured to update the pixel values of the pixels in the partial region PB in the square projection plane "0" by mixing the original pixel values of the pixels in the partial region PB in the square projection plane "0" with the original pixel values of the filler pixels added to the left filler region in the projection plane "4"; and the mixing circuit 3724/3824 is also for updating the pixel values of the filler pixels added to the left filler region of the projection plane "4" by mixing the original pixel values of the filler pixels added to the left filler region of the projection plane "4" with the original pixel values of the pixels in the partial region PB of the square projection plane "0". If the projection surface has different widths and/or heights due to filling, resampling of the filled region (e.g., interpolation of the filled pixels) for blending is required according to the ratio of the different widths and/or heights.

The reconstructed pixel value of the target pixel (i.e., the source pixel having the pixel value to be updated) in any one of the projection surface and the fill area may be calculated by using the following formula.

In the above formula (7), S' _REC Representing the target pixel (e.g., square throw as shown in fig. 34Pixel a in the image plane "0", or a reconstructed pixel value (updated pixel value) added to a filled pixel a ' in the filled region to the left of the square projection plane "4" shown in fig. 34, S ' represents a decoded pixel value (original pixel value) obtained for the target pixel, and T represents a decoded pixel value obtained for a corresponding pixel to be mixed with the target pixel (for example, a filled pixel a ' added to the filled region to the left of the square projection plane "4" shown in fig. 34, or a pixel a in the square projection plane "0" shown in fig. 34); m represents the filling width of the filling area, and N' represents the distance between the target pixel and one side of the projection surface. In the case where the target pixel is the pixel a in the square projection plane "0" shown in fig. 34, N' is set by a value indicating the distance between the pixel a and the top edge of the square projection plane "0". In another case where the target pixel is a fill pixel a ' added to the fill region on the left side of the square projection plane "4" shown in fig. 34, N ' is set by a value representing the distance between the fill pixel a ' and the left side of the square projection plane "4".

In fig. 34, the distance between the pixel a and the top edge of the square projection plane "0" is denoted by d, and the distance between the filler pixel a 'and the left edge of the square projection plane "4" is denoted by d'. According to the third exemplary fill design described above, since the fill area of one projection plane is obtained by copying a partial area in another projection plane, the fill pixel a 'in the fill area is located at an integer position (i.e., (x, y), where x and y are integer positions), and the value of d is equal to the value of d'.

However, according to the aforementioned first exemplary fill design, if the fill area of the projection surface is obtained by applying geometric fill to the projection surface, if the target pixel (having the pixel value to be updated) is located at an integer position (i.e., (x, y), where x and y are integer positions), the non-target pixel (which will be the corresponding pixel mixed with the target pixel) may be located at a non-integer position (i.e., (x ', y'), where x 'is not an integer position, and/or y' is not an integer position). In the case where the target pixel is pixel a in the square projection plane "0", the non-target pixel is a fill pixel a ', which may be located at a non-integer position in the fill area, so d may be different from the value of d'. Specifically, due to the geometric mapping, the 2D coordinates of the filler pixel a' are transformed from the 2D coordinates of the pixel a. That is, pixel a located at an integer position (i.e., (x, y) where x and y are integer positions) in the square projection plane "0" may be mapped to a filler pixel a ' located at a non-integer position (i.e., (x ', y ') where x ' is not an integer position and/or y ' is not an integer position) in the filler region. Since the pixel value of the filler pixel a 'located at the non-integer position is not directly available in the filler region, the mixing circuit 3724/3824 can determine the pixel value of the filler pixel a' located at the non-integer position in the filler region by processing the filler pixel located at the integer position in the filler region using the interpolation filter. After determining the pixel value of the filler pixel a' at the non-integer position, the updated pixel value of pixel a is calculated using equation (7) above.

In another case where the target pixel is a fill pixel a 'added to the fill area to the left of the square projection plane "4", the non-target pixel may be a pixel a located at a non-integer position (i.e., (x', y '), where x' is not an integer position, and/or y 'is not an integer position) in the square projection plane "0", and thus the value of d may be different from the value of d'. Specifically, due to the geometric mapping, the 2D coordinates of pixel a are transformed from the 2D coordinates of the fill pixel a'. That is, a fill pixel a ' located at an integer position (i.e., (x, y), where x and y are integer positions) in the fill area may be mapped to a pixel a located at a non-integer position (i.e., (x ', y '), where x ' is not an integer position, and/or y ' is not an integer position) in the square projection plane "0". Since the pixel value of the pixel a located at the non-integer position is not directly available in the square projection plane "0", the mixing circuit 3724/3824 can determine the pixel value of the pixel a located at the non-integer position in the square projection plane "0" by processing the pixel located at the integer position in the square projection plane "0" using the interpolation filter. After determining the pixel value of pixel a at a non-integer position, the updated pixel value of the filler pixel a' is calculated using equation (7) above.

In the above formula (7), N' represents a distance between a target pixel and one side of the projection plane, wherein the target pixel may be a pixel in the projection plane or a filler pixel in the filler region. In a first exemplary design, N' is set by a non-integer value. For example, N' =i+0.5, where i is an index (distance) counted from one side of the projection surface. In the present embodiment, when the target pixel is a filler pixel in a filler region located outside the projection plane, i is set to a negative integer value; and when the target pixel is a pixel included in the projection plane, i is set to a non-negative integer value. Fig. 39 is a schematic diagram of a relationship between a weight value of a pixel and an index value of the pixel, which are involved in updating a pixel value of a pixel in a projection plane and a pixel value of a filling pixel in a filling area, according to an embodiment of the present invention. Let m=4 and n=i+0.5. The above formula (7) can be rewritten as follows.

In the above formula (8), A _i A decoded pixel value, a, represented as a target pixel having index i _i,updated A reconstructed pixel value (updated pixel value) representing the target pixel, and A' _i Representing decoded pixel values obtained for corresponding non-target pixels. As shown in fig. 39, a target pixel A3 (i.e., a of i=3 in the projection plane _i ) Is equal to 7.5 (i.e., 4+3+0.5), and fills the corresponding non-target pixel a 'in the region' ₃ (i.e., a 'of i=3' _i ) The weight of (a) is equal to 0.5 (i.e., 4-3-0.5); target pixel A in projection plane ₂ (i.e., i=2 a _i ) Is equal to 6.5 (i.e. 4+2+ 0.5) and fills the corresponding non-target pixel a 'in the area' ₂ (i.e., a 'of i=2' _i ) The weight of (a) is equal to 1.5 (i.e., 4-2-0.5); target pixel A in projection plane ₁ (i.e., ai with i=1) is equal to 5.5 (i.e., 4+1+0.5), and the corresponding non-target pixel a 'in the area is filled' ₁ (i.e., a 'of i=1' _i ) The weight of (2) is equal to 2.5 (i.e., 4-1-0.5); target pixel a in projection plane ₀ (i.e., i=0 a _i ) Is equal to 4.5 (i.e., 4+0+0.5), andcorresponding non-target pixels A 'in the filled region' ₀ (i.e., a 'where i=0' _i ) The weight of (a) is equal to 3.5 (i.e., 4-0-0.5).

Further, as shown in fig. 39, the target pixel a in the filled region _-1 (i.e., i= -1A _i ) Is equal to 3.5 (i.e., 4-1+ 0.5) and the corresponding non-target pixel a 'in the projection plane' _-1 (i.e., a 'with i = -1' _i ) The weight of (a) is equal to 4.5 (i.e., 4+1-0.5); the target pixel a-2 in the filled region (i.e., a of i= -2 _i ) Is equal to 2.5 (i.e., 4-2+ 0.5) and the corresponding non-target pixel a 'in the projection plane' _-2 (i.e., a 'of i = -2' _i ) The weight of (2) is equal to 5.5 (i.e., 4+2-0.5); target pixel a in the fill region _-3 (i.e., a of i= -3 _i ) Is equal to 1.5 (i.e., 4-3+ 0.5) and the corresponding non-target pixel a 'in the projection plane' _-3 (i.e., a 'of i = -3' _i ) The weight of (2) is equal to 6.5 (i.e., 4+3-0.5); target pixel a in the fill region _-4 (i.e., a of i= -4 _i ) Is equal to 0.5 (i.e., 4-4+ 0.5) and the corresponding non-target pixel a 'in the projection plane' _-4 (i.e., a 'of i = -4' _i ) The weight of (2) is equal to 7.5 (i.e. 4+ 4-0.5).

In a second exemplary design, N' is set by an integer value. For example, N' =i+1, where i is an index (distance) counted from one side of the projection surface. In this embodiment, i is set to a negative integer value when the target pixel is a fill pixel in the fill region, and is set to a non-negative integer value when the target pixel is a pixel in the projection plane. Fig. 40 is a schematic diagram of another relationship between a weight value of a pixel and an index value of the pixel, which is involved in updating a pixel value of a pixel in a projection plane and a pixel value of a filling pixel in a filling area, according to an embodiment of the present invention. Let m=4 and N' =i+1. The above formula (7) can be rewritten as follows.

In the above formula (9), A _i Represented as a target image with index iThe decoded pixel value obtained by the pixel, ai, updated represents the reconstructed pixel value (updated pixel value) of the target pixel, and A' _i Representing decoded pixel values obtained for corresponding non-target pixels. As shown in fig. 40, a target pixel a in the projection plane ₃ (i.e., a of i=3 _i ) Is equal to 8 (i.e., 4+3+1), and fills the corresponding non-target pixel a 'in the region' ₃ (i.e., a 'of i=3' _i ) The weight of (2) is equal to 0 (i.e., 4-3-1); target pixel a in projection plane ₂ (i.e., i=2 a _i ) Is equal to 7 (i.e., 4+2+ 1) and fills the corresponding non-target pixel a 'in the region' ₂ (i.e., a 'of i=2' _i ) The weight of (2) is equal to 1 (i.e., 4-2-1); target pixel a in projection plane ₁ (i.e., i=1 a _i ) Is equal to 6 (i.e., 4+ 1) and fills the corresponding non-target pixel a 'in the region' ₁ (i.e., a 'of i=1' _i ) The weight of (2) (i.e., 4-1-1); target pixel a in projection plane ₀ (i.e., i=0 a _i ) Is equal to 5 (i.e., 4+0+1), and fills the corresponding non-target pixel a 'in the region' ₀ (i.e., a 'where i=0' _i ) The weight of (2) is equal to 3 (i.e., 4-0-1).

Further, as shown in fig. 40, the target pixel a in the filled region _-1 (i.e., i= -1A _i ) Is equal to 4 (i.e., 4-1+ 1), and the corresponding non-target pixel a 'in the projection plane' _-1 (i.e., a 'of i = -1' _i ) The weight of (2) is equal to 4 (i.e., 4+1-1); target pixel a in the fill region _-2 (i.e., a with i= -2 _i ) Is equal to 3 (i.e., 4-2+ 1) and the corresponding non-target pixel a 'in the projection plane' _-2 (i.e., a 'with i = -2' _i ) The weight of (2) is equal to 5 (i.e., 4+2-1); target pixel a in the fill region _-3 (i.e., a of i= -3 _i ) Is equal to 2 (i.e., 4-3+ 1) and the corresponding non-target pixel a 'in the projection plane' _-3 (i.e., a 'of i = -3' _i ) The weight of (2) is equal to 6 (i.e., 4+3-1); target pixel a in the fill region _-4 (i.e., a of i= -4 _i ) Is equal to 1 (i.e., 4-4+ 1) and the corresponding non-target pixel a 'in the projection plane' _-4 (i.e., a 'of i = -4' _i ) The weight of (2) is equal to 7 (i.e., 4+4-1).

Those skilled in the art will readily observe that numerous modifications and alterations of the apparatus and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the scope and metes of the appended claims.

Claims

1. A video processing method, comprising:

obtaining a plurality of projection surfaces from the omnidirectional content of the sphere according to the cube map projection;

Generating at least one fill area by a fill circuit;

generating a projection-based frame by encapsulating the plurality of projection facets and the at least one fill region in a projection layout of the cube projection, wherein the plurality of projection facets encapsulated in the projection layout include a first projection facet; the at least one fill area encapsulated in the projection layout includes a first fill area; the first filling area is connected with at least the first projection surface, and forms at least a part of a boundary of the projection layout; and

the projection-based frame is encoded to generate a portion of a bitstream.

2. The video processing method of claim 1, wherein the at least one fill area encapsulated in the projection layout further comprises a second fill area, a third fill area, and a fourth fill area, and the first fill area, the second fill area, the third fill area, and the fourth fill area forming four boundaries of the projection layout, respectively.

3. The video processing method of claim 1, wherein the plurality of projection surfaces further comprises a second projection surface; if one side of the first projection surface is connected with one side of the second projection surface, an image content discontinuous boundary exists between the one side of the first projection surface and the one side of the second projection surface; the at least one fill region further comprises a second fill region; the second filling area is connected with the one side of the first projection surface and the one side of the second projection surface and is used for isolating the one side of the first projection surface from the one side of the second projection surface in the projection layout.

4. The video processing method of claim 1, wherein the plurality of projection surfaces further comprises a second projection surface; if one side of the first projection surface is connected with one side of the second projection surface, an image content continuity boundary exists between the one side of the first projection surface and the one side of the second projection surface; the at least one fill region further comprises a second fill region; the second filling area is connected with the one side of the first projection surface and the one side of the second projection surface and is used for isolating the one side of the first projection surface from the one side of the second projection surface in the projection layout.

5. The video processing method of claim 1, wherein the step of generating the at least one fill area comprises:

a geometric fill is applied to the first projection surface to determine pixel values for pixels included in the first fill region.

6. The video processing method of claim 1, wherein the step of generating the at least one fill area comprises:

the pixel values of the pixels included in the first filling area are set by copying the pixel values of the specific pixels included in the first projection plane.

7. The video processing method of claim 1, wherein the plurality of projection surfaces encapsulated in the projection layout further comprises a second projection surface not connected to the first fill region, and the step of generating the at least one fill region comprises:

the pixel values of the pixels included in the first filling area are set by copying the pixel values of the specific pixels included in the second projection plane.

8. The video processing method of claim 1, wherein the fill width of the first fill area is 4 pixels.

9. A video processing method, comprising:

receiving a portion of a bitstream; and

decoding a portion of the bitstream to generate a decoded projection-based frame having at least one projection surface and at least one fill region encapsulated in a projection layout of a 360 degree virtual reality projection, the steps comprising:

reconstructing the first pixels included in the at least one fill region by mixing decoded pixel values obtained by the first pixels included in the at least one fill region and decoded pixel values obtained by the second pixels included in the at least one projection plane.

10. The video processing method of claim 9, wherein the first pixel is included in a first fill region of the at least one fill region and the second pixel is included in a first projection plane of the at least one projection plane; a portion of the bitstream is generated by encoding a projection-based frame, wherein the projection-based frame includes the at least one projection surface and the at least one fill region encapsulated in the projection layout of the 360 degree virtual reality projection; the pixel values of the pixels included in the first fill area encapsulated in the projection-based frame are generated by copying the pixel values of the pixels included in the first projection plane.

11. The video processing method of claim 9, wherein the first pixel is included in a first fill region of the at least one fill region and the second pixel is included in a first projection plane of the at least one projection plane; a portion of the bitstream is generated by encoding a projection-based frame, wherein the projection-based frame includes the at least one projection surface and the at least one fill region encapsulated in the projection layout of the 360 degree virtual reality projection; by applying geometric padding to the first projection surface, pixel values of pixels included in the first padding region encapsulated in the projection-based frame are generated.

12. The video processing method of claim 9, wherein the updated pixel value of the first pixel is calculated by using the formula:

wherein S' _REC An updated pixel value representing the first pixel, S 'representing a decoded pixel value obtained for the first pixel, T' representing a decoded pixel value obtained for the second pixel, M representing a fill width of a first fill area, and N representing a distance between the first pixel and a side of a projection surface connected to the first fill area.

13. The video processing method according to claim 12, wherein n=i+0.5, i is an index of a first pixel counted from one side of the projection surface connected to the first filling area, and i is a negative integer not smaller than-M.

14. The video processing method of claim 9, wherein the projection layout is a cube projection layout with padding.