WO2018117587A1

WO2018117587A1 - Method and system for producing 360 degree content on rectangular projection in electronic device

Info

Publication number: WO2018117587A1
Application number: PCT/KR2017/015004
Authority: WO
Inventors: Sri Nitchith AKULA; Amith DSOUZA; Anubhav Singh; Ramkumaar Kovil KANTHADAI; Vladyslav ZAKHARCHENKO
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2016-12-19
Filing date: 2017-12-19
Publication date: 2018-06-28
Also published as: CN110115026A; CN110115026B

Abstract

Accordingly embodiments herein disclose a method for producing a 360 degree content on a rectangular projection in an electronic device. The method includes generating an icosahedron projection having vertices and triangles in equal shape. Each of the triangles represent a segment of the 360 degree content. Further, the method includes forming a rectangular image frame displaying the 360 degree content by rearranging the triangles. The triangles are rearranged to represent a maximum continuity of the 360 degree content in the rectangular image frame. Further, the method includes storing the rectangular image frame in the electronic device.

Description

METHOD AND SYSTEM FOR PRODUCING 360 DEGREE CONTENT ON RECTANGULAR PROJECTION IN ELECTRONIC DEVICE

The present disclosure relates to a content processing system, and more specifically related to a method and system for producing a 360 degree content (e.g., 360 degree image content, 360 degree video content, 360 degree multimedia content or the like) on a rectangular projection in an electronic device. The present application is based on, and claims priority from an Indian Application Number 201641043297 filed on 19th December, 2016 the disclosure of which is hereby incorporated by reference herein.

Virtual Reality (VR) based entertainment/gaming is a consumer application that is increasing exponentially in popularity. Typically, VR videos/games are played in high resolution (i.e., 4K) and the VR videos/games are processed on an electronic device (e.g., smart phone, VR viewer or the like). Thus, a network bandwidth and processing power/battery life of the electronic device are two major problems faced while implementing a VR pipeline. A 360 degree video that is captured by a camera associated with the electronic device is in an equi-rectangular Projection (ERP) format which has high redundancy resulting in extremely high bitrates (~40Mbps) and file sizes (~4GB for 10 minutes). Different projection methods are used to represent the 360 degree video(s) before encoding. Icosahedron Projection (ISP) is one of the projection formats used to represent the 360 degree videos.

The various projection formats which are known to represent the 360 degree video content include the equi-rectangular format, an Icosahedron format, an octahedral format and a cube mapping format. Many conventional designs are proposed for producing the 360 degree video on a rectangular projection in the electronic device, the conventional methods, however, includes advantages and disadvantages in terms of file large file size(s), discontinuities, compression efficiency, quality of encoded bit-stream, reliability, cost, complexity, hardware components used, size and so on.

Thus, it is desired to address the above mentioned disadvantages or other shortcomings or at least provide a useful alternative.

The principal object of the embodiments herein is to provide a method and system for producing a 360 degree content on a rectangular projection in an electronic device.

Another object of the embodiment herein is to generate an icosahedron projection including vertices and triangles in equal shape.

Another object of the embodiment herein is to identify and merge continuous segments of the 360 degree content in a top pole of the icosahedron projection.

Another object of the embodiment herein is to identify and merge continuous segments of the 360 degree content in a bottom pole of the icosahedron projection.

Another object of the embodiment herein is to arrange a segment of the top pole and a segment of the bottom pole between an equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve a continuity in the segment of the 360 degree content.

Another object of the embodiment herein is to arrange the segments of the top pole and the segments of the bottom pole such a way that discontinuities in the segment of the 360 degree content coincide with boundaries of a coding unit.

Another object of the embodiment herein is to form a rectangular image frame displaying the 360 degree content by re-arrangement of triangles.

Another object of the embodiment herein is to present a continuous display of a set of segment of the 360 degree content in the rectangular image frame based on the re-arranged triangles.

Another object of the embodiment herein is to introduce a margin to reduce the discontinuity in the 360 degree content when at least two triangles are not neighbouring on the icosahedron projection but put next to each other on a 2D rectangular image frame.

Another object of the embodiment herein is to provide a continuous display of the set of segments of the 360 degree content in the rectangular image frame to reduce a bit-rate of the 360 degree content.

Another object of the embodiment herein is to apply a filter to reduce an effect of the discontinuities among the rearranged triangles.

Another object of the embodiment herein is to apply a padding model to reduce an effect of the discontinuities among the rearranged triangles.

Accordingly embodiments herein disclose a method for producing a 360 degree content on a rectangular projection in an electronic device. The method includes generating an icosahedron projection comprising vertices and triangles in equal shape. Each of the triangles represents the segment of 360 degree content. Further, the method includes forming a rectangular image frame displaying the 360 degree content by re-arranging the triangles. The triangles are rearranged to represent a maximum continuity of the 360 degree content in the rectangular image frame. Furthermore, the method includes storing the rectangular image frame in the electronic device.

Accordingly embodiments herein disclose an electronic device for producing a 360 degree content on a rectangular projection. The electronic device includes a 360 degree content controller coupled to a memory and a processor. The 360 degree content controller is configured to generate an icosahedron projection including vertices and triangles in equal shape, where each of the triangles represent the segment of 360 degree content. The 360 degree content controller is configured to form a rectangular image frame displaying the 360 degree content by re-arranging the triangles. The triangles are rearranged to represent a maximum continuity of the 360 degree content in the rectangular image frame. The 360 degree content controller is configured to store the rectangular image frame in the electronic device.

This method is illustrated in the accompanying drawings, throughout which like reference letters indicate corresponding parts in the various figures. The embodiments herein will be better understood from the following description with reference to the drawings, in which:

FIG. 1 is a block diagram of an electronic device for producing a 360 degree content on a rectangular projection, according to an embodiment as disclosed herein;

FIG. 2 is a block diagram of a 360 degree content controller of the electronic device, according to an embodiment as disclosed herein;

FIG. 3 is a flow diagram illustrating a method for producing the 360 degree content on the rectangular projection in the electronic device, according to an embodiment as disclosed herein;

FIG. 4 is a flow diagram illustrating a method for forming a rectangular image frame displaying the 360 degree content by re-arranging triangles while producing the 360 degree content on the rectangular projection in the electronic device, according to an embodiment as disclosed herein;

FIGS. 5a-5d are example illustrations in which a ISP frame packing is explained from a non-frame packed ISP, according to an embodiment as disclosed herein;

FIG. 6a is an example illustration in which the 360 degree content is displayed in an Icosahedron projection format, according to prior art;

FIG. 6b is an example illustration in which the 360 degree content is displayed in an Icosahedron projection format, according to an embodiment as disclosed herein;

FIG. 7 is an example illustration in which the 360 degree content is displayed based on the current and existing methods;

FIGS. 8a-8j are example illustrations in which the 360 degree content is displayed on the rectangular projection, according to an embodiment as disclosed herein;

FIG. 9 is an example illustration in which comparison of slant edges in the Icosahedron projection format between existing method and the proposed method;

FIG. 10 is an example illustration in which a bleeding effect across discontinuity in the input image and reconstructed image is explained, according to an embodiment as disclosed herein; and

FIG. 11 is an example illustration in which a compact ISP layout is explained, according to an embodiment as disclosed herein.

In an embodiment, the icosahedron projection is generated by mapping points of the 360 degree content to a surface of the icosahedron projection.

In an embodiment, the icosahedron projection includes 12 vertices and 20 faces, where each of the faces is the triangle in equal shape and represents a face of the icosahedron projection.

In an embodiment, forming the rectangular image frame displaying the 360 degree content by re-arranging the triangles includes identifying and merging continuous segments of the 360 degree content in the top pole of the icosahedron projection, identifying and merging continuous segments of 360 degree content in the bottom pole of the icosahedron projection, dividing the middle segment of the icosahedron projection from the center into two equal segments, and arranging a segment of the top pole and a segment of the bottom pole between the equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve a continuity in the segment2D representation of the 360 degree content.

In an embodiment, the icosahedron projection is divided into the two segments vertically.

In an embodiment, the segment of the top pole and the segment of the bottom pole are arranged in such a way that most of the discontinuities in a segment of the 360 degree content coincide with boundaries of a coding unit.

In an embodiment, discontinuities in the segment of the 360 degree content is at least one of horizontal discontinuities, vertical discontinuities and angular discontinuities.

In an embodiment, a margin is introduced in order to resolve the discontinuity in the 360 degree content when at least two triangles are not neighbouring on the icosahedron projection but put next to each other on the rectangular image frame.

In an embodiment, the continuous display of the set of segments of the 360 degree content in the rectangular image frame reduces a bit-rate of the 360 degree content.

These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating preferred embodiments and numerous specific details thereof, are given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit thereof, and the embodiments herein include all such modifications.

The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. The term “or” as used herein, refers to a non-exclusive or, unless otherwise indicated. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein can be practiced and to further enable those skilled in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

As is traditional in the field, embodiments may be described and illustrated in terms of blocks which carry out a described function or functions. These blocks, which may be referred to herein as units or modules or the like, are physically implemented by analog or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware and software. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like. The circuits constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block. Each block of the embodiments may be physically separated into two or more interacting and discrete blocks without departing from the scope of the invention. Likewise, the blocks of the embodiments may be physically combined into more complex blocks without departing from the scope of the invention

The accompanying drawings are used to help easily understand various technical features and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. As such, the present disclosure should be construed to extend to any alterations, equivalents and substitutes in addition to those which are particularly set out in the accompanying drawings. Although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are generally only used to distinguish one element from another.

The terms, the rearranging and reshaping are used interchangeably in the patent document.

Accordingly embodiments herein achieve a method for producing a 360 degree content on a rectangular projection in an electronic device. The method includes generating an icosahedron projection having vertices and triangles in equal shape. Each of the triangles represent a segment of the 360 degree content. Further, the method includes forming a rectangular image frame displaying the 360 degree content by re-arranging the triangles. The triangles are rearranged to represent a maximum continuity of the 360 degree content in the rectangular image frame (e.g., 2D rectangular image frame or the like). Further, the method includes storing the rectangular image frame in the electronic device.

Unlike conventional systems and methods, the proposed method can be used to achieve a reduced bit rate, a reduced file size, better image quality and an enhanced coding efficiency, while producing the 360 degree content on the rectangular projection in the electronic device. The reduced bit rate can be achieved due to proposed rearranged features (i.e., a segment of a top pole and a segment of the bottom pole of the icosahedron projection are arranged between an equator of the icosahedron projection such that arrangement of the segment of the top pole and the segment of the bottom pole preserve a continuity in the segment of the 360 degree content). Further, the bitrate of the encoded bit-stream is considerably reduced as compared to existing methods. The file size of an encoded bit-stream is reduced. The image quality of the encoded bit-stream is improved.

The segment of the top pole and the segment of the bottom pole of the icosahedron projection are arranged between the equator of the icosahedron projection such that arrangement of the segment of the top pole and the segment of the bottom pole preserve the continuity in the segment of the 360 degree content. This results in providing the smaller encoded file sizes so as to lead to a lower storage and bandwidth requirement, as compared to existing reshaping solutions. The proposed method provides fewer angular discontinuities and better horizontal discontinuities which provide gains during the encoding process.

The proposed method can be used to reshape a non-compact ISP to a compact rectangular ISP, so as to guarantee fewer discontinuities as compared to previous rearrangement methods. This results in improving a compression/coding efficiency of the 360 degree video content.

The method can be implemented in a reverse manner during rendering of the 360 degree content. The method can be used to assist in enhancing a motion estimation procedure so as to improve the compression performance.

The proposed method is accepted in the JVET-E1003 standard.

In the proposed method, the reshaped Icosahedron Projection is provided as input to Virtual Reality (VR) and video applications.

Referring now to the drawings, and more particularly to FIGS. 1 through 11, there are shown preferred embodiments.

FIG. 1 is a block diagram of an electronic device 100 for producing a 360 degree content on a rectangular projection, according to an embodiment as disclosed herein. The electronic device 100 can be, for example, but not limited to a smart phone, a Personal Digital Assistant (PDA), a tablet computer, a laptop computer, a VR system, a server or the like. The 360 degree content can be, for example but not limited to, a 360 degree image content, a 360 degree video content, a 360 degree multimedia content or the like.

In an embodiment, the electronic device 100 includes a 360 degree content controller 110, a display 120, a memory 130, and a processor 140. The 360 degree content controller 110 is configured to generate an icosahedron projection having vertices and triangles in equal shape. Each of the triangles represent the 360 degree content.

In an embodiment, the icosahedron projection includes at least 12 vertices and at least 20 faces, where each of the faces is the triangle in equal shape and represents a face of the icosahedron projection.

After generating the icosahedron projection having vertices and triangles in equal shape, the 360 degree content controller 110 is configured to form a rectangular image frame displaying the 360 degree content by reshaping the triangles (i.e., rearranging the triangles).

In an embodiment, the 360 degree content controller 110 is configured to identify and merge continuous segments of the 360 degree content in a top pole of the icosahedron projection. Further, the 360 degree content controller 110 is configured to identify and merge the continuous segments of the 360 degree content in a bottom pole of the icosahedron projection. Further, the 360 degree content controller 110 is configured to divide a middle segment of the icosahedron projection from a center into two segments. Further, the 360 degree content controller 110 is configured to arrange the segment of the top pole and the segment of the bottom pole between the equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve a continuity in the segment of the 360 degree content. Further, the segment of the top pole and the segment of the bottom pole are arranged in such a way that discontinuities in a segment of the 360 degree content coincide with boundaries of a coding unit.

In an embodiment, discontinuities in remaining segment of the 360 degree content is at least one of horizontal discontinuities, vertical discontinuities and angular discontinuities.

In an embodiment, a filter (not shown) is applied to include a margin between the triangle faces to reduce an effect of the discontinuities in the reshaped triangles.

In an embodiment, a padding model is applied to include the margin between the triangle faces to reduce the effect of the discontinuities in the reshaped triangles.

In an embodiment, the 360 degree content controller 110 can be used to reduce the effect of discontinuities in the reshaped triangles by applying a filter based padding between the reshaped triangles. The filter based padding corresponds to any padding procedure (e.g., Bilinear padding procedure, weighted averaging padding procedure or the like).

In an embodiment, the continues segments of the top pole and the bottom pole are detected when at least two triangles faces next to each other.

In an embodiment, the reshaped triangles present a continuous display of the set of segment of the 360 degree content in the rectangular image frame.

Further, the memory 130 stores the rectangular image frame in the electronic device 100. The memory 130 stores the rectangular image frame to display the 360 degree content on a display screen (not shown). The memory 130 also stores instructions to be executed by the processor 140. The memory 130 may include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the memory 130 may, in some examples, be considered a non-transitory storage medium. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted that the memory 130 is non-movable. In some examples, the memory 130 can be configured to store larger amounts of information than the memory. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in Random Access Memory (RAM) or cache).

The processor 140 is configured to execute instructions stored in the memory 130 and to perform various processes. The display 120 is configured to display the 360 degree content on the rectangular projection. A communicator (not shown) is configured for communicating internally between internal hardware components and with external devices via one or more networks. The communicator is configured for communicating with the 360 degree content controller 110.

Further, the processor 140 encodes the rectangular image frame for streaming and decodes the rectangular image frame. Further, the processor 140 utilizes the decoded reshaped rectangular image frame as input to a VR application and a video application.

Although the FIG. 1 shows various hardware components of the electronic device 100 but it is to be understood that other embodiments are not limited thereon. In other embodiments, the electronic device 100 may include less or more number of components. Further, the labels or names of the components are used only for illustrative purpose and does not limit the scope of the invention. One or more components can be combined together to perform same or substantially similar function to produce the 360 degree content on the rectangular projection in the electronic device 100.

FIG. 2 is a block diagram of the 360 degree content controller 110 of the electronic device 100, according to an embodiment as disclosed herein. In an embodiment, the 360 degree content controller 110 includes an icosahedron projection controller 112, a triangle shape controller 114, and a filter 116. The triangle shape controller 114 includes a continuous segments detector 114a and a pole merger 114b.

In an embodiment, the icosahedron projection controller 112 is configured to generate the icosahedron projection including vertices and triangles in equal shape. After generating the icosahedron projection having the vertices and the triangles in equal shape, the triangle shape controller 114 is configured to form the rectangular image frame displaying the 360 degree content by reshaping the triangles.

In an embodiment, the continuous segments detector 114a is configured to identify and merge continuous segments of the 360 degree content in the top pole of the icosahedron projection. Further, the continuous segments detector 114a is configured to identify and merge the continuous segments of the 360 degree content in the bottom pole of the icosahedron projection. Further, the pole merger 114b is configured to divide the middle segment of the icosahedron projection from a center into two segments. Further, the pole merger 114b is configured to arrange the segment of the top pole and the segment of the bottom pole between the equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve a continuity in the segment of the 360 degree content.

In an embodiment, the pole merger 114b is configured to arrange the segment of the top pole and the segment of the bottom pole such a way that discontinuities in the segment of the 360 degree content coincide with boundaries of the coding unit.

In an embodiment, the filter 116 is applied to reduce an effect of the discontinuities in the reshaped triangles. The filter 116 can be a low pass filter.

Although the FIG. 2 shows various hardware components of the 360 degree content controller 110 but it is to be understood that other embodiments are not limited thereon. In other embodiments, the 360 degree content controller 110 may include less or more number of components. Further, the labels or names of the components are used only for illustrative purpose and does not limit the scope of the invention. One or more components can be combined together to perform same or substantially similar function to produce the 360 degree content on the rectangular projection in the electronic device 100.

FIG. 3 is a flow diagram 300 illustrating a method for producing the 360 degree content on the rectangular projection in the electronic device 100, according to an embodiment as disclosed herein.

At 302, the method includes generating the icosahedron projection including vertices and triangles in equal shape. Each of the triangles represent the 360 degree content. In an embodiment, the method allows the 360 degree content controller 110 to generate the icosahedron projection including vertices and triangles in equal shape.

At 304, the method includes forming the rectangular image frame displaying the 360 degree content by reshaping the triangles. The reshaped triangles presents the continuous display of the set of segment of the 360 degree content in the rectangular image frame. In an embodiment, the method allows the 360 degree content controller 110 to form the rectangular image frame displaying the 360 degree content by reshaping the triangles.

At 306, the method includes storing the rectangular image frame in the electronic device 100. In an embodiment, the method allows the memory 130 to store the rectangular image frame in the electronic device 100.

The various actions, acts, blocks, steps, or the like in the flow diagram 300 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some of the actions, acts, blocks, steps, or the like may be omitted, added, modified, skipped, or the like without departing from the scope of the invention.

FIG. 4 is a flow diagram 304 illustrating a method for forming the rectangular image frame displaying the 360 degree content by reshaping the triangles while producing the 360 degree content on the rectangular projection in the electronic device 100, according to an embodiment as disclosed herein.

At 304a, the method includes identifying and merging the continuous segments of the 360 degree content in the top pole of the icosahedron projection. In an embodiment, the method allows the continuous segments detector 114a to identify and merge continuous segments of the 360 degree content in the top pole of the icosahedron projection.

At 304b, the method includes identifying and merging the continuous segments of the 360 degree content in the bottom pole of the icosahedron projection. In an embodiment, the method allows the continuous segments detector 114a to identify and merge continuous segments of the 360 degree content in the bottom pole of the icosahedron projection.

At 304c, the method includes dividing the middle segment of the icosahedron projection from the center into two segments. In an embodiment, the method allows the pole merger 114b to divide the middle segment of the icosahedron projection from the center into two segments.

At 304d, the method includes arranging the segment of the top pole and the segment of the bottom pole between the equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve a continuity in the segment of the 360 degree content. In an embodiment, the method allows the pole merger 114b to arrange the segment of the top pole and the segment of the bottom pole between the equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve a continuity in the segment of the 360 degree content.

The various actions, acts, blocks, steps, or the like in the flow diagram 304 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some of the actions, acts, blocks, steps, or the like may be omitted, added, modified, skipped, or the like without departing from the scope of the invention.

FIGS. 5a-5d are example illustrations in which an ISP frame packing is explained from a non-frame packed ISP, according to an embodiment as disclosed herein.

The FIG. 5a shows a non-compact ISP. The triangles are marked from 0 to 19 (i.e., totally 20 triangles). Referring to the FIG. 5b, the top pole is in a light gray color, the equator is in a medium gray color, and the bottom pole is in a dark gray color. In the top pole, a triangle 0 is inverted and placed next to a triangle 8. A triangle 6 is rotated by 60 degrees and placed next to the triangle 8 to make the triangle 8 and the triangle 6 continuous. Further, a triangle 4 is rotated by 120 degrees and placed next to the triangle 4. Further, the triangle 2 is rotated by 180 degrees and placed next to the triangle 4. Further, the similar rotation and placements are followed by the bottom pole.

For the equator, first adjacent triangles are placed together as shown in the FIG. 5b, and then to divide them in to equal half triangle no.3 and 13 are split in the half and remaining half part of triangle 3 has been kept at an end next to a triangle 17. The left half of the triangle 3 is kept on the top of the layout and right half of the triangle 3 is placed towards the bottom of a layout as shown in the FIG. 5c.

Further, the

triangles

4 and 6 are split in to half and half part of 4 and 6 triangles are placed next to

triangle

14 and 12 as shown in the FIG. 5d. This completed the frame packing the layout in the FIG. 5a to rectangular layout in the FIG. 5d with no gaps and little discontinuities.

FIG. 6a is an example illustration in which the 360 degree content is displayed in the ISP projection format, according to prior art. As shown in the FIG. 6a, there is a lot of discontinuities in the non-compact ISP format. In the traditional video coding procedures consider video frame as a 2D plane. The ISP format as described in existing methods utilizes 20 triangles to project an omnidirectional video onto a 2D plane. In addition, frame packing procedures is used in the existing methods to remove unused parts from being treated as valid input to an encoder. The depicted compact ISP layout contains lot of discontinuous edges.

FIG. 6b is an example illustration in which the 360 degree content is displayed in the ISP projection format, according to an embodiment as disclosed herein. As shown in the FIG. 6b, there is no discontinuities in the non-compact ISP format. This results in increasing the compression efficiency and quality of encoded bit-stream. Unlike to FIG. 6a, the proposed method, as shown in the FIG. 6b, can be used to achieve the maximal continuity between face edges resulting in higher compression efficiency. The proposed method of rearrangement of the triangles is shown in the FIG. 6b. The proposed method can be used to reduce the discontinuities to 8 from 10 between the triangles, four of which are horizontal. Thus the effect of horizontal discontinuities on the compression efficiency is minimal.

FIG. 7 is an example illustration in which the 360 degree content is displayed based on the current and existing methods. The notation “a” of the FIG. 7 depicts the ERP. The notation “b” of the FIG. 7 depicts the existing reshaped ISP, and the notation “c” of the FIG. 7 depicts the proposed reshaped ISP.

FIGS. 8a-8j are example illustrations in which the 360 degree content is displayed on the rectangular projection, according to an embodiment as disclosed herein.

The electronic device 100 receives the ERP image as shown in the FIG. 8a. After receiving the ERP image, the electronic device 100 converts the ERP image into the 360 ISP format as shown in the FIG. 8b. The electronic device 100 identifies and merges the continuous segments of the 360 degree content in the top pole of the 360 ISP format as shown in the FIG. 8c. The electronic device 100 identifies and merges the continuous segments of the 360 degree content in the bottom pole of the 360 ISP format as shown in the FIG. 8d and the FIG. 8e. The adjacent triangles are placed together and the electronic device 100 aligns the continuous segments of the 360 degree content in the bottom pole and the top pole of the 360 ISP format as shown in the FIGS. 8f.

Further, the electronic device 100 divides specific segments of the triangles in to equal half triangle. First segments of the half triangle is kept on the top pole of the 360 degree content and second segments of the half triangle is placed towards the bottom pole of the 360 degree content as shown in the FIGS. 8g and FIGS. 8h. Further, the electronic device 100 arranges the segment of the top pole and the segment of the bottom pole between the equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve the continuity in the segment of the 360 degree content as shown in the FIGS. 8i and FIGS. 8j.

FIG. 9 is an example illustration in which comparison of slant edges in the Icosahedron projection format between existing method and the proposed method. The continuity within the top pole and the bottom pole of the 360 degree content as shown by the arrows in notation “b” of the FIG. 9. As the continuity within the top pole and the bottom pole of the 360 degree content, the boundaries of the coding unit coincide with the triangle edges, so that less number of angular discontinuities compared to existing reshaped ISP in notation “a” of the FIG. 9

In the proposed method, the electronic device 100 can be used to minimize the number of discontinuities in the 360 degree content. Further, the segment of the top pole and the segment of the bottom pole are arranged in such a way that discontinuities in the segment of the 360 degree content coincide with boundaries of the coding unit. This results in higher compression efficiency.

FIG. 10 is an example illustration in which a bleeding effect across discontinuity in the input image and reconstructed image is explained, according to an embodiment as disclosed herein. It has been observed that discontinuities between triangles result in the bleeding effect on the reconstructed image as shown in FIG. 10. This effect is prominently observed especially at high QP values. To prevent such visual artifacts, it is recommended to introduce padding between the discontinuous edges.

The bleeding effect across discontinuity in the input image shown in notation “a” of the FIG. 10, and the bleeding effect across discontinuity in the reconstructed image shown in notation “b” of the FIG. 10.

In an example, the ISP projection format has 12 vertices and 20 faces. Each face is the triangle in the same shape as the face of the icosahedron.

For a compact ISP (CISP), all 20 triangle faces are compacted to the rectangular frame as shown on the FIG. 11. While compacting rectangular frame some triangles are split into the 2 parts vertically, some are flipped vertically or horizontally.

In an embodiment, 4 samples margin horizontal per boundary between 2 triangles are introduced in order to ensure always usage of proper chroma sample for each luma sample in non 4:4:4 formats and keeping resulting frame size multiple of 8. Those extra samples are not used in the CISP to sphere (or viewport) projection, so they are just padded using either the nearest sample from triangle face or from the corresponding location on the sphere.

If two triangle faces are next to each other both on icosahedron and on rectangular coding projection then there is no discontinuity. But if two triangle faces are not neighboring on the icosahedron but put next to each other on rectangular frame then the margin is introduced in order to resolve discontinuity, simplify encoding and reduce artifacts which might appear on the triangles boundaries during the encoding. Again extra samples from the margin are not used in the CISP to sphere (or viewport) projection, so they are just filled using bilinear combination of the nearest available samples from opposite edges of adjacent faces (or just copied if only one side neighbor is available). Size of the margin resolving discontinuity is 64 samples in horizontal directions and 32 samples in vertical directions correspondently.

The notation “C” of the FIG. 11, the CISP has three continuous sets of polygons representing different parts of the icosahedron. Their triangle indexes are:

Set A: 2; 4-1; 6-2; 8; 13-2; 9; 15; 1; 17; 3-2.

Set B: 3-1; 19; 5; 11; 7; 13-1; 18; 10; 12; 14; 16.

Set C: 6-1; 4-2.

There is a vertical discontinuity between faces: 19 and 2; 18 and 0; 16 and 1; 13-1 and 6-1. There is a horizontal discontinuity between faces: 1 and 18; 0 and 16; 14 and 4-2; 12 and 6-1

Displacements for set of triangle faces are summarized in Table 1. Table 1 shows a displacement for faces set in the CISP.

Shift	Set A	Set B	Set C
Vertical Shift	V_a	V_b	V_c
Horizontal Shift	H_a	H_b	H_c

The notation “B” of the FIG. 11 specifies procedures of padding on each boundary:

H_b pixels in top row and H_c pixels in bottom row; copy padding method is used,

V_a pixels are padded with bilinear filter,

H_b pixels are padded with bilinear filter,

V_c pixels are padded with bilinear filter.

Further, the size of the CISP rectangular frame is calculated based on width (W) and height (H) of rectangular faces and horizontal (p_h) and vertical (p_v) padding sizes as follows:

W_CISP=5·(W/2+4)+2·p_h

H_CISP=4·H+p_v

Where for W calculation there is multiple 5 because the CISP has 5 triangles boundaries in each horizontal line. For H calculation there is multiple 4 because CISP has 4 triangles boundaries in each vertical line.

The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the embodiments as described herein.

Claims

A method for producing a 360 degree content on a rectangular projection in an electronic device, comprising:

generating an icosahedron projection comprising vertices and triangles in equal shape, wherein each of the triangles represent a segment of the 360 degree content;

forming a rectangular image frame displaying the 360 degree content by re-arranging the triangles, wherein the triangles are rearranged to represent a maximum continuity of the 360 degree content in the rectangular image frame; and

storing the rectangular image frame in the electronic device.
The method of claim 1, wherein the icosahedron projection is generated by mapping points of the 360 degree content to a surface of the icosahedron projection.
The method of claim 1, wherein the icosahedron projection comprises at least 12 vertices and at least 20 triangular faces, wherein each of the triangular faces is in equal shape and represents a face of the icosahedron projection.
The method of claim 1, wherein forming the rectangular image frame displaying the 360 degree content by re-arranging the triangles comprises:

identifying and merging continuous segments of the 360 degree content in a top pole of the icosahedron projection;

identifying and merging continuous segments of the 360 degree content in a bottom pole of the icosahedron projection;

dividing an equator of the icosahedron projection from a center into two equal segments; and

arranging a segment of the top pole and a segment of the bottom pole between the equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve continuity in the segment of the 360 degree content.
The method of claim 4, wherein the icosahedron projection is divided into two segments vertically.
The method of claim 4, wherein the segment of the top pole and the segment of the bottom pole are arranged in such a way that discontinuities in a segment of the 360 degree content coincide with boundaries of a coding unit.
The method of claim 1, wherein discontinuities in remaining segment of the 360 degree content is at least one of horizontal, vertical and angular discontinuities.
The method of claim 6, wherein a margin is introduced in order to reduce the effect of discontinuity in the 360 degree content when at least two triangles are non-neighboring on the icosahedron projection but put next to each other on the rectangular image frame.
The method of claim 6, wherein at least one of a filter and a padding model is applied to reduce an effect of the discontinuities in the reshaped triangles.
The method of claim 1, wherein the continuous display of the set of segments of the 360 degree content in the rectangular image frame reduces a bit-rate of the 360 degree content.
The method of claim 11, wherein the continuous segment of the top pole and the bottom pole is detected when at least two triangles faces next to each other.
An electronic device for producing a 360 degree content on a rectangular projection, comprising:

a memory;

a processor; and

a 360 degree content controller, coupled to the memory and the processor, configured for:

generating an icosahedron projection comprising vertices and triangles in equal shape, wherein each of the triangles represent the 360 degree content;

forming a rectangular image frame displaying the 360 degree content by re-arranging the triangles, wherein the triangles are rearranged to represent a maximum continuity of the 360 degree content in the rectangular image frame; and

storing the rectangular image frame in the electronic device.
The electronic device of claim 12, wherein the icosahedron projection is generated by mapping points of the 360 degree content to a surface of the icosahedron projection.
The electronic device of claim 12, wherein icosahedron projection comprises at least 12 vertices and at least 20 faces, wherein each of the faces is the triangle in equal shape and represents a face of the icosahedron projection.
The electronic device of claim 12, wherein forming the rectangular image frame displaying the 360 degree content by re-arranging the triangles comprises:

identifying and merging continuous segments of the 360 degree content in a top pole of the icosahedron projection;

identifying and merging continuous segments of 360 degree content in a bottom pole of the icosahedron projection;

dividing an equator of the icosahedron projection from a center into two segments; and

arranging a segment of the top pole and a segment of the bottom pole between the equator such that arrangement of the segment of the top pole and the segment of the bottom pole preserve a continuity in the segment of the 360 degree content.
The electronic device of claim 15, wherein the icosahedron projection is divided into the two segments vertically.
The electronic device of claim 15, wherein the segment of the top pole and the segment of the bottom pole are arranged in such a way that discontinuities in a segment of the 360 degree content coincide with boundaries of a coding unit.
The electronic device of claim 12, wherein discontinuities in remaining segments of the 360 degree content is at least one of horizontal, vertical and angular discontinuities.
The electronic device of claim 17, wherein a margin is introduced in order to resolve the discontinuity in the 360 degree content when at least two triangles are not neighboring on the icosahedron projection but put next to each other on the rectangular image frame.
The electronic device of claim 17, wherein at least one of a filter and a padding model is applied to reduce an effect of the discontinuities in the reshaped triangles.
The electronic device of claim 12, wherein the continuous display of the set of segments of the 360 degree content in the rectangular image frame reduces a bit-rate of the 360 degree content.
The electronic device of claim 12, wherein the continuous segment of the top pole and the bottom pole is detected when at least two triangles faces next to each other.