EP3419301B1 - Verfahren zum senden und empfangen von metadaten eines omnidirektionalen bildes - Google Patents

Verfahren zum senden und empfangen von metadaten eines omnidirektionalen bildes Download PDF

Info

Publication number
EP3419301B1
EP3419301B1 EP17753516.8A EP17753516A EP3419301B1 EP 3419301 B1 EP3419301 B1 EP 3419301B1 EP 17753516 A EP17753516 A EP 17753516A EP 3419301 B1 EP3419301 B1 EP 3419301B1
Authority
EP
European Patent Office
Prior art keywords
region
geometry
video
field
metadata
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP17753516.8A
Other languages
English (en)
French (fr)
Other versions
EP3419301A1 (de
EP3419301A4 (de
Inventor
Byeong-Doo Choi
Eric Yip
Jae-Yeon Song
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of EP3419301A1 publication Critical patent/EP3419301A1/de
Publication of EP3419301A4 publication Critical patent/EP3419301A4/de
Application granted granted Critical
Publication of EP3419301B1 publication Critical patent/EP3419301B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/06—Topological mapping of higher dimensional structures onto lower dimensional surfaces
    • G06T3/067—Reshaping or unfolding 3D tree structures onto 2D planes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
    • H04N21/2353—Processing of additional data, e.g. scrambling of additional data or processing content descriptors specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/23614—Multiplexing of additional data and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/84—Generation or processing of descriptive data, e.g. content descriptors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

  • the present disclosure generally relates to a method for transmitting and receiving omnidirectional media and metadata, and more particularly, to metadata for transmitting, receiving, and reconstructing omnidirectional media, such as Virtual Reality (VR) content, mobile VR content, and a 360-degree image, and a method and an apparatus for transmitting and receiving the same.
  • VR Virtual Reality
  • High-Definition (HD) content Consumer media content continues to evolve from black-and-white content to color content, High-Definition (HD) content, and Ultra-High-Definition (UHD) content.
  • HD High-Definition
  • UHD Ultra-High-Definition
  • high-Dynamic-Range (HDR) content has been standardized, and specifications for HDR content have been prepared.
  • ISO/IEC International Organization for Standardization/International Electrotechnical Commission 23000 specifics a file format designed to store, exchange, manage, edit, and present image content based on ISO Base Media File Format (ISO BMFF).
  • VR media content captured by a video camera may be mastered by a mastering device, may be encoded/multiplexed by an encoding/multiplexing (Mux) device, and may be distributed as VR content in various media formats.
  • the distributed VR content may be decoded/demultiplexed (Demux) at a receiver, may be transmitted to a display device, and may be VR-displayed.
  • VR content is significantly different in property from conventional two-dimensional (2D) or three-dimensional (3D) content.
  • VR content provides a user with an entire 360-degree view, enabling the user to have an immersive experience.
  • the present disclosure specifies a file format that enables storage, exchange, management, edition, and presentation of content.
  • image content based on ISO Base Media File Format ISO BMFF
  • ISO BMFF ISO Base Media File Format
  • metadata provides an overall method for storing omnidirectional media along with projection-related information.
  • the present disclosure proposes a method for transmitting, by a transmission apparatus, metadata of an omnidirectional video according to claim 1.
  • the present disclosure proposes a method for receiving, by a reception apparatus, metadata of an omnidirectional video according to claim 7.
  • the present disclosure proposes a transmission apparatus for transmitting metadata of an omnidirectional video according to claim 10.
  • the present disclosure proposes a reception apparatus for receiving metadata of an omnidirectional video according to claim 11.
  • a transmitting side may map an omnidirectional video to a planar video to transmit the omnidirectional video, and a receiving side may reconstruct an omnidirectional video from a planar video to render the omnidirectional video.
  • a base station is a subject communicating with a User Equipment (UE), and may be referred to as a BS, a Node B (NB), an eNode B (eNB), an Access Point (AP) or the like.
  • UE User Equipment
  • BS Node B
  • eNB eNode B
  • AP Access Point
  • the user equipment is a subject communicating with the BS, and may be referred to as a UE, a Mobile Station (MS), a Mobile Equipment (ME), a device, a terminal or the like.
  • MS Mobile Station
  • ME Mobile Equipment
  • Omnidirectional media may also be referred to as omnidirectional content, an omnidirectional video, 360-degree media, 360-degree content, a 360-degree video, VR media, VR content, or a VR video.
  • FIG. 1A illustrates an operation of a transmitting side encoding and transmitting an omnidirectional video and an operation of a receiving side receiving and reconstructing an omnidirectional video according to the present disclosure.
  • FIG 1A An overall process for omnidirectional (360-degree) content may be illustrated, for example, in FIG 1A .
  • a 360-degree view may be captured using a plurality of cameras. Images from the plurality of cameras may be aligned and stitched together and may be mapped into a single image (170). The image or video may be encoded (172) and stored.
  • a file generator for an omnidirectional Media Application Format (MAF) accepts omnidirectional media content including video, audio, and metadata streams. Any metadata for converting a projected 2D image into a sphere 3D image may be included in the stream.
  • MAF omnidirectional Media Application Format
  • a 3D model and projection metadata may be parsed and are shared with a 3D projector.
  • the 2D image or video may be decoded (180).
  • the 3D projector may render the decoded 2D image or video into a 360-degree image or video (182) by mapping the decoded 2D image or video onto the surface of a particular 3D model using the shared projection metadata.
  • a specific view from the entire 360-degree image or video may be extracted and may be displayed (184) on a Head-Mounted Display (HMD) or other display devices (e.g., a mobile phone or a tablet PC).
  • HMD Head-Mounted Display
  • FIG. 1B illustrates in detail an operation of a transmitting side encoding and transmitting an omnidirectional video and an operation of a receiving side receiving and reconstructing an omnidirectional video according to the present disclosure.
  • an omnidirectional video may be presented through a 3D geometry (3D geometric structure, which is hereinafter referred as a "geometry").
  • the geometry may be various shapes.
  • the geometry may be a sphere 110, a cylinder 112, a platonic solid, a pyramid, a truncated pyramid, and the like.
  • the platonic solid may include a tetrahedron, a cube or hexahedron 114, an octahedron, a dodecahedron, and an icosahedron 116.
  • Platonic solids except for a cube with square surfaces and a dodecahedron with regular pentagonal surfaces have equilateral trianglular surfaces.
  • the surfaces may be a patch.
  • the patch is fitted to a unit area used for rendering the omnidirectional video. That is, the omnidirectional video may be encoded and transmitted in one or more areas by the transmitting side 100, and the transmitted one or more areas may be decoded and then rendered (reconstructed as a geometry) and reproduced as an omnidirectional image by the receiving side 102.
  • the patch (or surface) fitted to the video may also be referred to as a patch (or surface) mapped to the video.
  • a patch refers to a separate unit of the omnidirectional video and may also be referred to as a surface, a tile, or a cell.
  • the patch may be variously determined depending on the geometry type.
  • the sphere may have a single patch in a rectangular shape 124 by equirectangular projection.
  • the cylinder may have, as patches, a top surface and a bottom surface in a circular shape 120 and one side surface in a square shape 124.
  • the platonic solid may have, as patches, surfaces in a triangular shape 126 (when the geometry is a tetrahedron, an octahedron, or an icosahedron), in a square shape 124 (when the geometry is a cube) or in a pentagonal shape (when the geometry is a dodecahedron).
  • the pyramid may have, as patches, a bottom surface in a square shape 124 and four side surfaces in a triangular shape 126.
  • the truncated pyramid may have, as patches, a top surface and a bottom surface in a square shape 124 and four side surfaces in a trapezoidal shape 122. Therefore, the shape of the patch may be explicitly indicated but may also be implicitly indicated by the geometry type.
  • the present disclosure proposes a method for mapping a patch 120, 122, 124 or 126 on a geometry corresponding to the omnidirectional image to a planar video (or 2D video), using a rectangular region 130 defined on the planar video. That is, according to the present disclosure, the transmitting side 100 maps (or projects) an area (including a patch) of the omnidirectional video to a region 130 of the planar image, encodes the planar image, and transmits the encoded media, along with information related to the mapping (i.e., metadata), to the receiving side.
  • the transmitting side 100 may encode the mapped planar video using a High-Efficiency Video Coding (HEVC) encoder and may multiplex the encoded media with the metadata using an Omnidirectional Media Application Format (OMAF) multiplexer.
  • HEVC High-Efficiency Video Coding
  • OMAF Omnidirectional Media Application Format
  • the receiving side 102 receives the encoded media and the metadata.
  • the receiving side 102 may decode the encoded media to obtain an area mapped to a region 140 and may reconstruct (i.e., render) a geometry 160, 162, 164, or 166 of the omnidirectional video from a patch 150, 152, 154, or 156 included in the obtained area using the metadata.
  • the region is a generalized framework to present mapping (i.e., projecting) to various geometries and may also be referred to as a Super Rectangular Mapping Framework (SRMF).
  • SRMF Super Rectangular Mapping Framework
  • the region on the planar video may have, for example, a rectangular shape.
  • a patch having a triangular, rectangular, circular, trapezoidal, or another shape (e.g., diamond shape) may be fitted to the area, and at least one side of the patch may be accurately fitted in the region on the planar video.
  • the patch being fitted in the region means a case that the patch is disposed such that the entire patch is included in the region, the entirety or part of the boundary line of the patch meets the boundary line of the region, and a non-overlapping portion of the patch and the region is arranged to be minimum.
  • the patch 126 when a triangular patch 126 is disposed such that the bottom left vertex and the bottom right vertex thereof meet the lower side of the region 130 and the top vertex thereof meets the upper side of the region 130, the patch 126 may be described as being fitted to the region 130.
  • each surface of the geometry is a triangular, rectangular, circular, or trapezoidal patch and is arranged in a region on the planar video.
  • video data of the triangular, square, circular, or trapezoidal patch is mapped (projected) to a specified position of the geometry and is rendered. That is, the region may include a patch in various shapes, such as triangular, rectangular, diamond, trapezoidal, or circular shape, inside, and the receiving side uses the video data having the shape of the patch to reconstruct (i.e., render) the omnidirectional video.
  • the shape of the patch fitted in the region may be indicated by an index value for each region or may be implicitly determined according to the shape of the surface of the geometry.
  • Each of one or more regions may be arbitrarily selected by the transmitting side, may be selected on the basis of a surface of the geometry in a particular shape, may be selected on the basis of a factor related to encoding and transmission, or may be selected by a combination thereof.
  • FIG. 1C illustrates an example of an ISO-based media file format to which a technique according to the present disclosure is applicable.
  • ISO/IEC 14496-12 ISO-based media file format is a flexible and extensible format and may include, as illustrated in FIG. 1C , timed media information 194 to allow interchange, management, edition, and presentation of media.
  • the ISO-based media file 190 may be a default format for an omnidirectional MAF.
  • the media file 190 may include at least one of a moov file 192 and an mdat file 194.
  • the moov file 192 corresponds to, for example, movie data and may include a video trak file 196 or an audio trak file 198.
  • the mdat file corresponds to media data.
  • mapping examples of mapping the geometry of various types of omnidirectional videos to one or more regions on a planar video will be described with reference to drawings.
  • Different mapping methods may commonly have one region (i.e., SRMF).
  • a variety of mapping (projecting) forms may be presented by a transmitting side and may be signaled to a receiving side using a rectangular region illustrated above. That is, various geometries that support different mapping methods may be presented using a common region.
  • the opposite mapping (projecting) operation i.e., the operation of the transmitting side mapping (projecting) a geometry to a region of a planar video
  • the opposite mapping (projecting) operation i.e., the operation of the transmitting side mapping (projecting) a geometry to a region of a planar video
  • each region may have a patch in the same shape as that of a corresponding surface.
  • each region may have a triangular patch.
  • each region maps video data corresponding to the entirety of each surface to the patch.
  • a surface of the geometry corresponding to each region may be designated by an index for each region.
  • the index to designate the surface of the geometry that is transmitted via metadata is 'cylinder_surface_id', 'cube_surface_id', 'triangle_surface_id', or 'ref_triangle_surface_id'.
  • the structure of a metadata to transmit an index will be described later.
  • FIG. 2 illustrates a method for reconstructing an omnidirectional video from a region on a planar video when a geometry is a sphere.
  • the geometry of an omnidirectional video may be indicated, for example, via a 'geometry_type' field included in metadata.
  • equirectangular projection which is a method for projecting a planar video into a complete spherical geometry, may be used.
  • the geometry is a sphere 202
  • a region 200 of the planar video may be mapped to the sphere 202.
  • FIG. 3 illustrates a method for projecting a planar video onto a sphere using equirectangular projection.
  • one point P'( ⁇ , ⁇ ) 302 on a spherical geometry 300 may be projected onto a point P(x, y) 312 on a planar video 310 by the following equation.
  • the longitude ⁇ may be represented by ( ⁇ - ⁇ 0 ).
  • FIG. 4 illustrates a case where the geometry of an omnidirectional video is a squished sphere.
  • a squished sphere is a sphere that is transformed (generally transformed to have a smaller volume) by pressing the entirety or part of a geometry.
  • a video corresponding to the squished portion of the squished sphere has less importance or loss rate than that of videos corresponding to other portions and thus may be down-sampled.
  • top height portion 402 of the sphere starting relative to an angle 400 indicated by a 'squish_start_pitch_top' field may be squished at a ratio of the value of a 'squish_ratio_top' field.
  • a top height portion 404 in (b) of FIG. 4 is a result of squishing at the ratio of in the value of the 'squish ratio_top' field.
  • the value of the squishing ratio may be normalized to 255.
  • a bottom height portion 412 of the sphere starting relative to an angle 410 indicated by a 'squish start_pitch_bottom' field may be squished at a ratio of the value of a 'squish_ratio_bottom' field.
  • a bottom height portion 414 in (b) of FIG. 4 is a result of squishing at the ratio of the value of the 'squish ratio top' field.
  • the value of the squishing ratio may be normalized to 255.
  • FIG. 5 illustrates a method for projecting an area on an omnidirectional video onto a region on a planar video when a geometry is a sphere or squished sphere.
  • One region 502 on a planar video 500 may be mapped to one area 512 of a squished sphere 510.
  • the position of the region 502 on the planar video 500 may be determined by 'region_top_left_x' and 'region_top_left_y' fields, and the extent of the region 502 may be determined by 'region_height' and 'region width' fields.
  • the determined region may be mapped to the area 512, determined by 'yaw_end', 'yaw_start', 'pitch_start', and 'pitch_end', on the geometry of the sphere 510.
  • FIG. 6 illustrates a specific example of projecting an omnidirectional video onto a planar video when a geometry is a sphere or squished sphere.
  • a pixel (x, y) 602 in a rectangular region 600 with a width of w and a height of h on a planar video may be projected onto a point ( ⁇ ', ⁇ ') 612 in a curved surface area 610 on a 3D sphere.
  • the curved surface area 610 may be defined by a yaw angle of ⁇ 614 and a pitch angle of ⁇ 616.
  • the projected point ( ⁇ ', ⁇ ') 612 may be defined by the following equation.
  • the origin of a (squished) spherical geometry is the geometric center 620 of the sphere.
  • the geometric center 620 of the sphere also corresponds to an observation position for a viewer in the sphere.
  • the yaw angle 614 is determined to range between -180° and 180° on the horizontal plane.
  • the pitch angle 616 is determined to range between -90° and 90° on the vertical plane.
  • FIG. 7 illustrates reconstruction between regions on an omnidirectional video and a planar video when a geometry is a cylinder.
  • the planar video mapped to the cylinder may include a rectangular area 700 on the left side, which is mapped to the lateral side of the cylinder, and two rectangular regions 704 and 708 on the right side, into which the circular surfaces 702 and 706 having a radius of the value of a 'radius' field is fitted.
  • the region 704 including an upper circle 702 is mapped to the upper surface 710 of the cylinder, and the region 708 including a lower circle 706 is mapped to the lower surface 712 of the cylinder.
  • the arrangement of the regions is not limited to the above example.
  • the regions 704 and 708 including the circles may be disposed on the left side, and the rectangular area 700 mapped to the lateral side of the cylinder may be disposed on the right side of the regions 704 and 7085 including the circles.
  • i with a value of 0 or 1 may indicate a top area or bottom area.
  • FIG. 8 illustrates an example of projecting a region on a planar video onto an area in a cylinder when a geometry is the cylinder.
  • One region 802 on a planar video 800 may be mapped to one area 812 of a cylinder 810.
  • the position of the region 802 on the planar video 800 may be determined by 'region_top_left_x' and 'region top_left_y' fields, and the extent of the region 802 may be determined by 'region height' and 'region_width' fields.
  • the determined region may be mapped to the area 812, determined by 'height_end', 'height start', 'pitch_start', and 'pitch end', on the cylinder 810.
  • FIG. 9 illustrates a method for projecting a region on a planar video onto a surface of a cylinder when a geometry is the cylinder.
  • a pixel (x, y) 902 in a rectangular region 900 with a width of w and a height of h of a planar video may be projected onto a point ( ⁇ ', y') 912 in a curved surface area 910 of a 3D cylinder.
  • the 3D cylinder may be defined by a height of h' and a pitch angle of ⁇ .
  • the projected point ( ⁇ ', y') may be defined by the following equation.
  • FIG. 10A illustrates a method for projecting a region on a planar video onto an area of a cube when a geometry is the cube.
  • each region may be mapped to one cube surface 1002.
  • the position and size of each region may be indicated by an 'OmniMediaTextureMappingMetadataSample' or 'OmnidirectionalMediaTextureMappingMetadataSample' box.
  • FIG. 10B illustrates another method for projecting a region on a planar video onto an area of a cube when a geometry is the cube.
  • each region may be mapped to one cube surface 1002.
  • the position and size of each region may be indicated by an 'OmniMediaTextureMappingMetadataSample' or 'OmnidirectionalMediaTextureMappingMetadataSample' box.
  • the 'OmniMediaTextureMappingMetadataSample' or 'OmnidirectionalMediaTextureMappingMetadataSample' box will be described later.
  • FIG. 11 illustrates a method for projecting a region on a planar video onto an area on a cube when a geometry is the cube.
  • One region 1102 on a planar video 1100 may be mapped to one area 1112 on a surface 1111 indicated by 'cube_surface_id' of a cube 1110.
  • the position of the region 1102 on the planar video 1100 may be determined by 'region top_left_x' and 'region_top_left_y' fields, and the extent of the region 1102 may be determined by 'region_height' and 'region_width' fields.
  • the determined region may be mapped to the area 1112, determined by 'area_top_left_x', 'area_top_left_y', 'area_width', and 'area_height' fields, on the surface 1111 of the cube 1110.
  • FIG. 12 illustrates a method for projecting a region on a planar video onto a surface on a cube when a geometry is the cube.
  • a pixel (x, y) 1202 in a rectangular region 1200 with a width of w and a height of h of a planar video may be projected onto a point (x', y') 1212 on a rectangular area 1210 of a 3D geometric surface having a width of w' and a height of h'.
  • FIG. 13 illustrates an example of projecting a region on a planar video onto an area on a platonic solid when a geometry is a triangle-based octahedron.
  • a geometry is an octahedron
  • eight triangular surfaces may be mapped to regions of a planar video.
  • the triangular patches mapped to respective surfaces may be arranged to fit in the regions of the planar video as shown in FIG. 13 .
  • a patch 1302 corresponding to surface ⁇ ,1 1310 of the octahedron is arranged to fit in a rectangular region 1300 on the planar video.
  • areas and patches may be arranged as shown in FIG. 13 .
  • the position and size of the triangular patches i.e., areas) may be defined by FIGS. 17 to 19 and the following equation.
  • FIG. 14 illustrates another example of projecting a region on a planar video onto an area on a platonic solid when a geometry is a triangle-based octahedron.
  • a geometry is an octahedron
  • eight triangular surfaces may be mapped to regions of a planar video. At least one side of the triangular patches mapped to respective surfaces may be arranged to fit in the regions on the planar video as shown in FIG. 14 .
  • a region including one patch partially overlaps with another region.
  • a patch 1402 corresponding to surface ⁇ ,1 1420 of the octahedron is arranged to fit in a rectangular region 1400 on the planar video.
  • a patch 1412 corresponding to surface ⁇ ,2 1422 of the octahedron is arranged to fit in a rectangular region 1410 in a dotted line on the planar video.
  • the region 1400 and the region 1410 partially overlap.
  • resources required for data storage of the planar video may be reduced.
  • the value of a 'predefined_pattern' field in metadata is 2, areas and patches may be arranged as shown in FIG. 14 .
  • the patches arranged in the regions may be arranged not to overlap.
  • FIG. 15 illustrates an example of projecting a region on a planar video onto an area on a platonic solid when a geometry is a triangle-based icosahedron.
  • a geometry is an icosahedron
  • 20 triangular patches may be mapped to regions of a planar video.
  • the triangular patches mapped to respective surfaces may be arranged to fit in the regions of the planar video as shown in FIG. 15 .
  • a patch 1502 corresponding to surface ⁇ ,1 1510 of the octahedron is arranged to fit in a rectangular region 1500 on the planar video.
  • areas and patches may be arranged as shown in FIG. 15 .
  • FIG. 16 illustrates another example of projecting a region on a planar video onto an area on a platonic solid when a geometry is a triangle-based icosahedron.
  • a geometry is an icosahedron
  • 20 triangular surfaces may be mapped to regions of a planar video.
  • the triangular patches mapped to respective surfaces may be arranged to fit in the regions on the planar video as shown in FIG. 16 .
  • a region including one patch partially overlaps with another region.
  • a patch 1602 corresponding to surface ⁇ ,1 1620 of the icosahedron is arranged to fit in a rectangular region 1600 on the planar video.
  • a patch 1612 corresponding to surface ⁇ ,2 1622 of the icosahedron is arranged to fit in a rectangular region 1610 in a dotted line on the planar video.
  • the region 1600 and the region 1610 partially overlap.
  • resources required for data storage of the planar video may be reduced. For example, when the value of a 'predefined_pattern' field in metadata is 2, areas and patches may be arranged as shown in FIG. 16 .
  • FIG. 24 illustrates still another example of projecting a region on a planar video onto an area on a platonic solid when a geometry is a triangle-based octahedron.
  • a geometry is an octahedron
  • eight triangular surfaces may be mapped to regions of a planar video. At least one side of the triangular patches mapped to respective surfaces may be arranged to fit in the regions on the planar video as shown in FIG. 24 .
  • a region including one patch partially overlaps with another region.
  • one patch may be divided into two or more portions, which may be included in separate regions. For example, a patch 2402 corresponding to a half of surface ⁇ ,1 2420 of the octahedron is arranged to fit in a rectangular region 2400 on the planar video.
  • the region 2400 is a region including the entire patch 2404 corresponding to surface ⁇ ,2 2422 of the octahedron.
  • a patch corresponding to a half of a particular surface may be included in an unoccupied space in the region, thereby minimizing an unoccupied space and reducing resources required for data storage of the planar video.
  • the value of a 'predefinedpattern' field in metadata is 3, areas and patches may be arranged as shown in FIG. 24 .
  • the patches arranged in the regions may be arranged not to overlap.
  • FIG. 25 illustrates yet another example of projecting a region on a planar video onto an area on a platonic solid when a geometry is a triangle-based octahedron.
  • a geometry is an octahedron
  • eight triangular surfaces may be mapped to regions of a planar video. At least one side of the triangular patches mapped to respective surfaces may be arranged to fit in the regions on the planar video as shown in FIG. 25 .
  • a region including one patch partially overlaps with another region.
  • one patch may be divided into two or more portions, which may be included in separate regions. Unlike in FIG. 24 , it is shown in FIG. 25 that, for flexibility in arrangement, a patch is rotated by 90 degrees and is fit in a region.
  • a patch corresponding to a half of a particular surface may be included in an unoccupied space in the region, thereby minimizing an unoccupied space and reducing resources required for data storage of the planar video.
  • a 'predefined_pattern' field in metadata is 4, areas and patches may be arranged as shown in FIG. 25 .
  • the patches arranged in the regions may be arranged not to overlap.
  • FIG. 26 illustrates still another example of projecting a region on a planar video onto an area on a platonic solid when a geometry is a triangle-based octahedron.
  • a geometry is an octahedron
  • eight triangular surfaces may be mapped to regions of a planar video. At least one side of the triangular patches mapped to respective surfaces may be arranged to fit in the regions on the planar video as shown in FIG. 26 .
  • a region including one patch partially overlaps with another region.
  • one patch may be divided into two or more portions, which may be included in separate regions.
  • patches corresponding to surfaces ⁇ ,1, ⁇ ,2, ⁇ ,3 , and ⁇ ,4 of the octahedron are rotated by 180 degrees and are fit in respective regions.
  • a patch corresponding to a half of a particular surface may be included in an unoccupied space in the region, thereby minimizing an unoccupied space and reducing resources required for data storage of the planar video.
  • a 'predefined_pattern' field in metadata is 5
  • areas and patches may be arranged as shown in FIG. 26 .
  • the patches arranged in the regions may be arranged not to overlap.
  • FIG. 17A illustrates metadata related to a region on a planar video and the arrangement of triangular patches when a geometry is a triangle-based platonic solid.
  • 'region_top_left_x' and 'region_top_left_y' included in metadata are fields indicating the left-top position of a region 1700, and 'region_height' and 'region_width' defines the extent of the region 1700.
  • an octahedron is mapped to eight triangular patch-based regions, in which each patch may be inverted, flipped, or rotated 90 degrees, 180 degrees, or 270 degrees.
  • rotation is an operation of rotating the patch with an axis perpendicular to a plane as the rotation axis
  • flip is an operation of rotating the patch 180 degrees with one axis horizontal with the plane as the rotation axis
  • inversion is an operation of rotating the patch 180 degrees with respect to each of two axes, which are horizontal with the plane and are perpendicular to each other, as rotation axes.
  • the orientation (i.e., rotation, flip, or inversion) of the patch may be indicated by the value of an 'is_inverted' field included in the metadata.
  • the 'is_inverted' field may be referred to as an 'orientation_of_triangle' field.
  • a triangular patch 1702 illustrated in FIG. 17 illustrates a case where the value of the 'is inverted' field is 0.
  • the 'is inverted' field indicating whether the triangular patch is inverted may be included in the metadata when there is no predefined pattern for the patch, that is, when the value of a 'predefined_pattern' field is 0.
  • FIG. 17B illustrates metadata related to a region on a planar video and the arrangement of triangular patches when a geometry is a triangle-based platonic solid.
  • 'region top_left_x' and 'region_top_left_y' included in metadata are fields indicating the left-top position of a region 1710, and 'region_height' and 'region_width' defines the extent of the region 1710.
  • an octahedron is mapped to eight triangular patch-based regions, in which each patch may be inverted, flipped, or rotated 90 degrees, 180 degrees, or 270 degrees.
  • Rotation, flip, or inversion may be indicated by the value of an 'is_inverted' field included in the metadata.
  • the 'is inverted' field may be referred to as an 'orientation_of_triangle' field.
  • a triangular patch 1712 illustrated in FIG. 17B which has an inverted triangular, illustrates a case where the value of the 'is inverted' field is 1.
  • the 'is_inverted' field indicating whether the triangular patch is inverted may be included in the metadata when there is no predefined pattern for the patch, that is, when the value of a 'predefined_pattern' field is 0.
  • FIG. 17C illustrates projection in a case where a patch in a region on a planar video is an isosceles triangle.
  • the region on the planar video may be mapped to a triangle surface as follows.
  • a pixel (x, y) 1722 in an isosceles-triangular region 1720, which has a width of w and a height of h, of the planar video, may be projected onto a point (x', y') in an isosceles-triangular patch 1730, which has a width of w' and a height of h', of a 3D geometry.
  • the projected point (x', y') may be defined by Equation 5.
  • FIG. 18 illustrates a method for projecting a region on a planar video into an area of a pyramid when a geometry is the pyramid.
  • a geometry is a pyramid
  • four triangular patches may be mapped to respective regions of a planar video, and one rectangular patch may be mapped to a region of the planar video.
  • the regions to be mapped to the respective patches may be arranged as illustrated in FIG. 18 .
  • a patch corresponding to one left surface 1800 of the pyramid may be mapped to one region 1802 of the planar video.
  • the position and size of each region may be indicated by an 'OmniMediaTextureMappingMetadataSample' or 'OmnidirectionalMediaTextureMappingMetadataSample' box.
  • information on a region mapped to a surface of the pyramid may be determined as in Table 1.
  • Table 1 Surface_id Position of top-left corner Width Height 2 (0,0) W v (h v -h 1 )/2 3 (0, (h v +h 1 )/2) W v (h v -h 1 )/2 4 (0, (h v -h 1 )/2) (w v -w 1 )/2 h 1 5 ((w v +w 1 )/2),(h v -h 1 )/2) (w v -w 1 )/2 h 1
  • surface IDs (surface_id) of 2, 3, 4, and 5 indicate a top surface, a bottom surface, a left surface, and a right surface of the pyramid, respectively.
  • w v denotes the width of the video
  • w 1 denotes the width of a region to be mapped to a front surface of the pyramid
  • h v denotes the height of the video
  • h 1 denotes the height of the region to be mapped to the front surface of the pyramid.
  • a geometry when a geometry is a truncated pyramid, four trapezoidal patches may be mapped to respective regions of a planar video, and two rectangular patches may be mapped to respective regions of the planar video.
  • a 'ftyp' box is used to identify the type of an Omnidirectional Media Application Format (OMAF) to which a file structure of an omnidirectional video according to the present disclosure conforms.
  • OMAF Omnidirectional Media Application Format
  • a brand to identify a file conformant to the present disclosure may be 'omf1 (omnidirectional media application format version 1)' or 'om01 (360-degrees content)'.
  • the metadata 'OmnidirectionalMediaTextureMappingMetadataSample' provides omnidirectional video information about a geometry type and a projection type and may have an entry type, for example, 'ovmi'.
  • An 'is_stereoscopic' field indicates whether stereoscopic media rendering is used.
  • a video in a reference track (unit omnidirectional video currently being transmitted and received) may be divided into two portions. The two divided portions separately provide different texture data for the left eye and the right eye depending on the composition type specified by a 'stereoscopic_type' field.
  • An 'is_plane' field indicates whether a current sample is a 2D planar image/video.
  • the current sample is a 2D planar image/video and does not need to be projected onto the surface of a rendering model. Otherwise, the current sample may be projected onto the surface of the rendering model, in which the rendering model is determined by 'rendering_model'.
  • a 'use_fine_prediction_of angle' field indicates whether a 'scale_of angle' field is present. When the value of the 'use_fine_prediction_of_angle' field is 0, there is no 'scale_of angle' field.
  • An 'is_default_front' field indicates whether 'center_x' and 'center_y' or 'center_pitch' and 'center_yaw' are present.
  • An 'is_multiple_regions' field indicates whether a video is divided into a plurality of regions.
  • the respective regions may have different resolutions and may be independently mapped onto a corresponding area of a geometry.
  • the value of the 'is_multiple_regions' field is 1, the video in the reference track is divided into a plurality of non-overlapping regions.
  • each area provides texture data for a specific area of the geometry. Otherwise (when the value is not 1), the entire image data in a 2D image is mapped onto the surface of the geometry without being separated.
  • a 'denom' field indicates a denominator for normalization.
  • a 'stereoscopic_type' field may indicate the composition type of a stereoscopic video in the reference track. Examples of the composition type of the stereoscopic video are illustrated below in Table 3. [Table 3] Value Stereoscopic_type 0x00 Reserved 0x01 Side-by-side type 0x02 Top-and-bottom type 0x03-0xFF Reserved
  • a 'geometry_type' field indicates a geometry type for rendering omnidirectional media.
  • a mathematical expression for each geometry type has been described above with drawings of various geometries.
  • Geometry_type 0x00 Reserved 0x01 Sphere 0x02 Squished Sphere 0x03 Cylinder 0x04 Cube 0x05 Platonic Solid: Triangle-based 0x06-0xFF Reserved
  • a 'sphere_to_plane_projection_type' field indicates a method to be used to map texture in the video in the reference track onto the geometry for rendering of the omnidirectional media when the value of the 'geometry_type' field is '0x01' (i.e., a sphere).
  • a mathematical expression for a projection method for each particular geometry type has been described above with drawings of various geometries.
  • a 'squish_start_pitch_top' field and a 'squish_start_pitch_bottom' field respectively indicate the top pitch angle and the bottom pitch angle of a sphere when squishing is applied.
  • Top and bottom portions of the sphere indicated by the fields may be squished at a ratio given by the value of a 'squish_ratio' field.
  • the 'squish_ratio' field indicates the squishing ratio of the spherical geometry to be squished.
  • the top and bottom portions of the sphere may be squished by the field according to the present disclosure.
  • a 'radius' field indicates the radius of a circular region for top and bottom surfaces of a cylinder.
  • a texture region for the top surface may have a circular shape having the radius indicated by the field and may be located at the top-right corner of the video.
  • a texture region for the bottom surface may have a circular shape having the radius indicated by the field and may be located at the bottom-right corner of the video.
  • a 'scale_of angle' field indicates the resolution of an angle.
  • the value of the 'scale_of_angle' field is inferred to be 1.
  • An 'is_2D_center' field indicates by default whether the center point of a view is set to 'center_x' and 'center_y' or 'center_pitch' and 'center_yaw'.
  • the center point of a front view is set to (center_x, center_y) in a 2D image/video. Otherwise (when the value of the field is not 1), the center point is set to 3D coordinates of a point where a pitch angle and a yaw angle are (center_pitch/scale_of_angle, center_yaw/scale_of_angle).
  • a 'center_x' field and a 'center_y' field respectively indicate the horizontal coordinate and the vertical coordinate of a pixel to be rendered at the center of the front of a geometry.
  • pixel data of the 'center_x' field and the 'center_y' field may be rendered at a default point according to the geometry type.
  • the values of center x and center_y are inferred to be 0.
  • the values of center_x and center_y are smaller than the width and the height of the image/video.
  • a 'center_pitch' field and a 'center_yaw' field respectively indicate the pitch angle and the yaw angle of coordinates of a point at which the center pixel of the video is rendered.
  • the values of the fields may be inferred to be 0.
  • center_pitch/scale_of angle and center_yaw/scale_of_angle are smaller than 360.
  • a 'num of regions' field indicates the number of regions into which the video in the reference track is divided.
  • the video in the reference track may be divided into as many non-overlapping regions as the value of the field, and each region may be separately mapped to a specific area of a geometry.
  • An 'is_available' field indicates whether video data for a current region is present in the reference track.
  • the value of the 'is_available' field is 1, an area corresponding to the current region may be rendered using the video in the reference track.
  • An 'is_static' field indicates whether texture mapping of a current region changes with time.
  • the mapping does not change during the duration of the entire video in the reference track.
  • the mapping changes with time.
  • a 'region top_left_x' field and a 'region top_left_y' field respectively indicate the horizontal coordinate and the vertical coordinate of the top-left corner of a rectangular region of the video in the reference track.
  • a 'region_width' field and a 'region height' field respectively indicate the width and the height of a rectangular region of the video in the reference track.
  • a 'sphere_pitch_start' field and a 'sphere_pitch_end' field respectively indicate the starting pitch angle and the ending pitch angle of a particular area of a sphere.
  • a 'sphere_yaw_start' field and a 'sphere_yaw _end' field respectively indicate the starting yaw angle and the ending yaw angle of a particular area of a sphere.
  • a 'cylinder_pitch_start' field and a 'cylinder_pitch_end' field respectively indicate the starting pitch angle and the ending pitch angle of a particular area of a sphere.
  • a 'cylinder_start height' field and a 'cylinder_end height' field respectively indicate the normalized starting height and the normalized ending height of a particular area of a cylindrical geometry. Values obtained by dividing the 'cylinder_start height' field and the 'cylinder_end_height' field by 'denom' are smaller than 1.
  • a current region indicates a top area or bottom area, and the 'cylinder_pitch_start', 'cylinder_pitch_end', 'cylinder_start height', and 'cylinder_end_height' fields do not exist.
  • a 'cube_surface_id' field indicates the identifier of a surface of a cube (regular hexahedron) as illustrated below in Table 6.
  • Table 6 Value Cube_surface_id 0x00 Reserved 0x01 Front 0x02 Top 0x03 Bottom 0x04 Left 0x05 Right 0x06 Back 0x07-0xFF Reserved
  • a 'cube_area_top_left_x' field and a 'cube_area_top_left_y' field respectively indicate the normalized horizontal coordinate and the normalized vertical coordinate of the top-left corner of a particular area of a cube geometry. Values obtained by dividing the 'cube_area_top_left_x' field and the 'cube_area_top_left_y' field by 'denom' are smaller than 1.
  • a 'cube_area_width' field and a 'cube_area_height' field respectively indicate the normalized width and the normalized height of a particular region of a cube geometry. Values obtained by dividing the 'cube_area_width' field and the 'cube_area_height' field by 'denom' are smaller than 1.
  • a 'pyramid_surface_id' field indicates the identifier of a surface of a pyramid as illustrated below in Table 7.
  • Value Pyramid_surface_id 0x00 Reserved 0x01 Front 0x02 Top 0x03 Bottom 0x04 Left 0x05 Right 0x06-0xFF Reserved
  • a 'pyramid_front_area_top_left_x' field and a 'pyramid_front_area_top_left_y' field respectively indicate the horizontal coordinate and the vertical coordinate of the top-left corner of a normalized area of a pyramid. Values obtained by dividing the 'pyramid_front_area_top_left_x' field and the 'pyramid_front_area_top_left_y' field by 'denom' are smaller than 1.
  • a 'pyramid_front_area_width' field and a 'pyramid_front_area_height' field respectively indicate the normalized width and the normalized height of a particular region of a pyramid geometry. Values obtained by dividing the 'pyramid_front_area_width' field and the 'pyramid_front_area_height' field by 'denom' are smaller than 1.
  • FIG. 19 illustrates another method for projecting a region on a planar video into an area of a pyramid when a geometry is the pyramid.
  • a region 1900 of a planar video is mapped to an area 1910, which is one side surface of a pyramid.
  • a 'pyramid_side_area_top_x' field and a 'pyramid_side_area_top_y' field respectively indicate the horizontal coordinate and the vertical coordinate of a top vertex 1912 of the normalized area 1910 in a side surface of the pyramid.
  • a 'pyramid_side_area_bottom_left_x' field and a 'pyramid_side_area_bottom_left_y' field respectively indicate the horizontal coordinate and the vertical coordinate of a bottom-left vertex 1914 of the normalized area 1910 in the side surface of the pyramid. Values obtained by dividing the 'pyramid_side_area_bottom_left_x' field and the 'pyramid_side_area_bottom_left_y' field by 'denom' are smaller than 1.
  • a 'pyramid_side_area_bottom_right_x' field and a 'pyramid_side_area_bottom_right_y' field respectively indicate the horizontal coordinate and the vertical coordinate of a bottom-right vertex 1916 of the normalized area 1910 in the side surface of the pyramid. Values obtained by dividing the 'pyramid_side_area_bottom_right_x' field and the 'pyramid_side_area_bottom_right_y' field by 'denom' are smaller than 1.
  • An 'is_stereoscopic' field indicates whether stereoscopic media rendering is used.
  • a video in a reference track (unit omnidirectional video currently being transmitted and received) may be divided into two portions for the two eyes.
  • the two divided portions separately provide different texture data for the left eye and the right eye depending on the composition type specified by a 'stereoscopic_type' field.
  • An 'is_default_front' field may indicate whether a 'center_pitch' field and a 'center_yaw' field are present.
  • An 'entire_volume' field indicates whether the video in the reference track covers the entire volume of a geometry.
  • the value of the 'entire_volume' field is 1, the entire volume of the geometry may be rendered by the video in the reference track.
  • the texture of a particular region of the geometry may be provided by a means other than the video in the reference track.
  • a 'static' field indicates whether texture mapping (mapping between a region and a patch) changes with time.
  • the mapping may not change during the entire video in the reference track.
  • the mapping may change with time.
  • a 'static_top' field indicates whether texture data is provided instead of the video in the reference track.
  • the value of the 'static_top' field is 1, image data used as a top area of the geometry may be provided.
  • a 'static_bottom' field indicates whether texture data is provided instead of the video in the reference track.
  • the value of the 'static_bottom' field is 1, image data used as a bottom area of the geometry may be provided.
  • a 'stereoscopic type' field may indicate the composition type of a stereoscopic video in the reference track.
  • composition type of the stereoscopic video is illustrated above in Table 3.
  • a 'geometry_type' field indicates a geometry type for rendering an omnidirectional video.
  • a mathematical expression for each geometry type has been described above with drawings of various geometries.
  • a 'squish_start_pitch_top' field and a 'squish_start_pitch_bottom' field respectively indicate the top pitch angle and the bottom pitch angle of a sphere when squishing is applied.
  • Top and bottom portions of the sphere indicated by the 'squish_start_pitch_top' and 'squish_start_pitch_bottom' fields may be squished at a ratio given by the value of a 'squish ratio' field.
  • a 'squish_ratio_top' field indicates the squishing ratio of the spherical geometry to be squished.
  • the top portion of the sphere may be squished by the 'squish_ratio_top' field according to the present disclosure.
  • the 'squish_ratio_bottom' field indicates the squishing ratio of the spherical geometry to be squished.
  • the bottom portion of the sphere may be squished by the 'squish_ratio_bottom' field according to the present disclosure.
  • a 'radius' field indicates the radius of a circular patch for top and bottom surfaces of a cylinder.
  • a region for the top surface may have a circular shape having the radius indicated by the field and may be located at the top-right corner of the planar video.
  • a region for the bottom surface may have a circular shape having the radius indicated by the field and may be located at the bottom-right corner of the planar video.
  • a 'num_triangles' field may indicate a platonic solid type for rendering the omnidirectional video.
  • a mathematical expression for each geometry type has been described above.
  • Table 9 illustrates the number of triangles according to the omnidirectional video.
  • Num_triangles 0x00 Reserved 0x01 Octahedron: 8 sides 0x02 Icosahedrons: 20 sides 0x03-0xFF Reserved
  • a 'center_pitch' field and a 'center_yaw' field respectively indicate the pitch angle and the yaw angle of coordinates of a point at which the center pixel of the video is rendered.
  • the values of the fields may be inferred to be 0.
  • a 'center_pitch_offset' field and a 'center_yaw_offset' field respectively indicate an offset for the pitch angle and an offset for the yaw angle of coordinates of a point at which the center pixel of the omnidirectional video is rendered.
  • 'center_pitch_offset' + 'center_pitch' and 'center_yaw_offset' + 'center_yaw' indicate the center point (x, y) of a current sample.
  • a 'num_of_regions' field indicates the number of regions into which the planar video in the reference track is divided.
  • the video in the reference track may be divided into as many non-overlapping regions as the value of the 'num_of_regions' field, and each region may be separately mapped to a specific area of a geometry.
  • the value of the 'geometry_type' field is 'Platonic Solid'
  • the value of the 'num_of_regions' field may be the same as the value of the 'num_triangles' field.
  • a 'region_top_left_x' field and a 'region_top_left_y' field respectively indicate the horizontal coordinate and the vertical coordinate of the top-left corner of a rectangular region in the reference track.
  • a 'region width' field and a 'region_height' field respectively indicate the width and the height of the rectangular region in the reference track.
  • a 'pitch_start' field and a 'pitch_end' field respectively indicate the starting pitch angle and the ending pitch angle of a particular area of the geometry.
  • a 'yaw_start' field and a 'yaw_end' field represent the starting yaw angle and the ending yaw angle of a particular area of the geometry.
  • a 'cylinder_surface_id' field indicates the identifier of a surface of a cylinder to be described below.
  • Table 11 illustrates examples of a cylinder surface identifier.
  • Cylinder_surface_id 0x00 Reserved 0x01 Top 0x02 Bottom 0x03 Side 0x04-0xFF Reserved
  • a 'height_start' field and a 'height_end' field respectively indicate the normalized starting height and the normalized ending height of a particular area of a cylindrical geometry.
  • a 'cube_surface_id' field indicates the identifier of a surface (patch) of a cube (regular hexahedron) to be described below.
  • An 'area_top_left_x' field and an 'area_top_left_y' field respectively indicate the horizontal coordinate and the vertical coordinate of the top-left corner of an area on the surface of a cube when a geometry is the cube.
  • An 'area_width' field and an 'area_height' field respectively indicate the width and the height of an area on the surface of the cube.
  • a 'predefined_pattern' field indicates a pattern for a triangle position.
  • 'triangle_surface_id', 'is_inverted', 'area_top_left_x', 'area top_left_y', 'area width', and 'area_height' fields for each triangle exist.
  • the value of the 'predefined_pattern' field is not 0, the value of the 'predefined_pattern' field specifies a predefined position for all triangles.
  • a 'triangle_surface_id' field indicates the identifier of a surface of a triangle for a platonic solid.
  • a 'relative_position' field indicates whether the 'area_top_left_x' field and the 'area_top_left_y' field exist or indicates whether an 'area_top_left_x_diff' field and an 'area_top_left_y_diff' field exist.
  • a 'ref_triangle_surface_id' field may indicate the identifier of a surface of a reference triangle for a platonic solid.
  • the 'area top_left_x_diff' field and the 'area_top_left_y_diff' field respectively indicate a horizontal difference and a vertical difference between the top-left corner of a reference triangle specified by 'ref_triangle_surface_id' and the top-left corner of a current triangle (a surface of the triangle mapped to a region on the planar video).
  • the 'is_inverted' field indicates the shape (e.g., rotation, inversion, or flip) of a triangle as described below.
  • the 'is inverted' field may be referred to as an 'orientation_of_triangle' field.
  • FIG. 20 illustrates a method in which a transmitter transmits metadata of an omnidirectional video according to the present disclosure.
  • the transmitter projects an omnidirectional video onto a planar video using a 3D geometry corresponding to the omnidirectional video (2000).
  • the transmitter transmits metadata including type information of the 3D geometry, information for a region on the projected planar video, and information for an area on the omnidirectional video which is mapped to the region (2002).
  • the information for the region on the planar video included in the metadata may include a coordinate to specify the position of the region in the planar video.
  • the region may have a rectangular shape, and thus the information for the region on the planar video may further include the width and the height of the region.
  • FIG. 21 illustrates a method in which a receiver receives metadata of an omnidirectional video according to the present disclosure.
  • the receiver receives metadata including type information of a 3D geometry corresponding to an omnidirectional video, information for a region on a planar video, and information for an area on the omnidirectional video which is mapped to the region (2100).
  • the receiver projects the planar video onto the omnidirectional video using the 3D geometry corresponding to the omnidirectional video (2012).
  • the information for the region on the planar video included in the metadata may include a coordinate to specify the position of the region in the planar video.
  • the region may have a rectangular shape, and thus the information on the region of the planar video may further include the width and the height of the region.
  • FIG. 22 illustrates the configuration of a transmission apparatus according to the present disclosure.
  • the transmission apparatus 2200 may include a transceiver 2205 to perform data transmission and reception with a reception apparatus and a controller 2210 to control all operations of the transmission apparatus 2200. All operations of a transmitter described above in the present disclosure may be construed as being performed under the control of the controller 2210.
  • the controller 2210 and the transceiver 2205 are not necessarily configured as separate devices but may be configured as one component, such as a single chip.
  • FIG. 23 illustrates the configuration of a reception apparatus according to the present disclosure.
  • the reception apparatus 2300 may include a transceiver 2305 to perform data transmission and reception with a transmission apparatus and a controller 2310 to control all operations of the reception apparatus 2300. All operations of a receiver described above in the present disclosure may be construed as being performed under the control of the controller 2310.
  • the controller 2310 and the transceiver 2305 are not necessarily configured as separate devices but may be configured as one component, such as a single chip.
  • FIGS. 1 to 26 are not intended to limit the scope of the present disclosure. That is, all components or stages of operations described in FIGS. 1 to 26 are not construed as being essential elements to achieve the present disclosure, but the present disclosure may be implemented with only some components without departing from the essence of the present disclosure.
  • the above described operations may be implemented by providing a memory device storing corresponding program codes in any constituent unit of a server or UE apparatus in a communication system. That is, the controller of the base station or UE may perform the above described operations by reading and executing the program code stored in the memory device by means of a processor or a Central Processing Unit (CPU).
  • a processor or a Central Processing Unit (CPU).
  • the entity, the function, the base station, the load manager, various structural elements of the terminal, modules and the like may be operated by using a hardware circuit, e.g, a complementary metal oxide semiconductor based logic circuit, firmware, software, and/or a combination of hardware and the firmware and/or software embedded in a machine readable medium.
  • a hardware circuit e.g, a complementary metal oxide semiconductor based logic circuit, firmware, software, and/or a combination of hardware and the firmware and/or software embedded in a machine readable medium.
  • various electric configurations and methods may be carried out by using electric circuits such as transistors, logic gates, and an Application Specific Integrated Circuit (ASIC).
  • ASIC Application Specific Integrated Circuit

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Geometry (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Library & Information Science (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Processing Or Creating Images (AREA)

Claims (11)

  1. Verfahren zum Übertragen von Metadaten eines omnidirektionalen Bildes durch eine Übertragungsvorrichtung, wobei das Verfahren Folgendes umfasst:
    Erzeugen von Metadaten in Bezug auf ein omnidirektionales Bild, wobei ein planares Bild basierend auf der Projektion des omnidirektionalen Bildes erzeugt wird; und Übertragen der erzeugten Metadaten,
    wobei die Metadaten Folgendes umfassen:
    Informationen über einen Geometrietyp zum Rendern eines 3-dimensionalen, 3D, Bildes basierend auf dem planaren Bild, wobei die Informationen über den Geometrietyp eine Kugel, einen Kubus, einen Zylinder oder eine gequetschte Kugel angeben, gekennzeichnet dadurch, dass besagte Metadaten ferner Folgendes umfassen:
    Informationen über die Anzahl der nicht überlappenden Bereiche, in die das planare Bild unterteilt ist,
    Informationen zum Spezifizieren von jedem der Bereiche.
  2. Verfahren nach Anspruch 1, wobei die Informationen zum Spezifizieren von jedem der Bereiche Informationen über eine Position von jedem der Bereiche in dem planaren Bild umfassen.
  3. Verfahren nach Anspruch 1, wobei die Metadaten ferner Informationen über eine Oberfläche des 3D-Bildes umfassen, die in jedem der Bereiche abgebildet ist.
  4. Verfahren nach Anspruch 1, wobei die Informationen zum Spezifizieren von jedem der Bereiche Informationen über eine Breite und eine Höhe von jedem der Bereiche umfassen.
  5. Verfahren nach Anspruch 1, wobei, wenn die Informationen über den Geometrietyp die gequetschte Kugel angeben, die Metadaten ferner Informationen über den oberen Steigungswinkel der gequetschten Kugel, Informationen über den unteren Steigungswinkel der gequetschten Kugel, Informationen über das obere Quetschverhältnis der gequetschten Kugel und Informationen über das untere Quetschverhältnis der gequetschten Kugel umfassen.
  6. Verfahren nach Anspruch 1, wobei, wenn die Informationen über den Geometrietyp den Zylinder angeben, die Metadaten ferner Informationen über den Radius des Zylinders umfassen.
  7. Verfahren zum Empfangen von Metadaten eines omnidirektionalen Bildes durch eine Empfangsvorrichtung, wobei das Verfahren Folgendes umfasst:
    Empfangen von Metadaten in Bezug auf ein omnidirektionales Bild,
    Verarbeiten eines planaren Bildes, das basierend auf der Projektion des omnidirektionalen Bildes erzeugt wird, basierend auf den empfangenen Metadaten,
    wobei die Metadaten Folgendes umfassen:
    Informationen über den Geometrietyp zum Rendern eines 3-dimensionalen, 3D, Bildes basierend auf dem planaren Bild, wobei die Informationen über den Geometrietyp eine Kugel, einen Kubus, einen Zylinder oder eine gequetschte Kugel angeben, gekennzeichnet dadurch, dass besagte Metadaten ferner Folgendes umfassen:
    Informationen über die Anzahl der nicht überlappenden Bereiche, in die das planare Bild unterteilt ist,
    Informationen zum Spezifizieren von jedem der Bereiche.
  8. Verfahren nach Anspruch 7, wobei die Informationen zum Spezifizieren von jedem der Bereiche Informationen über eine Position von jedem der Bereiche in dem planaren Bild umfassen.
  9. Verfahren nach Anspruch 7, wobei die Metadaten ferner Informationen über eine Oberfläche des 3D-Bildes umfassen, die in jedem der Bereiche abgebildet ist.
  10. Übertragungsvorrichtung zum Übertragen von Metadaten eines omnidirektionalen Bildes, wobei die Vorrichtung Folgendes umfasst:
    einen Transceiver; und
    eine Steuerung, die mit dem Transceiver gekoppelt ist, wobei die Steuerung konfiguriert ist, um gemäß einem der Verfahren der Ansprüche 1 bis 6 betrieben zu werden.
  11. Empfangsvorrichtung zum Empfangen von Metadaten eines omnidirektionalen Videos, wobei die Vorrichtung Folgendes umfasst:
    einen Transceiver; und
    eine Steuerung, die mit dem Transceiver gekoppelt ist, wobei die Steuerung konfiguriert ist, um gemäß einem der Verfahren der Ansprüche 7 bis 9 betrieben zu werden.
EP17753516.8A 2016-02-17 2017-02-17 Verfahren zum senden und empfangen von metadaten eines omnidirektionalen bildes Active EP3419301B1 (de)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201662296351P 2016-02-17 2016-02-17
US201662340683P 2016-05-24 2016-05-24
PCT/KR2017/001791 WO2017142355A1 (ko) 2016-02-17 2017-02-17 전방향성 영상의 메타데이터를 송수신하는 기법

Publications (3)

Publication Number Publication Date
EP3419301A1 EP3419301A1 (de) 2018-12-26
EP3419301A4 EP3419301A4 (de) 2019-02-27
EP3419301B1 true EP3419301B1 (de) 2021-08-18

Family

ID=59625244

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17753516.8A Active EP3419301B1 (de) 2016-02-17 2017-02-17 Verfahren zum senden und empfangen von metadaten eines omnidirektionalen bildes

Country Status (4)

Country Link
US (1) US11069025B2 (de)
EP (1) EP3419301B1 (de)
KR (1) KR20170096975A (de)
WO (1) WO2017142355A1 (de)

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102506480B1 (ko) * 2016-06-14 2023-03-07 삼성전자주식회사 영상 처리 장치 및 그 영상 처리 방법
KR102352933B1 (ko) * 2016-09-09 2022-01-20 삼성전자주식회사 3차원 이미지를 프로세싱하기 위한 방법 및 장치
US10791316B2 (en) 2017-03-28 2020-09-29 Samsung Electronics Co., Ltd. Method for transmitting data about three-dimensional image
KR20190113651A (ko) * 2018-03-27 2019-10-08 주식회사 케이티 비디오 신호 처리 방법 및 장치
WO2019193011A1 (en) * 2018-04-05 2019-10-10 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Region description for 360 or spherical video
WO2019203456A1 (ko) * 2018-04-15 2019-10-24 엘지전자 주식회사 복수의 뷰포인트들에 대한 메타데이터를 송수신하는 방법 및 장치
US11012657B2 (en) * 2018-06-08 2021-05-18 Lg Electronics Inc. Method for processing overlay in 360-degree video system and apparatus for the same
KR20190140387A (ko) * 2018-06-11 2019-12-19 에스케이텔레콤 주식회사 360도 영상의 인터 예측 방법 및 이를 이용한 영상 복호화 장치
WO2019240425A1 (ko) 2018-06-11 2019-12-19 에스케이텔레콤 주식회사 인터 예측 방법 및 영상 복호화 장치
WO2019245303A1 (en) * 2018-06-22 2019-12-26 Lg Electronics Inc. Method for transmitting 360-degree video, method for receiving 360-degree video, apparatus for transmitting 360-degree video, and apparatus for receiving 360-degree video
KR102656614B1 (ko) * 2018-10-01 2024-04-12 삼성전자주식회사 비디오 컨텐츠를 전송하는 방법 및 장치 및 비디오 컨텐츠를 수신하는 방법 및 장치
EP3691249A1 (de) * 2019-01-29 2020-08-05 Koninklijke Philips N.V. Bildsignal, das eine szene repräsentiert
US20210327119A1 (en) * 2020-04-17 2021-10-21 Occipital, Inc. System for Generating a Three-Dimensional Scene Reconstructions
US11394932B2 (en) 2020-06-03 2022-07-19 Honeywell International Inc. System and method for auto selecting a video for display on a mobile device based on the proximity of the mobile device relative to the video source

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010015751A1 (en) 1998-06-16 2001-08-23 Genex Technologies, Inc. Method and apparatus for omnidirectional imaging
US6754400B2 (en) 2001-02-06 2004-06-22 Richard Wilson, Jr. System and method for creation, processing and visualization of omni-directional images
US7308131B2 (en) * 2002-12-03 2007-12-11 Ntt Docomo, Inc. Representation and coding of panoramic and omnidirectional images
US7463280B2 (en) * 2003-06-03 2008-12-09 Steuart Iii Leonard P Digital 3D/360 degree camera system
KR100732958B1 (ko) 2004-08-13 2007-06-27 경희대학교 산학협력단 20면체 파노라마 영상의 부호화 및 복호화를 위한 방법 및장치
US7990394B2 (en) * 2007-05-25 2011-08-02 Google Inc. Viewing and navigating within panoramic images, and applications thereof
US9692965B2 (en) * 2011-04-25 2017-06-27 Mitsuo Hayashi Omnidirectional image editing program and omnidirectional image editing apparatus
US20120313938A1 (en) 2011-06-09 2012-12-13 Sohrab Mohajerin Interface and Mechanism for a 3D Geometry Transformer and Translator

Also Published As

Publication number Publication date
KR20170096975A (ko) 2017-08-25
US11069025B2 (en) 2021-07-20
US20190197661A1 (en) 2019-06-27
WO2017142355A1 (ko) 2017-08-24
EP3419301A1 (de) 2018-12-26
EP3419301A4 (de) 2019-02-27

Similar Documents

Publication Publication Date Title
EP3419301B1 (de) Verfahren zum senden und empfangen von metadaten eines omnidirektionalen bildes
US11758187B2 (en) Methods, devices and stream for encoding and decoding volumetric video
EP3489891B1 (de) Verfahren und vorrichtung zur verarbeitung von dreidimensionalen bilddaten
CN109644279B (zh) 用于用信号通知360度视频信息的方法和系统
US20210195162A1 (en) Method, apparatus and stream for volumetric video format
US11025955B2 (en) Methods, devices and stream for encoding and decoding volumetric video
US10523980B2 (en) Method, apparatus and stream of formatting an immersive video for legacy and immersive rendering devices
CN111264058B (zh) 用于对三自由度和体积兼容视频流进行编码和解码的方法、设备
CN110073662A (zh) 全景视频的建议视口指示
KR20200096575A (ko) 3차원 객체들을 나타내는 포인트 클라우드를 인코딩하기 위한 방법 및 장치
JP7499182B2 (ja) ボリュメトリックビデオフォーマット用の方法、装置及びストリーム
EP3562159A1 (de) Verfahren, vorrichtung und stream für volumenvideoformat
US20220343549A1 (en) A method and apparatus for encoding, transmitting and decoding volumetric video
EP4005202B1 (de) Verfahren und vorrichtung zur abgabe eines volumetrischen videoinhalts
CN113228658B (zh) 沉浸式视频比特流处理
US20230217006A1 (en) A method and apparatuses for delivering a volumetric video content
EP3873095A1 (de) Vorrichtung, verfahren und computerprogramm für omnidirektionales video

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20180917

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

A4 Supplementary search report drawn up and despatched

Effective date: 20190124

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 21/81 20110101ALI20190118BHEP

Ipc: G06T 17/10 20060101ALI20190118BHEP

Ipc: G06T 17/05 20110101ALI20190118BHEP

Ipc: H04N 21/84 20110101AFI20190118BHEP

Ipc: H04N 19/597 20140101ALI20190118BHEP

Ipc: H04N 9/31 20060101ALI20190118BHEP

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20201223

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20210331

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017044307

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Ref country code: AT

Ref legal event code: REF

Ref document number: 1422719

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210915

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20210818

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1422719

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210818

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211118

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211220

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211118

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211119

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017044307

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20220519

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602017044307

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20220228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220217

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220228

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220217

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220901

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20170217

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210818

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240122

Year of fee payment: 8