WO2019077199A1 - Appareil, procédé, et programme d'ordinateur pour vidéo volumétrique - Google Patents

Appareil, procédé, et programme d'ordinateur pour vidéo volumétrique Download PDF

Info

Publication number
WO2019077199A1
WO2019077199A1 PCT/FI2018/050730 FI2018050730W WO2019077199A1 WO 2019077199 A1 WO2019077199 A1 WO 2019077199A1 FI 2018050730 W FI2018050730 W FI 2018050730W WO 2019077199 A1 WO2019077199 A1 WO 2019077199A1
Authority
WO
WIPO (PCT)
Prior art keywords
projection surface
projection
auxiliary
occluded
main
Prior art date
Application number
PCT/FI2018/050730
Other languages
English (en)
Inventor
Vinod Kumar Malamal Vadakital
Sebastian Schwarz
Original Assignee
Nokia Technologies Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Technologies Oy filed Critical Nokia Technologies Oy
Priority to EP18867354.5A priority Critical patent/EP3698332A4/fr
Publication of WO2019077199A1 publication Critical patent/WO2019077199A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/001—Model-based coding, e.g. wire frame
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194—Transmission of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object

Definitions

  • the present invention relates to an apparatus, a method and a computer program for content dependent projection for volumetric video coding and decoding.
  • a video coding system may comprise an encoder that transforms an input video into a compressed representation suited for storage/transmission and a decoder that can uncompress the compressed video representation back into a viewable form.
  • the encoder may discard some information in the original video sequence in order to represent the video in a more compact form, for example, to enable the storage/transmission of the video information at a lower bitrate than otherwise might be needed.
  • Volumetric video data represents a three-dimensional scene or object and can be used as input for virtual reality (VR), augmented reality (AR) and mixed reality (MR) applications.
  • Such data describes the geometry, e.g. shape, size, position in three-dimensional (3D) space, and respective attributes, e.g. colour, opacity, reflectance and any possible temporal changes of the geometry and attributes at given time instances.
  • Volumetric video is either generated from 3D models through computer-generated imagery (CGI), or captured from real-world scenes using a variety of capture solutions, e.g. multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible.
  • CGI computer-generated imagery
  • Typical representation formats for such volumetric data are triangle meshes, point clouds (PCs), or voxel arrays.
  • Temporal information about the scene can be included in the form of individual capture instances, i.e. "frames" in 2D video, or other means, e.g. position of an object as a function of time.
  • 3D data acquisition devices has enabled reconstruction of highly detailed volumetric video representations of natural scenes.
  • Infrared, lasers, time-of- flight and structured light are all examples of devices that can be used to construct 3D video data.
  • Representation of the 3D data depends on how the 3D data is used.
  • Dense Voxel arrays have been used to represent volumetric medical data.
  • polygonal meshes are extensively used.
  • Point clouds on the other hand are well suited for applications such as capturing real world 3D scenes where the topology is not necessarily a 2D manifold.
  • Another way to represent 3D data is coding this 3D data as a set of texture and depth maps as is the case in the multi-view plus depth. Elevation maps and multilevel surface maps are other techniques to represent 3D data which are closely related to the multi-view plus depth technique
  • the reconstructed 3D scene may contain tens or even hundreds of millions of points.
  • One way to compress a time-varying volumetric scene/object is to project 3D surfaces on to some number of pre-defined 2D planes.
  • Regular 2D video compression algorithms can then be used to compress various aspects of the projected surfaces.
  • a time-varying 3D point cloud with spatial and texture coordinates, can be mapped into a sequence of at least two sets of planes, where one of the two sets carry the texture data and the other carries the distance of the mapped 3D surface points from the projection surfaces.
  • the 3D surface structure may be so complex that the projection process might not be able to map all the scene/object texture data on to the selected projection surfaces.
  • the primary reason for this is the occluding of parts of the 3D scene/object surfaces by other surfaces that are closer to the chosen projection surfaces.
  • a volumetric video for example dynamic point clouds, arrays of voxels or mesh models or a combination of such, may be projected onto a number of projection surfaces having simple geometries, for example sphere(s), cylinder(s), cube(s), polyhedron(s) and/or plane(s).
  • a projection surface may be a piece-wise continuous and smooth surface in three-dimensional space. Piece-wise smoothness may be understood so that there are regions of the surface where the direction of the surface normal does not change abruptly (i.e. the values of the coefficients of the surface normal's coordinate components are continuous).
  • a projection surface may comprise pieces of simple geometric surfaces.
  • a projection surface may also evolve (change) over time.
  • the texture and geometry of point clouds, voxel arrays or mesh models may form pixel images, e.g. texture images and depth images (indicative of distance of a point from the projection surface).
  • Such projection surfaces may be unfolded onto two-dimensional (2D) planes, e.g. resulting in a two-dimensional pixel image.
  • Standard 2D video coding may be applied for each projection to code the pixel information resulting from the texture data.
  • relevant projection geometry information comprising e.g. projection surface type, location and orientation of the projection surface in 3D space, and/or size of the projection surface, may be transmitted either in the same bitstream or separately along with the bitstream.
  • the bitstream may be decoded and volumetric video may be reconstructed from decoded 2D projections and projection geometry information.
  • each object in the 3D scene presentation is projected based on its skeleton where different parts of the skeleton are projected to different volumetric shapes e.g. cylinders.
  • Each cylinder has its own size and orientation.
  • criteria may be the amount of motion for the content presented by that cylinder, cylinders which include parts of the scene with high/low frequency components, cylinders which include parts of the scene which are closer/farther to the viewer, cylinders which include region(s) of interest (ROI) such as faces, written text, etc.
  • ROI region(s) of interest
  • a bitstream may be encoded or encapsulated, the bitstream comprising one or more coded texture pictures and one or more coded accompanying additional pictures per texture picture.
  • Each texture picture may represent a 2D projection of 3D data onto a given geometry. Alternatively or in addition, several projections may be frame-packed into a single picture.
  • An additional geometry picture may represent geometry data, e.g. depth.
  • Additional auxiliary pictures may represent further attributes, e.g. surface normal, reflectance and opacity, transmittance and/or refraction of the 3D data.
  • Relevant projection information may be indicated in or along with the bitstream.
  • general projection information may be given in or along the bitstream: number of coded texture pictures or frame-packing information.
  • Projection geometry information may be given for the projections: for example, comprising shape, size, location and orientation of the projection surfaces. Temporal changes in location, orientation and size of such geometries may be encoded, possibly as a function of time. Possible further sub-division of such geometries and resulting changes in geometry information may be given.
  • auxiliary geometry data may be given: bit depth, quantisation, value range and/or inverse values, that is, coding of negative radius or depth values of the geometry picture.
  • auxiliary data i.e. surface normal, reflectance and opacity, transmittance and/or refraction, etc. may be given.
  • a bitstream comprising one or more coded texture picture(s) and one or more coded accompanying geometry pictures and auxiliary pictures per texture picture may be received and decoded.
  • the texture picture represents a two-dimensional projection of three-dimensional data onto a projection surface with a given geometry.
  • a geometry picture may represent geometry data, e.g. depth from the projection surface.
  • Additional auxiliary pictures may represent further attributes, e.g. surface normal, reflectance, opacity, etc. as described earlier.
  • Relevant projection information may also be decoded from or along with a bitstream. From this decoded information, a 3D scene may be reconstructed accordingly.
  • Some embodiments provide a method for encoding and decoding volumetric video information.
  • a method, apparatus and computer program product for volumetric video coding as well as decoding is provided.
  • An apparatus comprises at least one processor and at least one memory, said at least one memory stored with code thereon, which when executed by said at least one processor, causes the apparatus to perform at least:
  • a computer readable storage medium comprises code for use by an apparatus, which when executed by a processor, causes the apparatus to perform:
  • An apparatus comprises:
  • means for defining an auxiliary projection surface for the occluded part means for projecting the occluded part to the auxiliary projection surface; and means for providing an indication of a relationship of the auxiliary projection surface with the main projection surface.
  • An apparatus comprises at least one processor and at least one memory, said at least one memory stored with code thereon, which when executed by said at least one processor, causes the apparatus to perform at least:
  • a computer readable storage medium comprises code for use by an apparatus, which when executed by a processor, causes the apparatus to perform:
  • An apparatus comprises:
  • Further aspects include at least apparatuses and computer program products/code stored on a non-transitory memory medium arranged to carry out the above methods.
  • Fig. 1 shows a system for capturing, encoding, decoding, reconstructing and viewing a three-dimensional scheme
  • FIGs. 2a and 2b show a capture device and a viewing device
  • Figs. 3a and 3b show an encoder and decoder for encoding and decoding texture pictures, geometry pictures and/or auxiliary pictures;
  • Figs. 4a, 4b, 4c and 4d show a setup for forming a stereo image of a scene to a user;
  • Figs. 5a illustrates projection of source volumes in a scene and parts of an object to projection surfaces, as well as determining depth information;
  • Fig. 5b shows an example of projecting an object using a cube map projection format
  • Fig. 6 shows a projection of a source volume to a projection surface, and inpainting of a sparse projection
  • Fig. 7 shows an examples of occlusion of surfaces
  • Fig. 8a shows an example of a cross section of a surface of a three-dimensional object enclosed in a bounding box as viewed from the top of the bounding box;
  • Fig. 8b shows new projection surfaces established to project the occluded surfaces of the three-dimensional object of Fig. 8a;
  • Fig. 8c illustrates calculation of the relative pose of one projection surface with respect to another projection surface, in accordance with an embodiment
  • Figs. 9a and 9b show an example of texture planes and depth planes projected on to seven projection surfaces
  • Fig. 9c illustrates an example of a coded bit stream after layering of respective attributes of the projection surfaces of Figs. 9a and 9b;
  • Figs. 10a and 10b show flow charts for compressing, encoding and decoding of a scene of a volumetric video
  • Fig. 11a illustrates an example of an encoding element
  • Fig. 1 lb illustrates an example of a decoding element.
  • a sparse voxel octree (SVO) describes a volume of a space containing a set of solid voxels of varying sizes. Empty areas within the volume are absent from the tree, which is why it is called "sparse”.
  • a three-dimensional volumetric representation of a scene is determined as a plurality of voxels on the basis of input streams of at least one multicamera device.
  • at least one but preferably a plurality (i.e. 2, 3, 4, 5 or more) of multicamera devices are used to capture 3D video representation of a scene.
  • the multicamera devices are distributed in different locations in respect to the scene, and therefore each multicamera device captures a different 3D video representation of the scene.
  • the 3D video representations captured by each multicamera device may be used as input streams for creating a 3D volumetric representation of the scene, said 3D volumetric representation comprising a plurality of voxels.
  • Voxels may be formed from the captured 3D points e.g.
  • Voxels may also be formed through the construction of the sparse voxel octree. Each leaf of such a tree represents a solid voxel in world space; the root node of the tree represents the bounds of the world.
  • the sparse voxel octree construction may have the following steps: 1) map each input depth map to a world space point cloud, where each pixel of the depth map is mapped to one or more 3D points; 2) determine voxel attributes such as colour and surface normal vector by examining the neighbourhood of the source pixel(s) in the camera images and the depth map; 3) determine the size of the voxel based on the depth value from the depth map and the resolution of the depth map; 4) determine the SVO level for the solid voxel as a function of its size relative to the world bounds; 5) determine the voxel coordinates on that level relative to the world bounds; 6) create new and/or traversing existing SVO nodes until arriving at the determined voxel coordinates; 7) insert the solid voxel as a leaf of the tree, possibly replacing or merging attributes from a previously existing voxel at those coordinates.
  • a volumetric video frame is a complete sparse voxel octree that models the world at a specific point in time in a video sequence.
  • Voxel attributes contain information like colour, opacity, surface normal vectors, and surface material properties. These are referenced in the sparse voxel octrees (e.g. colour of a solid voxel), but can also be stored separately.
  • Point clouds are commonly used data structures for storing volumetric content. Compared to point clouds, sparse voxel octrees describe a recursive subdivision of a finite volume with solid voxels of varying sizes, while point clouds describe an unorganized set of separate points limited only by the precision of the used coordinate values.
  • each frame may produce several hundred megabytes or several gigabytes of voxel data which needs to be converted to a format that can be streamed to the viewer, and rendered in real-time.
  • the amount of data depends on the world complexity and the number of cameras. The larger impact comes in a multi-device recording setup with a number of separate locations where the cameras are recording. Such a setup produces more information than a camera at a single location.
  • Fig. 1 shows a system for capturing, encoding, decoding, reconstructing and viewing a three-dimensional scheme, that is, for 3D video and 3D audio digital creation and playback.
  • the task of the system is that of capturing sufficient visual and auditory information from a specific scene to be able to create a scene model such that a convincing reproduction of the experience, or presence, of being in that location can be achieved by one or more viewers physically located in different locations and optionally at a time later in the future.
  • Such reproduction requires more information that can be captured by a single camera or microphone, in order that a viewer can determine the distance and location of objects within the scene using their eyes and their ears.
  • two camera sources are used.
  • the human auditory system can be able to sense the direction of sound, at least two microphones are used (the commonly known stereo sound is created by recording two audio channels).
  • the human auditory system can detect the cues, e.g. in timing difference of the audio signals to detect the direction of sound.
  • the system of Fig. 1 may consist of three main parts: image sources, a server and a rendering device.
  • a video source SRC1 may comprise multiple cameras CAM1, CAM2, ..., CAMN with overlapping field of view so that regions of the view around the video capture device is captured from at least two cameras.
  • the video source SRC1 may comprise multiple microphones to capture the timing and phase differences of audio originating from different directions.
  • the video source SRC1 may comprise a high-resolution orientation sensor so that the orientation (direction of view) of the plurality of cameras CAM1, CAM2, ..., CAMN can be detected and recorded.
  • the cameras or the computers may also comprise or be functionally connected to means for forming distance information corresponding to the captured images, for example so that the pixels have corresponding depth data.
  • Such depth data may be formed by scanning the depth or it may be computed from the different images captured by the cameras.
  • the video source SRCl comprises or is functionally connected to, or each of the plurality of cameras CAM1, CAM2, ..., CAMN comprises or is functionally connected to a computer processor and memory, the memory comprising computer program code for controlling the source and/or the plurality of cameras.
  • the image stream captured by the video source i.e. the plurality of the cameras, may be stored on a memory device for use in another device, e.g. a viewer, and/or transmitted to a server using a communication interface. It needs to be understood that although a video source comprising three cameras is described here as part of the system, another amount of camera devices may be used instead as part of the system.
  • one or more sources SRC2 of synthetic imagery may be present in the system, comprising a scene model. Such sources may be used to create and transmit the scene model and its development over time, e.g. instantaneous states of the model.
  • the model can be created or provided by the source SRCl and/or SRC2, or by the server SERVER. Such sources may also use the model of the scene to compute various video bitstreams for transmission.
  • One or more two-dimensional video bitstreams may be computed at the server SERVER or a device RENDERER used for rendering, or another device at the receiving end.
  • the viewer may see a three- dimensional virtual world as described in the context of Figs 4a— 4d.
  • the devices SRCl and SRC2 may comprise or be functionally connected to one or more computer processors
  • the image stream captured by the device and the scene model may be stored on a memory device for use in another device, e.g. a viewer, or transmitted to a server or the viewer using a communication interface COMM2.
  • a storage, processing and data stream serving network in addition to the capture device SRCl .
  • the device SERVER comprises or is functionally connected to a computer processor PROC3 and memory MEM3, the memory comprising computer program PROGR3 code for controlling the server.
  • the device SERVER may be connected by a wired or wireless network connection, or both, to sources SRC1 and/or SRC2, as well as the viewer devices VIEWER1 and VIEWER2 over the communication interface COMM3.
  • the creation of a three-dimensional scene model may take place at the server SERVER or another device by using the images captured by the devices SRC1.
  • the scene model may be a model created from captured image data (a real-world model), or a synthetic model such as on device SRC2, or a combination of such.
  • the scene model may be encoded to reduce its size and transmitted to a decoder, for example viewer devices.
  • the viewer devices may have a rendering module and a display module, or these functionalities may be combined in a single device.
  • the devices may comprise or be functionally connected to a computer processor PROC4 and memory MEM4, the memory comprising computer program PROG4 code for controlling the viewing devices.
  • the viewer (playback) devices may consist of a data stream receiver for receiving a video data stream and for decoding the video data stream.
  • the video data stream may be received from the server SERVER or from some other entity, such as a proxy server, an edge server of a content delivery network, or a file available locally in the viewer device.
  • the data stream may be received over a network connection through communications interface COMM4, or from a memory device MEM6 like a memory card CARD2.
  • the viewer devices may have a graphics processing unit for processing of the data to a suitable format for viewing.
  • the viewer VIEWER1 may comprise a high-resolution stereo-image head-mounted display for viewing the rendered stereo video sequence.
  • the head-mounted display may have an orientation sensor DETl and stereo audio headphones.
  • the viewer VIEWER2 may comprise a display (either two-dimensional or a display enabled with 3D technology for displaying stereo video), and the rendering device may have an orientation detector DET2 connected to it.
  • the viewer VIEWER2 may comprise a 2D display, since the volumetric video rendering can be done in 2D by rendering the viewpoint from a single eye instead of a stereo eye pair.
  • Fig. 1 depicts one SRC1 device and one SRC2 device, but generally the system may comprise more than one SRC1 device and/or SRC2 device.
  • any of the devices may be a computer or a portable computing device, or be connected to such or configured to be connected to such.
  • the devices SRCl, SRC2, SERVER, RENDERER, VIEWER1, VIEWER2
  • the devices may comprise multiple parts or may be comprised of multiple connected devices.
  • SERVER may comprise several devices, some of which may be used for editing the content produced by SRCl and/or SRC2 devices, some others for compressing the edited content, and a third set of devices may be used for transmitting the compressed content.
  • Such devices may have computer program code for carrying out methods according to various examples described in this text.
  • Figs. 2a and 2b show a capture device and a viewing device, respectively.
  • Fig. 2a illustrates a camera CAM1.
  • the camera has a camera detector CAMDET1, comprising a plurality of sensor elements for sensing intensity of the light hitting the sensor element.
  • the camera has a lens OBJ1 (or a lens arrangement of a plurality of lenses), the lens being positioned so that the light hitting the sensor elements travels through the lens to the sensor elements.
  • the camera detector CAMDET1 has a nominal centre point CP1 that is a middle point of the plurality of sensor elements, for example for a rectangular sensor the crossing point of diagonals of the rectangular sensor.
  • the lens has a nominal centre point PP1, as well, lying for example on the axis of symmetry of the lens.
  • the direction of orientation of the camera is defined by the line passing through the centre point CP1 of the camera sensor and the centre point PP1 of the lens.
  • the direction of the camera is a vector along this line pointing in the direction from the camera sensor to the lens.
  • the optical axis of the camera is understood to be this line CP1-PP1.
  • the optical path from the lens to the camera detector need not always be a straight line but there may be mirrors and/or some other elements which may affect the optical path between the lens and the camera detector.
  • Fig. 2b shows a head-mounted display (HMD) for stereo viewing.
  • the head- mounted display comprises two screen sections or two screens DISP1 and DISP2 for displaying the left and right eye images.
  • the displays are close to the eyes, and therefore lenses are used to make the images easily viewable and for spreading the images to cover as much as possible of the eyes' field of view.
  • the device When the device will be used by a user, the user may put the device on her/his head so that it will be attached to the head of the user so that it stays in place even when the user turns his head.
  • the device may have an orientation detecting module ORDETl for determining the head movements and direction of the head.
  • the head-mounted display gives a three-dimensional (3D) perception of the
  • Time-synchronized video and orientation data is first recorded with the capture devices. This can consist of multiple concurrent video streams as described above.
  • One or more time-synchronized audio streams may also be recorded with the capture devices.
  • the different capture devices may form image and geometry information of the scene from different directions. For example, there may be three, four, five, six or more cameras capturing the scene from different sides, like front, back, left and right, and/or at directions between these, as well as from the top or bottom, or any combination of these.
  • the cameras may be at different distances, for example some of the cameras may capture the whole scene and some of the cameras may be capturing one or more objects in the scene.
  • the cameras may be directed towards an object, looking onto the object from different directions, where the object is e.g. in the middle of the cameras. In this manner, the texture and geometry of the scene and the objects within the scene may be captured adequately.
  • the cameras or the system may comprise means for determining geometry
  • a computer model of a scene may be created.
  • a synthetic computer model of a virtual scene may be used.
  • the models (at successive time instances) are then transmitted immediately or later to the storage and processing network for processing and conversion into a format suitable for subsequent delivery to playback devices.
  • the conversion may involve processing and coding to improve the quality and/or reduce the quantity of the scene model data while preserving the quality at a desired level.
  • Each playback device receives a stream of the data (either computed video data or scene model data) from the network, and renders it into a viewing reproduction of the original location which can be experienced by a user.
  • the reproduction may be two- dimensional or three-dimensional (stereo image pairs).
  • Figs. 3a and 3b show an encoder and decoder for encoding and decoding texture pictures, geometry pictures and/or auxiliary pictures.
  • a video codec consists of an encoder that transforms an input video into a compressed representation suited for
  • Figure 3 a illustrates an image to be encoded (P); a predicted representation of an image block (P' n ); a prediction error signal (D n ); a reconstructed prediction error signal (D' n ); a preliminary reconstructed image (I' n ); a final reconstructed image (R' n ); a transform (T) and inverse transform (T 1 ); a quantization (Q) and inverse quantization (Q 1 ); entropy encoding (E); a reference frame memory (RFM); inter prediction (Pinter); intra prediction (Pmtra); mode selection (MS) and filtering (F).
  • Figure 3b illustrates a predicted representation of an image block (P' n ); a reconstructed prediction error signal (D' n ); a preliminary reconstructed image (I' n ); a final reconstructed image (R' n ); an inverse transform (T 1 ); an inverse quantization (Q 1 ); an entropy decoding (E 1 ); a reference frame memory (RFM); a prediction (either inter or intra) (P); and filtering (F).
  • Figs. 4a, 4b, 4c and 4d show a setup for forming a stereo image of a scene to a user, for example a video frame of a 3D video.
  • Fig. 4a a situation is shown where a human being is viewing two spheres Al and A2 using both eyes El and E2.
  • the sphere Al is closer to the viewer than the sphere A2, the respective distances to the first eye El being LEI,AI an d LEI,A2-
  • the different objects reside in space at their respective (x,y,z) coordinates, defined by the coordinate system SZ, SY and SZ.
  • the distance di 2 between the eyes of a human being may be approximately 62-64 mm on average, and varying from person to person between 55 and 74 mm. This distance is referred to as the parallax, on which stereoscopic view of the human vision is based on.
  • the viewing directions (optical axes) DIR1 and DIR2 are typically essentially parallel, possibly having a small deviation from being parallel, and define the field of view for the eyes.
  • the head of the user has an orientation (head orientation) in relation to the surroundings, most easily defined by the common direction of the eyes when the eyes are looking straight ahead. That is, the head orientation tells the yaw, pitch and roll of the head in respect of a coordinate system of the scene where the user is.
  • the spheres Al and A2 are in the field of view of both eyes.
  • the centre-point O12 between the eyes and the spheres are on the same line. That is, from the centre-point, the sphere A2 is behind the sphere Al .
  • each eye sees part of sphere A2 from behind Al, because the spheres are not on the same line of view from either of the eyes.
  • Fig. 4b there is a setup shown, where the eyes have been replaced by cameras CI and C2, positioned at the location where the eyes were in Fig. 4a.
  • the distances and directions of the setup are otherwise the same.
  • the purpose of the setup of Fig. 4b is to be able to take a stereo image of the spheres Al and A2.
  • the two images resulting from image capture are Fci and Fc2.
  • the "left eye” image Fci shows the image SA2 of the sphere A2 partly visible on the left side of the image SAI of the sphere Al .
  • the "right eye” image Fc2 shows the image SA2 of the sphere A2 partly visible on the right side of the image SAI of the sphere Al .
  • This difference between the right and left images is called disparity, and this disparity, being the basic mechanism with which the HVS determines depth information and creates a 3D view of the scene, can be used to create an illusion of a 3D image.
  • the camera pair CI and C2 has a natural parallax, that is, it has the property of creating natural disparity in the two images of the cameras. Natural disparity may be understood to be created even though the distance between the two cameras forming the stereo camera pair is somewhat smaller or larger than the normal distance (parallax) between the human eyes, e.g. essentially between 40 mm and 100 mm or even 30 mm and 120 mm.
  • the images Fci and Fc2 may be captured by cameras CI and C2, where the cameras CI and C2 may be real- world cameras or they may be virtual cameras. In the case of virtual cameras, the images Fci and Fc2 may be computed from a computer model of a scene by setting the direction, orientation and viewport of the cameras CI and C2 appropriately such that a stereo image pair suitable for viewing by the human visual system (HVS) is created.
  • HVS human visual system
  • Fig. 4c the creating of this 3D illusion is shown.
  • the images Fci and Fc2 captured or computed by the cameras CI and C2 are displayed to the eyes El and E2, using displays Dl and D2, respectively.
  • the disparity between the images is processed by the human visual system so that an understanding of depth is created. That is, when the left eye sees the image SA2 of the sphere A2 on the left side of the image SAI of sphere Al, and respectively the right eye sees the image SA2 of the sphere A2 on the right side, the human visual system creates an understanding that there is a sphere V2 behind the sphere VI in a three-dimensional world.
  • the images Fci and Fc2 can also be synthetic, that is, created by a computer. If they carry the disparity information, synthetic images will also be seen as three-dimensional by the human visual system. That is, a pair of computer-generated images can be formed so that they can be used as a stereo image.
  • Fig. 4d illustrates how the principle of displaying stereo images to the eyes can be used to create 3D movies or virtual reality scenes having an illusion of being three- dimensional.
  • the images Fxi and Fx2 are either captured with a stereo camera or computed from a model so that the images have the appropriate disparity.
  • a large number e.g. 30
  • the human visual system will create a cognition of a moving, three-dimensional image.
  • the field of view represented by the content may be greater than the displayed field of view e.g. in an arrangement depicted in Fig. 4d. Consequently, only a part of the content along the direction of view (a.k.a. viewing orientation) is displayed at a single time.
  • This direction of view that is, the head orientation
  • This direction of view may be determined as a real orientation of the head e.g. by an orientation detector mounted on the head, or as a virtual orientation determined by a control device such as a joystick or mouse that can be used to manipulate the direction of view without the user actually moving his head.
  • head orientation may be used to refer to the actual, physical orientation of the user's head and changes in the same, or it may be used to refer to the virtual direction of the user's view that is determined by a computer program or a computer input device.
  • the content may enable viewing from several viewing positions within the 3D space.
  • the texture picture(s), the geometry picture(s) and the geometry information may be used to synthesize the images Fxi and/or Fx 2 as if the displayed content was captured by camera(s) located at the viewing position.
  • volumetric video describes a 3D scene or object at different (successive) time instances, such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for any augmented reality, virtual reality and mixed reality applications, especially for providing viewing capabilities having six degrees of freedom (so-called 6DOF viewing).
  • FIG. 5a illustrates projection of source volumes in a digital scene model SCE and parts of an object model OBJ1, OBJ2, OBJ3, BG4 to projection surfaces SI, S2, S3, S4, as well as determining depth information for the purpose of encoding volumetric video.
  • the projection of source volumes SV1, SV2, SV3, SV4 may result in texture pictures and geometry pictures, and there may be geometry information related to the projection source volumes and/or projection surfaces.
  • Texture pictures, geometry pictures and projection geometry information may be encoded into a bitstream.
  • a texture picture may comprise information on the colour data of the source of the projection. Through the projection, such colour data may result in pixel colour information in the texture picture. Pixels may be coded in groups, e.g. coding units of rectangular shape.
  • the projection geometry information may comprise but is not limited to one or more of the following: - projection type, such as planar projection or equirectangular projection
  • - projection surface type such as a cube, sphere, cylinder, polyhedron
  • a projection centre such as a projection centre point, axis, or plane, from which a geometry primitive is projected onto the projection surface
  • the projection may take place by projecting the geometry primitives (points of a point could, triangles of a triangle mesh or voxels of a voxel array) of a source volume SV1 , SV2, SV3, SV4 (or an object OBJ1, OBJ2, OBJ3, BG4) onto a projection surface SI, S2, S3, S4.
  • the geometry primitives may comprise information on the texture, for example a colour value or values of a point, a triangle or a voxel.
  • the projection surface may surround the source volume at least partially such that projection of the geometry primitives happens from the centre of the projection surface outwards to the surface.
  • a cylindrical surface has a projection centre axis and a spherical surface has a projection centre point.
  • a cubical or rectangular surface may have projection centre planes or a projection centre axis or point and the projection of the geometry primitives may take place either orthogonally to the sides of the surface or from the projection centre axis or point outwards to the surface.
  • the projection surfaces e.g. cylindrical and rectangular, may be open from the top and the bottom such that when the surface is cut and rolled out on a two-dimensional plane, it forms a rectangular shape.
  • Such rectangular shape with pixel data can be encoded and decoded with a video codec.
  • the projection surface such as a planar surface or a sphere may be inside group of geometry primitives, e.g. inside a point cloud that defines a surface.
  • the projection may take place from outside in towards the centre and may result in sub-sampling of the texture data of the source.
  • points may be represented with any floating point coordinates.
  • a quantized point cloud may be used to reduce the amount of data, whereby the coordinate values of the point cloud are represented e.g. with 10-bit, 12-bit or 16-bit integers. Integers may be used because hardware accelerators may be able to operate on integers more efficiently.
  • the points in the point cloud may have associated colour, reflectance, opacity etc. texture values.
  • the points in the point cloud may also have a size, or a size may be the same for all points. The size of the points may be understood as indicating how large an object the point appears to be in the model in the projection.
  • the point cloud is projected by ray casting from the projection surface to find out the pixel values of the projection surface. In such a manner, the topmost point remains visible in the projection, while points closer to the centre of the projection surface may be occluded.
  • the original point cloud, meshes, voxels, or any other model is projected outwards to a simple geometrical shape, this simple geometrical shape being the projection surface.
  • Different projection surfaces may have different characteristics in terms of projection and reconstruction.
  • a projection to a cubical surface may be the most efficient, and a cylindrical projection surface may provide accurate results efficiently.
  • cones, polyhedron-based parallelepipeds (hexagonal or octagonal, for example) and spheres or a simple plane may be used as projection surfaces.
  • the phrase along the bitstream may be defined to refer to out-of-band transmission, signalling, or storage in a manner that the out-of-band data is associated with the bitstream.
  • the phrase decoding along the bitstream or alike may refer to decoding the referred out-of-band data (which may be obtained from out-of-band transmission, signalling, or storage) that is associated with the bitstream.
  • an indication along the bitstream may refer to metadata in a container file that encapsulates the bitstream.
  • a first texture picture may be encoded into a bitstream, and the first texture picture may comprise a first projection of texture data of a first source volume SV1 of a scene model SCE onto a first projection surface SI .
  • the scene model SCE may comprise a number of further source volumes SV2, SV3, SV4.
  • data on the position of the originating geometry primitive may also be determined, and based on this determination, a geometry picture may be formed. This may happen for example so that depth data is determined for each or some of the texture pixels of the texture picture. Depth data is formed such that the distance from the originating geometry primitive such as a point to the projection surface is determined for the pixels. Such depth data may be represented as a depth picture, and similarly to the texture picture, such geometry picture (in this example, depth picture) may be encoded and decoded with a video codec.
  • This first geometry picture may be seen to represent a mapping of the first projection surface to the first source volume, and the decoder may use this information to determine the location of geometry primitives in the model to be reconstructed.
  • a picture may be defined to be either a frame or a field.
  • a frame may be defined to comprise a matrix of luma samples and possibly the corresponding chroma samples.
  • a field may be defined to be a set of alternate sample rows of a frame. Fields may be used as encoder input for example when the source signal is interlaced. Chroma sample arrays may be absent (and hence monochrome sampling may be in use) or may be subsampled when compared to luma sample arrays.
  • each of the two chroma arrays has half the height and half the width of the luma array.
  • each of the two chroma arrays has the same height and half the width of the luma array.
  • arrays has the same height and width as the luma array.
  • Texture picture(s) and the respective geometry picture(s) may have the same or different chroma format.
  • a pixel may be defined to a be a sample of one of the sample arrays of the picture or may be defined to comprise the collocated samples of all the sample arrays of the picture.
  • multiple source volumes may be encoded as texture pictures, geometry pictures and projection geometry information into the bitstream in a similar manner. That is, as in Fig. 5a, the scene model SCE may comprise multiple objects OBJ1, OBJ2, OBJ3, OBJ4, and these may be treated as source volumes SV1, SV2, SV3, SV4 and each object may be coded as a texture picture, geometry picture and projection geometry information.
  • the first texture picture of the first source volume SV1 and further texture pictures of the other source volumes SV2, SV3, SV4 may represent the same time instance. That is, there may be a plurality of texture and geometry pictures and projection geometry information for one time instance, and the other time instances may be coded in a similar manner. Since the various source volumes are in this way producing sequences of texture pictures and sequences of geometry pictures, as well as sequences of projection geometry information, the inter-picture redundancy in the picture sequences can be used to encode the texture and geometry data for the source volumes efficiently, compared to the presently known ways of encoding volume data.
  • An object OBJ3 (source volume SV3) may be projected onto a projection surface S3 and encoded into the bitstream as a texture picture, geometry picture and projection geometry information as described above. Furthermore, such source volume may be indicated to be static by encoding information into said bitstream on said fourth projection geometry being static.
  • a static source volume or object may be understood to be an object whose position with respect to the scene model remains the same over two or more or all time instances of the video sequence.
  • the geometry data may also stay the same, that is, the object's shape remains the same over two or more time instances.
  • some or all of the texture data may stay the same over two or more time instances.
  • the decoder By encoding information into the bitstream of the static nature of the source volume the encoding efficiency may further be improved, as the same information may not need to be coded multiple times. In this manner, the decoder will also be able to use the same reconstruction or partially same reconstruction of the source volume (object) over multiple time instances.
  • the different source volumes may be coded into the bitstream with different frame rates.
  • a slow-moving or relatively unchanging object may be encoded with a first frame rate
  • a fast-moving and/or changing object may be coded with a second frame rate.
  • the first frame rate may be slower than the second frame rate, for example one half or one quarter of the second frame rate, or even slower.
  • the second frame rate may be 15 frames per second, or 1 frame per second.
  • the first and second object (source volumes) may be "sampled" in synchrony such that some frames of the faster frame rate coincide with frames of the slower frame rate.
  • the scene model may have a coordinate system and one or more of the objects (source volumes) in the scene model may have their local coordinate systems.
  • the shape, size, location and orientation of one or more projection surfaces may be encoded into or along the bitstream with respect to the scene model coordinates.
  • the encoding may be done with respect to coordinates of the scene model or said first source volume. The choice of coordinate systems may improve the coding efficiency.
  • Information on temporal changes in location, orientation and size of one or more said projection surfaces may be encoded into or along the bitstream. For example, if one or more of the objects (source volumes) being encoded is moving or rotating with respect to the scene model, the projection surface moves or rotates with the object to preserve the projection as similar as possible.
  • the projection surfaces may be sub-divided respectively. Therefore, information on sub- division of one or more of the source volumes and respective changes in one or more of the projection surfaces may be encoded into or along the bitstream.
  • the resulting bitstream may then be output to be stored or transmitted for later decoding and reconstruction of the scene model.
  • a first texture picture may be decoded from a bitstream to obtain first decoded texture data, where the first texture picture comprises a first projection of texture data of a first source volume of the scene model to be reconstructed onto a first projection surface.
  • the scene model may comprise a number of further source volumes.
  • a first geometry picture may be decoded from the bitstream to obtain first decoded scene model geometry data.
  • the first geometry picture may represent a mapping of the first projection surface to the first source volume.
  • First projection geometry information of the first projection may be decoded from the bitstream, the first projection geometry information comprising information of position of the first projection surface in the scene model.
  • a reconstructed scene model may be formed by projecting the first decoded texture data to a first destination volume using the first decoded scene model geometry data and said first projection geometry information to determine where the decoded texture information is to be placed in the scene model.
  • a 3D scene model may be classified into two parts: first all dynamic parts, and second all static parts.
  • the dynamic part of the 3D scene model may further be sub-divided into separate parts, each representing objects (or parts of) an object in the scene model, that is, source volumes.
  • the static parts of the scene model may include e.g. static room geometry (walls, ceiling, fixed furniture) and may be compressed either by known volumetric data compression solutions, or, similar to the dynamic part, sub-divided into individual objects for projection-based compression as described earlier, to be encoded into the bitstream.
  • some objects may be a chair (static), a television screen (static geometry, dynamic texture), a moving person (dynamic).
  • a suitable projection geometry may be found, e.g. cube projection to represent the chair, another cube for the screen, a cylinder for the person's torso, a sphere for a detailed representation of the person's head, and so on.
  • the 3D data of each object may then be projected onto the respective projection surface and 2D planes are derived by "unfolding" the projections from three dimensions to two dimensions (plane).
  • the unfolded planes will have several channels, typically three for the colour representation of the texture, e.g. RGB, YUV, and one additional plane for the geometry (depth) of each projected point for later
  • Frame packing may be defined to comprise arranging more than one input picture, which may be referred to as (input) constituent frames, into an output picture.
  • frame packing is not limited to any particular type of constituent frames or the constituent frames need not have a particular relation with each other.
  • frame packing is used for arranging constituent frames of a stereoscopic video clip into a single picture sequence.
  • the arranging may include placing the input pictures in spatially non-overlapping areas within the output picture. For example, in a side-by-side arrangement, two input pictures are placed within an output picture horizontally adjacently to each other.
  • the arranging may also include partitioning of one or more input pictures into two or more constituent frame partitions and placing the constituent frame partitions in spatially non-overlapping areas within the output picture.
  • the output picture or a sequence of frame-packed output pictures may be encoded into a bitstream e.g. by a video encoder.
  • the bitstream may be decoded e.g. by a video decoder.
  • the decoder or a post-processing operation after decoding may extract the decoded constituent frames from the decoded picture(s) e.g. for displaying.
  • a standard 2D video encoder may then receive the planes as inputs, either as individual layers per object, or as a frame-packed representation of all objects.
  • the texture picture may thus comprise a plurality of projections of texture data from further source volumes and the geometry picture may represent a plurality of mappings of projection surfaces to the source volume.
  • separation boundaries may be signalled to recreate the individual planes for each object
  • separation boundaries may be signalled to recreate the individual planes for each object
  • classification of each object as static/dynamic may be signalled
  • - relevant data to create real- world geometry data from the decoded (quantised) geometry channel(s), e.g. quantisation method, depth ranges, bit depth, etc. may be signalled, - initial state of each object: geometry shape, location, orientation, size may be signalled,
  • signalling may, for example, be as follows:
  • NUM OBJECTS 4 // folding-chair, TV, person body, person head
  • PROJ CENTRE ⁇ 7 ⁇ // projection centre in real world coordinates
  • DEPTH QUANT // depth quantisation, i.e. 0 for linear, ...
  • the decoder may receive the static 3D scene model data together with the video bitstreams representing the dynamic parts of the scene model. Based on the signalled information on the projection geometries, each object may be reconstructed in 3D space and the decoded scene model is created by fusing all reconstructed parts (objects or source volumes) together.
  • Standard video encoding hardware may be utilized for real-time
  • FIG. 5b shows an example of projecting an object OBJ1 using a cube map projection format, wherein there are six projection surfaces PS 1 , ... ,PS6 of the projection cube PC 1.
  • the projection surfaces are one on the left side PS1, one in front PS2, one on the right side PS3, one in the back PS4, one in the bottom PS5, and one in the top PS6 of the cube PCI in the setup of Figure 5b.
  • Figure 8a shows a cross section of a 3D object's surface enclosed in a bounding box as viewed from the top of the bounding box (cube). This part of the figure shows how the un-occluded surfaces of the 3D object is mapped to the four sides of the bounding box, while the occluded surfaces are not.
  • new projection surface poses are established to project the previously occluded surfaces of the 3D object. It should be noted that in Figure 8b the pose of the chosen projection surfaces is independent of other chosen projection surfaces, and are chosen such that they are sufficient enough to cover the surface that is to be projected.
  • the depth map for each projection surface PS1— PS6 may be formed on the basis of the distances PD1, PD2 of points of the object OBJ1 to the projection surface PS1— PS6. This distance is typically the distance of the surface point normal towards the projection surface, in other words a line from the 3D point to the projection surface so that the line is perpendicular to the projection surface.
  • Fig. 6 shows a projection of a source volume to a projection surface, and inpainting of a sparse projection.
  • a three-dimensional (3D) scene model, represented as objects OBJ1 comprising geometry primitives such as mesh elements, points, and/or voxel, may be projected onto one, or more, projection surfaces, as described earlier.
  • these projection surface geometries may be "unfolded" onto 2D planes (two planes per projected source volume: one for texture TP1, one for depth GP1), which may then be encoded using standard 2D video compression technologies. Relevant projection geometry information may be transmitted alongside the encoded video files to the decoder. The decoder may then decode the video and performs the inverse projection to regenerate the 3D scene model object ROBJl in any desired representation format, which may be different from the starting format e.g. reconstructing a point cloud from original mesh model data.
  • auxiliary pictures related to one or more said texture pictures and the pixels thereof may be encoded into or along with the bitstream.
  • the auxiliary pictures may e.g. represent texture surface properties related to one or more of the source volumes. Such texture surface properties may be e.g. surface normal information (e.g. with respect to the projection direction), reflectance and opacity (e.g. an alpha channel value).
  • An encoder may encode, in or along with the bitstream, indication(s) of the type(s) of texture surface properties represented by the auxiliary pictures, and a decoder may decode, from or along the bitstream, indication(s) of the type(s) of texture surface properties represented by the auxiliary pictures.
  • Mechanisms to represent an auxiliary picture may include but are not limited to the following:
  • a colour component sample array such as a chroma sample array, of the geometry picture.
  • a constituent frame of a frame-packed picture that may also comprise texture picture(s) and/or geometry picture(s).
  • auxiliary picture included in specific data units in the bitstream.
  • H.264/AVC Advanced Video Coding
  • NAL NAL unit for a coded slice of an auxiliary coded picture without partitioning.
  • An auxiliary picture layer within a layered bitstream comprises the feature of including auxiliary picture layers in the bitstream.
  • An auxiliary picture layer comprises auxiliary pictures.
  • An auxiliary picture bitstream separate from the bitstream(s) for the texture picture(s) and geometry picture(s).
  • the auxiliary picture bitstream may be indicated, for example in a container file, to be associated with the bitstream(s) for the texture pictures(s) and geometry picture(s).
  • the mechanism(s) to be used for auxiliary pictures may be pre-defined e.g. in a coding standard, or the mechanism(s) may be selected e.g. by an encoder and indicated in or along the bitstream.
  • the decoder may decode the mechanism(s) used for auxiliary pictures from or along the bitstream.
  • the projection surface of a source volume may encompass the source volume, and there may be a model of an object in that source volume. Encompassing may be understood so that the object (model) is inside the surface such that when looking from the centre axis or centre point of the surface, the object's points are closer to the centre than the points of the projection surface are.
  • the model may be made of geometry primitives, as described.
  • the geometry primitives of the model may be projected onto the projection surface to obtain projected pixels of the texture picture. This projection may happen from inside-out.
  • the projection may happen from outside-in.
  • Projecting 3D data onto 2D planes is independent from the 3D scene model representation format.
  • FIG. 7 illustrates an example of this kind of situation.
  • the person's left hand occludes a part of the body of the person so that when viewed (projected) from the left hand's side the occluded part of the body would not be projected.
  • the planar object on the person's right hand occludes some parts of the person's stomach when viewed from the front of the person.
  • Figure 8a shows an example of projection occlusions in a simplified manner.
  • the arrows indicate projections of points of the object to projection surfaces PS1— PS6 (of which only four are shown for clarity, as was explained above).
  • the arrows marked with X indicate projections which are not performed due to occlusions i.e. those points belong to a part of the 3D object which is not fully visible from that particular viewing direction.
  • occlusion examination is performed to detect points, surfaces, other parts of the object which will not be projected onto the projection surface in question.
  • some occluded parts of the example of Figure 8a are depicted in Figure 8b as lines OP1, OP2.
  • the occlusion detection may be performed during the projection operation wherein when an occlusion is detected, that part will not be projected to the projection surface in question.
  • Information of the non-projected (occluded) parts may be temporally saved, if necessary, and used to form auxiliary projection surfaces. The above procedure may be repeated for each projection surface of the projection model so that all possible occlusions could be detected for each projection surface.
  • the initial projection surfaces of the embodiment may also be called as main projection surfaces in this specification.
  • the main projection surfaces are the left PS1, front of PS2, right PS3, back PS4, bottom PS5 and top PS6 of the cube map format.
  • the occlusion detection may be time dependent wherein one occlusion detection and succeeding operations induced by the occlusion detection (some of which will be described below) may be valid only for one time instance of a time-varying 3D visual scene/object and similar operations may need to be performed for other time instances of a time-varying 3D visual scene/object as well.
  • the occlusion detection is performed for all time instances of a time- varying 3D visual scene/object.
  • Information of the detected occlusions may be utilized to form auxiliary projection surfaces.
  • a sequence of unique pose (location, orientation and size) of auxiliary projection surfaces with respect to a 3D object/scene are identified. For example, occluded points are examined to determine which of them belong to the same surface of the object.
  • information regarding that auxiliary projection surface with respect to a main projection surface will be obtained.
  • the pose of the auxiliary projection surfaces can be arbitrary in terms of location, orientation and size. This information may comprise the direction of the auxiliary projection surface with respect to a main projection surface, the size of the auxiliary projection surface (e.g. width and height), and location (distance) of the auxiliary projection surface with respect to the main projection surface.
  • the lines OP1, OP2 are examples of a set of occluded points of the object OBJl .
  • all occluded points which belong to the same surface of the object need not be projected to the same auxiliary projection surface but they may be projected to different auxiliary projection surfaces.
  • one point of the 3D object/scene should only be projected to one auxiliary projection surface, in accordance with an embodiment.
  • each of the determined auxiliary projection surfaces maps only those 3D surface points that are un-occluded in its viewing direction.
  • a texture plane is computed for each of the chosen poses of the auxiliary projection surfaces by projecting the texture of the un-occluded 3D surface that intersects its viewing direction.
  • the texture of the 3D surface which is visible (un-occluded) from the selected auxiliary projection and pose is projected to the selected auxiliary projection to obtain the texture plane.
  • a representation of the distance of 3D surface points from the auxiliary projection surface is computed for each of the chosen poses of the auxiliary projection surfaces, to generate a related depth plane.
  • auxiliary projection surfaces For each of the chosen poses of the auxiliary projection surfaces, other 3D surface point related attributes, such as normals and bi-directional reflectance distribution functions (BRDF), may be projected to form their own individual, but related auxiliary projection surfaces.
  • BRDF bi-directional reflectance distribution functions
  • the computed texture, depth and auxiliary projection surfaces are collected and may either be frame-packed or layered before encoding using 2D video compression algorithms.
  • auxiliary projection surfaces with respect to another of the projection surfaces are computed and signalled either within the encoded stream or by other external means. Additional signalling may also be formed that relates the depth and other auxiliary projection surfaces to their respective texture planes are included either within the encoded stream or by other external means.
  • Information of the auxiliary projection surface may comprise none, one or more of the following and possibly some other information not listed here:
  • - projection surface type such as a cube, cylinder, sphere
  • a projection centre such as a projection centre point, axis, or plane
  • the relative pose of the projection surfaces is computed and signalled either within the encoded bit stream or by external means. For example, if the faces of a bounding box are used as the initial (main) projection surfaces of a 3D point cloud, then the pose of an auxiliary projection surface is signalled as a surface generated as a rotation of the normal of the main projection surface, a displacement vector of the central point of the auxiliary projection surface from the main projection surface, and if needed the horizontal and vertical dimensions of the surface. This process is illustrated in Figure 8c.
  • the width and height of the projection surface can either be signalled explicitly or inferred from the spatial dimension of the coded texture, depth or other auxiliary projection surfaces. This process is iterated for all the projected surfaces of a 3D point cloud for one time instant.
  • Figure 8c illustrates the calculation of the relative pose of the projection surface PI with respect to the projection surface P0.
  • the pose of the projection surface PI can be signalled as a combination of parameters (d, ⁇ ) where d is the distance vector from the centre of the projection surface P0 to the centre of the projection surface PI, and ⁇ is the rotation of the planar normal of the projection surface P0 toward the direction of the planar normal of the projection surface PI .
  • Encoding the projection information may be performed, for example, as follows.
  • the projected texture planes of a point cloud for one time instant can all be collected and frame-packed and similarly the depth and the possible auxiliary planes are frame-packed such that the frame packing is consistent across all the planes for one time instant.
  • Each of the planes are then coded using traditional 2D video coders using layered video coding or coded independently and related to each other using some form of higher level signalling (for e.g. using tracks and track references of ISO Base Media File Format).
  • each texture, depth and auxiliary planes of a single time instant of a point cloud could be considered independent of each other in that time instant and coded serially one after the other.
  • Figures 9a, 9b and 9c illustrate the encoding, in accordance with an embodiment.
  • Figure 9a shows the texture planes projected on to seven projection surfaces Tl— T7.
  • the hashed boxes Tl— T4 are the four sides of a bounding box of a 3D object (the top and bottom planes are omitted in this figure for clarity).
  • the three other hashed boxes T5, T6, T7 are the auxiliary projection surfaces computed to project occluded surfaces after the initial four projections.
  • the arrows illustrate the signalling of the poses of the auxiliary projection surfaces with respect to the main projection surfaces Tl— T4.
  • Figure 9b shows the same process for the computing of the depth planes Dl— D7
  • Figure 9c illustrates the coded bit stream after layering the projection surfaces of respective attributes, in accordance with an embodiment.
  • the hashings of the boxes of the coded bit stream and the labels Tl— T7, Dl— D7 above the boxes illustrate the source of information which is coded in the boxes.
  • an entity that maps 3D texture data on to projection surfaces can choose the six sides of an oriented or an axis aligned bounding box of a 3D point cloud as the initial set of projection surfaces.
  • the mapping of 3D surface parts on to the projection surfaces only maps the closest coherent surface onto the projections surfaces. For example, if there are two surfaces of the 3D object where one surface occludes the other surface in the direction of the 2D surfaces normal, then only the occluding surface is mapped on to the projection surface.
  • the occluded surface may require the generation of another projection surface for mapping.
  • the pose of the projections surfaces for the occluded points in the point cloud can be chosen such that it maximizes the rate-distortion performance for encoding the texture, depth and other auxiliary surfaces.
  • the size of the projection surface that maps a partial 3D surface can be chosen such that the area of the projection surface that does not map any surface points of the 3D texture is minimized. This is done by, for example, ensuring that there is at least one row and one column of the projection surface that is fully mapped to a 3D surface.
  • the three projections surface that maps previously occluded surface as illustrated in Figure 8b are just sufficient enough to map the surface in its view direction.
  • a plane with the projected texture, and a plane projected depth is generated.
  • other attributes of the mapped points such as surface normal and bi-directional reflectance distribution functions (BRDF), can also be coded as additional auxiliary surfaces. Therefore, for every chosen size, orientation and location of a projection surface there could be one or more surfaces generated that maps the different attributes of the projected points of a point cloud.
  • BRDF surface normal and bi-directional reflectance distribution functions
  • a number of n coherent surfaces is defined.
  • the surfaces may be derived, for example, by a set of geometry criteria, such as:
  • Top-down occlusion separation If parts of a coherent surface that would be occluded by another surface, these parts are separated into a new surface.
  • each 3D point is assigned with a specific surface ID so that there is only a single ID per point.
  • resulting set of surfaces is processed according to encoding parameters. For example, if the number of projections is limited, surfaces are removed in a bottom-up approach, i.e. only the most outward lying surfaces are kept. All points of the removed surfaces are assigned with a surface ID indicating to ignore this point. If the number of projections is larger than required to cover all surfaces (or a predefined threshold number of surfaces), the number of projections may be reduced in the encoding parameters to save bit rate.
  • Remaining surface IDs may be converted to projection IDs so that each point of a volumetric video object has one single ID indicating on which face of the projection geometry it will be mapped.
  • Several surfaces can be mapped onto same projection face if they don't occlude eachother, i.e. there are as many projection Ids as there are projection faces but there might be a larger number of surface IDs.
  • a bit stream comprising a sequence of one or more coded sets of pictures is received.
  • the coded set of pictures in each unit of the sequence after decoding and reconstruction is a representation of a 3D scene/object at a particular instant in time.
  • Each unit of the sequence may contain a combination of zero, one, or more texture, depth and other auxiliary surfaces coded as pictures.
  • Each texture picture, if available in a time-sequenced unit may be a frame-packed representation of all the projected textures of a 3D object/scene or a single layer related to a particular pose of a projection surface, wherein a corresponding de-packing operations may be performed to the received sequences of one or more coded sets of pictures.
  • Zero or more depth projected surfaces that are related to the pose of its projection surface are reconstructed using signalling that is either encoded within the bit stream or that can be obtained from signalling using external means.
  • Zero or more auxiliary surfaces that carry other surface data, such as surface normals and BRDFs, and can be related to its projection surfaces pose are reconstructed using signalling either within the coded bit stream or signalled externally.
  • the 3D surface is reconstructed using the required projection surfaces for any time unit using the texture, depth and auxiliary data encoded as pictures.
  • a receiver of the coded point cloud data receives the coded projection surfaces of the point cloud and additional signalling about the position, orientation and size of the projection surfaces. Using the coded data and the signalled information, a receiving entity can reconstruct a reasonably accurate representation of the original point cloud.
  • Some embodiments may be some advantages. For example, better handling of occluded surfaces in a 3D object/scenes may be obtained. Also, coding of projected surfaces may be efficient. Furthermore, some embodiments could enable new six degrees of freedom (6DOF) experiences such as zooming into internal objects.
  • 6DOF six degrees of freedom
  • Figs. 9a and 9b show flow charts for compressing encoding and decoding of a scene of a volumetric video.
  • phase 802 projection data is generated from the volumetric presentations by projecting the object onto one or more projection surfaces using a projection geometry.
  • phase 803 occluded parts which did not project on the projection surface are examined.
  • phase 804 an auxiliary projection surface is defined for the occluded parts
  • phase 805 wherein in phase 805 the occluded parts are projected to the auxiliary projection surface.
  • phase 806 the main and auxiliary projection surfaces are encoded into a bit stream, and in phase 807 information of the relationship between the main projection surface and the auxiliary projection surface into a bit stream, which may be the same bit stream than in phase 805 or a different bit stream.
  • phase 821 main and auxiliary projection surfaces are received and decoded from a bit stream to obtain decoded projection data.
  • phase 822 information of the relationship between the main projection surfaces and the auxiliary projection surfaces is received and decoded from the same or a different bit stream to obtain decoded relationship data.
  • phase 823 the 3D object surfaces are reconstructed by using the decoded projection data and the relationship data.
  • FIG 11a illustrates an example of an encoding device 1000 which may perform the above described compression operations.
  • Objects of a source volume of a digital scene model may be input 1001 to a projection element 1002 in which different objects may be projected to projection surfaces.
  • An occlusion detection element 1003 examines which parts of the object were occluded and provides this information to an auxiliary projection element 1004.
  • the auxiliary projection element 1004 generates projections of the occluded elements.
  • the projection data may be provided for subsequent encoding e.g. in an encoding element 1005 and transmission. It should be noted that operations of the projection element 1002 and the auxiliary projection element 1002 may also be performed by the same element.
  • Figure 1 lb illustrates an example of a decoding device 1010 which may perform the above described decoding operations.
  • Main and auxiliary projection surfaces and information of the relationship between the main projection surfaces and the auxiliary projection surfaces are received 1011 and decoded from a bit stream by a decoding element 1012.
  • a reconstruction element 1013 reconstructs 3D object surfaces by using the decoded projection data and the relationship data. Reconstructed 3D objects may be output 1014 for subsequent displaying and/or storage.
  • encoding may comprise one or more of the following:
  • MPD Physical Description
  • SDP IETF Session Description Protocol
  • decoding may comprise one or more of the following: decoding image data from a bitstream, decapsulating the bitstream from a container file and/or from packet(s) or stream(s) of a communication protocol, and parsing a content description of the bitstream,
  • Embodiments of the inventions may be practiced in various components such as integrated circuit modules.
  • the design of integrated circuits is by and large a highly automated process.
  • Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
  • Programs such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre stored design modules.
  • the resultant design in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

La présente invention concerne divers procédés, appareils et produits-programmes d'ordinateur destinés au codage et au décodage de vidéo volumétrique. Selon certains modes de réalisation de codage, la présentation volumétrique d'un objet est obtenue. L'objet est projeté sur une surface de projection principale au moyen d'un format de projection. Une partie cachée de l'objet sur la surface de projection principale est déterminée, une surface de projection auxiliaire étant définie pour la partie cachée. La partie cachée est projetée sur la surface de projection auxiliaire ; et une indication de la relation entre la surface de projection auxiliaire et la surface de projection principale est fournie. Selon certains modes de réalisation de décodage, des informations sur la projection d'une première partie d'un objet d'une présentation volumétrique, se rapportant à une surface de projection principale, sont reçues. Des informations de projection d'une seconde partie de l'objet, se rapportant à une surface de projection auxiliaire, sont également reçues. Ladite seconde partie est cachée depuis une direction de visualisation de la surface de projection principale. Des informations sur la relation entre la surface de projection auxiliaire et la surface de projection principale sont en outre reçues. Les informations sur la relation sont utilisées pour obtenir la surface de projection auxiliaire. La première partie de l'objet est rétro-projetée à partir de la surface de projection principale ; et la seconde partie de l'objet est rétro-projetée à partir de la surface de projection auxiliaire.
PCT/FI2018/050730 2017-10-18 2018-10-10 Appareil, procédé, et programme d'ordinateur pour vidéo volumétrique WO2019077199A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP18867354.5A EP3698332A4 (fr) 2017-10-18 2018-10-10 Appareil, procédé, et programme d'ordinateur pour vidéo volumétrique

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FI20175914 2017-10-18
FI20175914 2017-10-18

Publications (1)

Publication Number Publication Date
WO2019077199A1 true WO2019077199A1 (fr) 2019-04-25

Family

ID=66174327

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2018/050730 WO2019077199A1 (fr) 2017-10-18 2018-10-10 Appareil, procédé, et programme d'ordinateur pour vidéo volumétrique

Country Status (2)

Country Link
EP (1) EP3698332A4 (fr)
WO (1) WO2019077199A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020011717A1 (fr) * 2018-07-11 2020-01-16 Telefonaktiebolaget Lm Ericsson (Publ) Spécification de flux binaire de codec en nuage de points reposant sur une vidéo
WO2023180843A1 (fr) * 2022-03-25 2023-09-28 Sony Group Corporation Génération de correctif pour codage maillé dynamique

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2340527A1 (fr) * 2008-10-21 2011-07-06 Koninklijke Philips Electronics N.V. Procédé et dispositif pour générer un modèle stratifié en profondeur d une scène et signal comprenant un modèle stratifié en profondeur d une scène
US20110205226A1 (en) * 2008-10-28 2011-08-25 Koninklijke Philips Electronics N.V. Generation of occlusion data for image properties
US20110261050A1 (en) * 2008-10-02 2011-10-27 Smolic Aljosa Intermediate View Synthesis and Multi-View Data Signal Extraction
EP2451164A1 (fr) * 2010-11-08 2012-05-09 Telefonaktiebolaget L M Ericsson AB (Publ) Synthèse à vue améliorée
WO2013074153A1 (fr) * 2011-11-17 2013-05-23 University Of Southern California Génération de modèles tridimensionnels à partir de données de capteur d'images
EP2611182A2 (fr) * 2011-12-26 2013-07-03 Samsung Electronics Co., Ltd Procédé et appareil de traitement d'image utilisant une représentation multicouche
US20130176394A1 (en) * 2010-09-14 2013-07-11 Thomson Licensing Compression methods and apparatus for occlusion data
US20130286012A1 (en) * 2012-04-25 2013-10-31 University Of Southern California 3d body modeling from one or more depth cameras in the presence of articulated motion
US20140176553A1 (en) 2011-08-10 2014-06-26 Telefonaktiebolaget L M Ericsson (Publ) Method and Apparatus for Creating a Disocclusion Map used for Coding a Three-Dimensional Video

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110261050A1 (en) * 2008-10-02 2011-10-27 Smolic Aljosa Intermediate View Synthesis and Multi-View Data Signal Extraction
EP2340527A1 (fr) * 2008-10-21 2011-07-06 Koninklijke Philips Electronics N.V. Procédé et dispositif pour générer un modèle stratifié en profondeur d une scène et signal comprenant un modèle stratifié en profondeur d une scène
US20110205226A1 (en) * 2008-10-28 2011-08-25 Koninklijke Philips Electronics N.V. Generation of occlusion data for image properties
US20130176394A1 (en) * 2010-09-14 2013-07-11 Thomson Licensing Compression methods and apparatus for occlusion data
EP2451164A1 (fr) * 2010-11-08 2012-05-09 Telefonaktiebolaget L M Ericsson AB (Publ) Synthèse à vue améliorée
US20140176553A1 (en) 2011-08-10 2014-06-26 Telefonaktiebolaget L M Ericsson (Publ) Method and Apparatus for Creating a Disocclusion Map used for Coding a Three-Dimensional Video
WO2013074153A1 (fr) * 2011-11-17 2013-05-23 University Of Southern California Génération de modèles tridimensionnels à partir de données de capteur d'images
EP2611182A2 (fr) * 2011-12-26 2013-07-03 Samsung Electronics Co., Ltd Procédé et appareil de traitement d'image utilisant une représentation multicouche
US20130286012A1 (en) * 2012-04-25 2013-10-31 University Of Southern California 3d body modeling from one or more depth cameras in the presence of articulated motion

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
CHU, H.-K. ET AL.: "Motion Manipulation for Characters in Videos", 2015 CONFERENCE ON TECHNOLOGIES AND APPLICATIONS OF ARTIFICIAL INTELLIGENCE (TAAI, vol. 512-516, 20 November 2015 (2015-11-20), pages 514 - 515, XP032864365, ISSN: 2376-6824, ISBN: 978-1-4673-9606-6, Retrieved from the Internet <URL:https://ieeexplore.ieee.org/abstract/document/7407085> [retrieved on 20190121] *
JANTET, V., LAYERED DEPTH IMAGES FOR MULTI-VIEW CODING, 14 February 2013 (2013-02-14), pages 40, 57 - 87, 97-115, XP055501173, [retrieved on 20190208] *
SCHWARZ, S. ET AL.: "Nokia's response to CfP for Point Cloud Compression (Category 2)", 120. MPEG MEETING; 23-10-2017 - 27-10-2017; MACAU; (MOTION PICTURE EXPERT GROUP OR ISO/IEC JTC1/SC29/WG11),20171017, no. m41779, 17 October 2017 (2017-10-17), pages 5 - 8, 10-12, XP030070121, Retrieved from the Internet <URL:http://phenix.int-evry.fr/mpeg> [retrieved on 20190131] *
See also references of EP3698332A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020011717A1 (fr) * 2018-07-11 2020-01-16 Telefonaktiebolaget Lm Ericsson (Publ) Spécification de flux binaire de codec en nuage de points reposant sur une vidéo
WO2023180843A1 (fr) * 2022-03-25 2023-09-28 Sony Group Corporation Génération de correctif pour codage maillé dynamique

Also Published As

Publication number Publication date
EP3698332A4 (fr) 2021-06-30
EP3698332A1 (fr) 2020-08-26

Similar Documents

Publication Publication Date Title
US11109066B2 (en) Encoding and decoding of volumetric video
US11599968B2 (en) Apparatus, a method and a computer program for volumetric video
EP3669333B1 (fr) Codage et décodage séquentiels de vidéo volumétrique
US11202086B2 (en) Apparatus, a method and a computer program for volumetric video
US11430156B2 (en) Apparatus, a method and a computer program for volumetric video
WO2019135024A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
US10499033B2 (en) Apparatus, a method and a computer program for coding and rendering volumetric video
EP3614674A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
US11659151B2 (en) Apparatus, a method and a computer program for volumetric video
WO2019158821A1 (fr) Appareil, procédé et programme informatique de vidéo volumétrique
US11463681B2 (en) Encoding and decoding of volumetric video
US11528509B2 (en) Video transmission method, video transmission device, video receiving method and video receiving device
WO2019229293A1 (fr) Appareil, procédé et programme d&#39;ordinateur pour vidéo volumétrique
WO2019115867A1 (fr) Appareil, procédé, et programme d&#39;ordinateur pour vidéo volumétrique
WO2019115866A1 (fr) Appareil, procédé, et programme d&#39;ordinateur pour vidéo volumétrique
JP7344988B2 (ja) ボリュメトリック映像の符号化および復号化のための方法、装置、およびコンピュータプログラム製品
EP3729805A1 (fr) Procédé de codage et de décodage de données vidéo volumétriques
WO2019077199A1 (fr) Appareil, procédé, et programme d&#39;ordinateur pour vidéo volumétrique
WO2018109265A1 (fr) Procédé et équipement technique de codage de contenu de média
WO2020157376A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
WO2019162564A1 (fr) Appareil, procédé et programme d&#39;ordinateur pour vidéo volumétrique
WO2019234290A1 (fr) Appareil, procédé et programme d&#39;ordinateur pour vidéo volumétrique
WO2022219230A1 (fr) Procédé, appareil et produit-programme d&#39;ordinateur de codage vidéo et de décodage vidéo

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18867354

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018867354

Country of ref document: EP

Effective date: 20200518