TW201813372A - Method and system for signaling of 360-degree video information - Google Patents

Abstract

Coding techniques for 360-degree video are described. An encoder selects a projection format and maps the 360-degree video to a 2D planar video using the selected projection format. The encoder encodes the 2D planar video in a bitstream and further signals, in the bitstream, parameters identifying the projection format. The parameters identifying the projection format may be signaled in a video parameter set, sequence parameter set, and/or picture parameter set of the bitstream. Different projection formats that may be signaled include formats using geometries such as equirectangular, cubemap, equal-area, octahedron, icosahedron, cylinder, and user-specified polygon. Other parameters that may be signaled include different arrangements of geometric faces or different encoding quality for different faces. Corresponding decoders are also described. In some embodiments, projection parameters may further include relative geometry rotation parameters that define an orientation of the projection geometry.

Description

360 degree video information communication method and system

Cross-Reference to Related Applications This application is a US Provisional Patent Application entitled "Method and System for Signaling of 360-Degree Video Information", filed on September 2, 2016, in accordance with 35 USC § 119(e) No. 62/383,367, and the benefit of U.S. Provisional Patent Application Serial No. 62/407,337, entitled "Method and System for Signaling of 360-Degree Video Information", filed on October 12, 2016, the entire contents of which is incorporated by reference. Incorporated herein by reference.

Virtual Reality (VR) is coming out of research laboratories and entering our daily lives. VR has many application areas: healthcare, education, social networking, industry design/training, gaming, movies, shopping, entertainment, and more. It is highly regarded by industry and consumers because VR can bring an immersive viewing experience. It creates a virtual environment around the viewer and may create a real feeling of "being there." How to provide a complete real feeling in a VR environment is important to the user's experience. For example, a VR system should support interaction via gestures, gestures, eye gaze, sound, and the like. In order to allow the user to interact with objects in the VR world in a natural manner, the system can also provide tactile feedback to the user.

Today's VR systems use 360-degree video to provide the user with the ability to view the scene from a 360-degree angle in the horizontal direction and a 180-degree angle in the vertical direction. At the same time, VR and 360-degree video are seen as the future direction of media consumption beyond ultra-high definition (UHD) services. In order to improve the quality of 360-degree video in VR and standardize the processing chain of VR client interoperability, in early 2016, the ad hoc group belonging to MPEG-A (Multimedia Application Format) part-19 has been established in ISO/IEC/MPEG. Committed to the requirements and potential technologies of a full range of media application formats. Another ad hoc group, Free Watch TV (FTV), released a 360-degree 3D video application exploration experiment. One of the main goals of FTV is to test the performance of two solutions: (1) systems based on 360-degree video (omnidirectional video); and (2) multi-view based systems. The Joint Video Discovery Team (JVET) from MPEG and ITU-T is exploring new technologies for next-generation video coding standards and has released a collection of test sequences including VR. At the June 2016 meeting, the ad hoc group (AHG8) was established. The task of the AHG8 group was to develop common test conditions, test sequence formats and evaluation criteria for 360 video coding. AHG8 will also study the effects of compression on different projection methods and the effects of compression on software.

The industry is working to improve the quality and user experience in all aspects of the VR processing chain, including capture, processing, display and application. In terms of capture, the VR system uses multiple camera systems to capture scenes from different divergent views (eg, in some cases, approximately 6 to 12 views). These views are stitched together to form a high resolution (eg 4K or 8K) 360 degree video. On the client or user side, current virtual reality systems typically include a computing platform, a head mounted display (HMD), and a head tracking sensor. The computing platform is responsible for receiving and decoding 360-degree video and generating a view for display. Two photos (one for each eye) are presented for viewing. These two photos are displayed in the HMD for stereo viewing. A lens can be used to magnify the image displayed in the HMD for better viewing. The head tracking sensor can continually track the viewer's head orientation and feed the orientation information to the system to display a viewfinder image for that orientation.

Some VR systems can provide viewers with specialized touch devices to interact with objects in the virtual world. There are existing VR systems available on the market. One is the Rift from Oculus, and the Gear VR from Samsung and Oculus. The Rift is powered by a powerful workstation with good GPU support. Gear VR is a lightweight VR system that uses a smartphone, HMD display and head tracking sensor as a computing platform. The second type of VR system is the HTC Vive system. Rift and Vive have similar performance. The spatial HMD resolution is 2160×1200, the refresh rate is 90 Hz, and the field of view (FOV) is about 110 degrees. The head tracking sensor has a sampling rate of 1000 Hz and can take very fast motion. Google also has a simple VR system called a cardboard. The Google Tray has a lens and tray component, similar to the Gear VR, which is powered by a smartphone. Sony also offers PlayStation VR (Game VR) for games. In the 360-degree video streaming service, YouTube and Facebook are among the early providers.

In these current VR systems, the quality of experiences such as interactive and tactile feedback still needs further improvement. For example, today's HMDs are still too big to wear. Furthermore, the current resolution of 2160 x 1200 for stereoscopic views provided by HMDs is insufficient and may cause dizziness and discomfort to some users. Therefore, it would be beneficial to increase the resolution. In addition, combining visual sensations in a VR environment with power feedback in the real world is an option to enhance the VR experience. The VR Cloud Speed is an example application.

Many companies are developing 360-degree video compression and delivery systems that have their own solutions. For example, Google YouTube provides a conduit for DASH-based 360-degree video streaming. Facebook also has a 360-degree video delivery solution.

The systems and methods herein are directed to solving problems associated with 360-degree video data encoding and decoding.

In an exemplary method of encoding 360 degree video, the encoder selects a projection format, wherein the projection format includes information such as geometric type and/or geometric orientation. The encoder maps 360-degree video to 2D flat video using the selected projection format. The encoder encodes the 2D planar video in the bitstream and further communicates in the bitstream to identify parameters of the projected format. Various geometry types can be used and can be signaled in the bitstream, including equirectangular, cubemap, equal area, octahedron, icosahedron, cylinder, and user-specified Polygon. For geometry types associated with multiple faces, the frame encapsulation parameters can be communicated to identify the position and/or orientation of those faces in the 2D planar video. Different faces can be coded with different sizes and/or different quality levels. The parameter identifying the geometric orientation may include at least one of a yaw parameter, a pitch parameter, and a rolling parameter.

The parameters identifying the projection format can be communicated in the video parameter set, the sequence parameter set, and/or the image parameter set of the bit stream. The projection parameters can be selected based on rate distortion optimization. Different images or different sequences in the video can be encoded using different projection formats (eg, when different images or sequences have higher rate distortion performance for different projection formats), providing a telegraphic projection format at the appropriate parameter set parameter. Corresponding decoding techniques are also described.

Described in the present disclosure is an exemplary syntax for 360 degree video coding. Syntax elements can be used to specify projection geometry and/or use a grid system to specify the placement of faces in the frame encapsulation image. The faces can have different sizes and/or orientations. In some embodiments, the face arrangement on the 2-D plane can have various features, such as a fixed face width/height along each row/column. In some embodiments, any polygon-based representation is used to describe an exemplary syntax of a user-specified geometry. Additional features used in some embodiments may include using a flag to skip samples for filling the frame-packaged image, uploading incremental quantization parameters (QP) on a face-by-face level, enabling/disabling specific faces The flag of the loop filter between, and/or the syntax of only the specific area of the 360 video.

In some embodiments, the projection parameters can also include relative geometric rotation parameters. Such parameters can define the orientation of the projection geometry. The projection geometry can be selectively oriented such that the object of interest is substantially completely included within a single plane of the projection geometry. In embodiments in which different faces are encoded with different quality levels (eg, different QP values), the projection geometry can be oriented such that the object of interest is substantially completely contained within the face encoded with a relatively high quality level.

A detailed description of the illustrative embodiments will now be provided with reference to the drawings. While the description provides a detailed example of possible implementations, it should be noted that the details are not intended to limit the scope of the application. 360- degree video encoding and decoding

One technique for 360-degree video delivery is to use sphere geometry to represent 360-degree information. For example, a plurality of synchronized views captured by a plurality of cameras are spliced onto a sphere as a unitary structure. The sphere information is then projected onto the 2D planar surface using a given geometric transformation process (eg, an equidistant columnar projection (ERP) method). Figure 1A shows sphere sampling on longitude (φ) and latitude (θ), and Figure 1B shows projection of spheres into 2D plane using equidistant cylindrical projection. In aeronautics, the longitude φ in the range [-π, π] is called yaw, and the latitude θ in the range of [-π/2, π/2] is called pitch, where π is the circumference of the circle. The ratio to its diameter. For convenience of explanation, (x, y, z) is used to represent coordinates of points in 3D space, and (ue, ve) is used to represent coordinates of points in a 2D plane having equidistant columnar projections. The equidistant columnar projection can be mathematically represented in equations (1) and (2): ue = (φ/(2* π)+0.5)*W (1) ve = (0.5 - θ/π)* H (2)

Where W and H are the width and height of the 2D planar image. As shown in Fig. 1A, the intersection (point P) between the longitude L4 and the latitude A1 on the sphere is mapped to the unique point q (Fig. 1B) in the 2D plane using equations (1) and (2). The point q in the 2D plane can be projected back to the point P on the sphere via backprojection. The field of view (FOV) in Figure 1B shows an example of mapping FOVs in a sphere to a 2D plane with a viewing angle along the X-axis of about 110 degrees.

Through ERP, 360-degree video can be mapped to regular 2D video. It can be encoded with an existing video codec such as H.264 or HEVC and then delivered to the client. On the user side, by projecting and displaying the portion of the FOV in the equidistant columnar image within the HMD, the equidistant columnar video is decoded and presented based on the user's view. Although spherical video can be converted to 2D planar images for encoding with equidistant cylindrical projections, the features of the equidistant cylindrical 2D images are different from those of conventional 2D images (also known as linear video). Figure 1C is a schematic illustration of an example equidistant columnar image inside a room. Since the equidistant columnar sampling in the 2D spatial domain is not uniform, the top of the image corresponding to the north pole and the bottom corresponding to the south pole are stretched compared to the middle portion of the image corresponding to the equator. Compared to motion in regular 2D video, the motion field in a 2D isometric columnar image becomes complex in the time direction.

Video codecs such as MPEG-2, H.264, and HEVC use a translation model to describe the motion field and do not effectively represent shape-changing motion in a 2D planar image of an equidistant columnar projection. Another disadvantage of equidistant columnar projections is that the area near the pole may be less interesting to the viewer and/or content provider than to the area closer to the equator. For example, a viewer may not focus on the top and bottom regions for any significant period of time. However, based on the warping effect, these regions are stretched into a large portion of the 2D plane after equidistant columnar projection, and compressing these regions may require a large number of bits.

Based on these observations, some processing methods are being studied to improve equidistant columnar image coding, for example by applying pre-processing such as smoothing to these pole regions to reduce the bandwidth required to encode them. In addition, different geometric projections have been proposed for representing 360 degree video, such as cube maps, equal areas, cylinders, pyramids, octahedrons, and the like. Among these projection methods, the most easily compressed geometry can be a cube map with a total of 6 faces, each face being a flat square. Figure 2A shows an example of cube texture geometry. The cube map consists of 6 square faces. Assuming that the radius of the tangent sphere is 1, the lateral length of each face (square) of the cube map is 2. Figure 2B shows an encapsulation method in which six faces are placed in a rectangle, which can be used for encoding and delivery. A schematic of an example image with a cubemap projection is shown in Figure 2C. The blank area (20) is a filled area to fill a rectangular image. For each face, the image looks the same as a regular 2D image. However, the boundaries of each face are not continuous. A line passing through two adjacent faces (eg, line 22 representing the junction between the wall and the ceiling) will bend at the boundary of the two faces. This means that the motion at the face boundaries will also be discontinuous.

Figures 3A through 3B show examples of geometrical structures of equal area projections. Unlike equidistant columnar projections, vertical sampling on a sphere is not based on an even spacing of pitch. The projection on the Y-axis of each sample latitude is evenly distributed to achieve the same area for each sample on the sphere. For those areas close to the pole area, the sampling in the vertical direction becomes more sparse. This also means there are more samples near the equator. In practice, this is desirable because the user typically views the area near the equator more frequently than the area near the pole. Figure 3C is a schematic diagram of an example image with an equal area projection. In the 3C figure, the area near the equator is enlarged, and the area near the pole is squeezed, as compared with the 1C figure.

Figure 4A shows an example of the geometry of an octahedral projection. The octahedron consists of 8 equilateral triangular faces. If the radius of the tangent sphere is 1, the lateral length of each triangle is √6. Fig. 4B shows a packaging method of arranging eight triangles into one rectangle. Figure 4C schematically shows an example diagram with an octahedral projection. Warping distortion is observed at the corners of the common boundary of two adjacent triangles, such as the distortion seen at the doorway 402.

In order to compare the coding efficiency of different geometric projection methods, Yu et al., M. Yu, H. Lakshman, B. Girod, "A Framework to Evaluate Omnidirectional Video Coding Scheme", IEEE International Symposium on Mixed and Augmented Reality (IEEE Hybrid and Augmented Reality) Latitude-based PSNR (L-PSNR) is proposed in International Conferences, 2015. It considers two factors: (1) uniform sampling on the sphere; (2) viewer's viewing behavior. It defines some samples that are evenly distributed over the sphere and also defines the weight of the sample based on its latitude. Distortion is measured by weighted mean square error (MSE) by considering all of these uniformly distributed samples. The weights are derived by tracking the viewer's perspective as the viewer observes the training sequences. If it is observed more frequently, the weight will be greater. According to these statistics, the weight near the equator is greater than the weight near the pole because the most interesting content is located near the equator. Using these evenly distributed samples on the sphere provides a measure to compare the performance of different projection methods. However, when different projections are applied, those predefined sphere samples may not be projected to the integer sampling position. If an interpolation filter based resampling method is applied, additional interpolation errors are introduced. If the nearest neighbor sampling is applied, uniform sampling cannot be guaranteed. Therefore, objective and subjective quality assessment methods remain the open theme of 360-degree video coding.

The 360-degree camera and stitching software support a wide range of equidistant column formats. In order to encode 360-degree video in cubemap geometry, the equidistant column format must be converted to a cubemap format. The relationship between equidistant columns and cube maps is as follows. In Figure 2A, each face relates to each of the three axes from the center of the sphere to the center of the face. "P" means positive, "N" means negative, PX means the direction from the center of the sphere along the positive X-axis, NX is the opposite direction of PX, and PY, NY, PZ and NZ are similarly marked. Then, there are six faces (PX, NX, PY, NY, PZ, NZ) corresponding to the front, back, top, bottom, right and left sides, respectively, and these faces are indexed from 0 to 5. Let Ps(X_s, Y_s, Z_s) be the point on the sphere with radius 1. Its yaw φ and pitch θ are expressed as follows: X_s = cos(θ)cos(φ) (3) Y_s = sin(θ) (4) Z_s = -cos(θ)sin(φ) (5)

Let Pf be the point on the cube map when extending the line from the ball to the center of the ball. In the case of no loss of generality, let Pf be on face NZ. The coordinates of Pf(X_f, Y_f, Z_f) can be calculated as: X_f = X_s/|Z_s| (6) Y_f = Y_s/|Z_s| (7) Z_f = -1 (8)

Where | x | is the absolute value of the variable x. Then, the coordinates of Pf(uc,vc) in the 2D plane of the face NZ are calculated as: uc = W*(1-X_f)/2 (9) vc = H*(1-Y_f)/2 (10)

From equations (3) through (10), the relationship between the coordinates (uc, vc) on a particular face and the coordinates (φ, θ) on the sphere in the cube map can be established. And from the equations (1) and (2), the relationship between the equidistant columnar points (ue, ve) and the points (φ, θ) on the sphere is known. Therefore, you can find the relationship between equidistant column geometry and cubemap geometry. The geometric mapping from cubemap to equidistant column can be summarized as follows. Given a point (uc, vc) on one face in a cube map, the output (ue, ve) on an equidistant columnar plane can be calculated as: 1) according to the relationship in equations (9) and (10), (uc,vc) Calculate the coordinates of the 3D point P_f on the surface; 2) Calculate the coordinates of the 3D point P_s on the sphere by P_f according to the relationship in equations (6), (7) and (8); 3) According to the equation (3), (4) and (5), using P_s to calculate (φ, θ) on the sphere; 4) according to the relationship in equations (1) and (2), from (φ, θ) Calculate the coordinates of the point (ue, ve) on the equidistant columnar image.

In order to represent a 360-degree video in a 2D image using a cube map, the six faces of the cube map can be encapsulated into a rectangular area, which is called a frame encapsulation. The frame encapsulated image is then processed (eg, encoded) as a regular 2D image. There are different frame packing configurations, such as 3x2 and 4x3. In a 3x2 configuration, six faces are packed into two columns, and three faces in one column. In a 4x3 configuration, four faces PX, NZ, NX, PZ are packaged in one column (eg, a center column), and faces PY and NY are packaged in two different columns (eg, top and bottom columns), respectively. Figure 2C uses a 4 x 3 frame encapsulation corresponding to the equidistant columnar image in Figure 1C.

In an exemplary scenario, 360-degree video in an equidistant column format is used as input, and it is desirable to convert the input to a cubemap format. Apply the following steps: 1) For each sample position (uc, vc) of the cubemap format, calculate the corresponding coordinates (ue, ve) of the equidistant column format by the method described above. 2) If the equidistant columnar coordinates (ue, ve) thus calculated are not at the integer sample position, an interpolation filter can be applied to obtain the sample values at its fractional position using the samples at their adjacent integer positions.

A workflow for a 360 degree video system is depicted in Figure 5. It includes a 360 degree video capture 502, for example using multiple cameras to capture video covering the entire sphere space. These images are then stitched together, for example, in an equidistant columnar geometry (504). The equidistant columnar geometry can be converted to another geometry (506), such as a cube map, for encoding, such as encoding with an existing video codec. Frame encapsulation 508 can be performed prior to encoding 510. The encoded video is delivered to the client via, for example, dynamic streaming or broadcast. At the receiver, the video is decoded (512), the decompressed frame is decapsulated (514) and converted (516) into display geometry (eg, equidistant columnar). It can then be used to render 518 via the viewfinder based on the user's perspective and displayed on the head mounted display 520.

In professional and/or consumer video applications, the chrominance component is typically subsampled to a smaller resolution than the luminance component. Chroma subsampling reduces the amount of video data to be encoded (and thus saves bandwidth and computational power) without significantly affecting video quality. For example, one of the widely used chroma formats is called the 4:2:0 chroma format, where two chroma components are subsampled to 1/4 of the luminance resolution (horizontal 1/2 and vertical 1/2). After chroma subsampling, the chroma sampling grid may have become different from the luma sampling grid. In Figure 5, the 360 degree video processed at each stage may be a chroma format in which the chroma components have been subsampled throughout the processing flow.

Figure 6 is a block diagram of one embodiment of a general block-based hybrid video coding system. The input video signal 102 is processed block by block. In HEVC, an extended block size (referred to as a "coding unit" or CU) is used to effectively compress high resolution (eg, 1080p and higher) video signals. In HEVC, the CU can reach 64x64 pixels. The CU can be further divided into prediction units or PUs for which a separate prediction method is applied. For each input video block (MB or CU), spatial prediction (160) and/or temporal prediction (162) may be performed. Spatial prediction (or "in-frame prediction") uses pixels from neighboring blocks that have been encoded in the same video image/slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in video signals. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses pixels from the already encoded video image to predict the current video block. Time prediction reduces the time redundancy inherent in video signals. The temporal prediction signal for a given video block is typically signaled by one or more motion vectors indicating the amount and direction of motion between the current block and its reference block. In addition, if multiple reference images are supported (as in the case of recent video coding standards such as H.264/AVC or HEVC), their reference image index is also sent for each video block; and the reference index is used to identify the time. The prediction signal is from which reference image in the reference image storage (164). After spatial and/or temporal prediction, the mode decision block (180) in the encoder selects the best prediction mode, for example based on a rate distortion optimization method. The prediction block is then subtracted from the current video block (116); and the prediction residuals are decorrelated using transform (104) and quantization (106) to achieve the target bit rate. The quantized residual coefficients are inverse quantized (110) and inverse transformed (112) to form a reconstructed residual, which is then added back to the prediction block (126) to form a reconstructed video block. Further loop filtering, such as deblocking filters, may be applied (166) to the reconstructed video block before the reconstructed video block is placed in the reference image store (164) and used to encode future video blocks. Adaptive loop filter. To form the output video bitstream 120, the coding mode (inter-frame or intra-frame), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit (108) for further compression and encapsulation. To form a bit stream.

Figure 7 is a general block diagram of a block based video decoder. Video bitstream 202 is first decapsulated at entropy decoding unit 208 and entropy decoded. The coding mode and prediction information are sent to spatial prediction unit 260 (if intra-frame coding) or temporal prediction unit 262 (if inter-frame coding) to form a prediction block. The residual transform coefficients are sent to inverse quantization unit 210 and inverse transform unit 212 to reconstruct the residual block. The prediction block and the residual block are then added together at 226. The reconstructed block may be further loop filtered before being stored in the reference image store 264. The reconstructed video in the reference image memory is then sent out to drive the display device and to predict future video blocks. Overview of an exemplary implementation

The 360 degree video data can be projected onto the 2D plane to encode the information using conventional 2D planar video coding. Since many kinds of geometric projections can be used to represent 360 degree data, and the projection data can be packaged into different configurations, this causes various problems.

One problem is that in order to properly reconstruct 360 video from decoded 2D planar video, geometry and frame encapsulation parameters should be available to the decoder to decapsulate the data and project it back into the 3D space from 2D space. For example, the cubemap format can be represented using different arrangements with different face orders, different face rotations, or different face sizes, such as 3x2, 4x3, 1x6, or 6x1. In addition, if a format different from the encoding format is used on the receiver side, it is also necessary to convert the geometry and frame encapsulation parameters to convert the encoding format to the desired format. For example, if the encoding format is a cubemap, but the display format is an equidistant column, you must convert it. In fact, when the file format multiplexer multiplexes these basic streams, it is better to capture the information from the video itself rather than relying on external relay data.

Another problem is that for some frame packing configurations, the unfolded faces are filled so that the resulting frame-packaged image forms a rectangular area, which may be beneficial for storage or compression purposes. For example, in the cubemap 4x3 format, additional pixels must be added to the top right and bottom right edges (see Figures 2B and 2C). Encoding these additional pixels consumes bits, but does not convey any meaningful information. Therefore, if the encoder skips these pixels, bit rate savings can be achieved. In this case, the compact configuration should be communicated to the decoder for proper reconstruction of the 360-degree video. In addition, unlike conventional 2D planar video, only a portion of 360 video (eg, video) is presented and displayed to the user at any time during replay of the video (see Figure 5). Statistics show that the observation probability around the equator is usually higher than the pole point, and the observation probability near the front view is higher than that near the back view. Therefore, identifying the information in the projection format will allow the encoder to identify these regions in the projected 2D video (ie, the equator vs pole and the front vs) and apply different coding strategies (eg, to the equator and/or the front region). Corresponding regions cost more bits and/or apply more complex optimization strategies, and spend less bits and/or applications for regions that correspond to poles and/or back regions. The strategy is to use the user's viewing behavior to allocate bits and/or computing resources in a smarter way.

Another problem is that existing codecs such as MPEG-2, H.264 and HEVC are designed for traditional 2D video, regardless of any attributes of the 360 degree data representation. For better compression efficiency, advanced 360 video encoding tools can take advantage of the full 3D representation, but due to the encoding performed on the projected 2D planar video, these tools may benefit from information about geometry and frame packaging. Thus, information about geometry and frame packing parameters can be available to both the encoder and the decoder to enable proper and more efficient encoding and decoding of 360 video. For example, in a cubemap format, the expanded face has only a few correctly positioned adjacent faces on the 2D planar video, which limits the ability of the codec to utilize redundant information between adjacent faces. However, if the codec has information about the 3D representation where each face of the cube has exactly 4 adjacent faces, then more redundant information can be utilized to reduce the amount of data that must be encoded.

Another problem is that the geometry and frame encapsulation parameters can vary over the duration of the 360-degree video. Therefore, if the geometry and frame encapsulation parameters change over time, these parameters should be available to both the encoder and the decoder for each frame of 360 video. For example, the encoding format can be changed from a cubemap to an equidistant column at a particular moment to achieve better compression performance, or the size of a particular cubemap surface collection can be changed to accommodate lower or higher bandwidth requirements in a particular video segment. .

The systems and methods disclosed herein address these and other issues.

In some embodiments, one or more of the above problems with 360 degree video coding are addressed by interfacing the geometry and frame encapsulation parameters in the bitstream with additional higher order syntax elements. In particular, the projection geometry type can be specified, including different parameters for the geometric face, to be positioned on the 2D planar video. 360 video parameters can be signaled at different levels. The following sections describe how to store projection format parameters at the video level (for example, video parameter set or VPS level) to minimize the amount of information that must be transmitted when different layers and/or sequences and/or images use the same projection format. . The other section below describes how to route a projection format at a sequence level (for example, a sequence parameter set or an SPS level), allowing different sequences of the same video to use different projection formats, or changing parameters associated with a given projection format. The other section below describes how to communicate a projection format at an image level (eg, image parameter set or PPS level), allowing different images of the same sequence to use different projection formats, or changing parameters associated with a given projection format. Another aspect of the systems and methods disclosed herein is to enable different geometric faces to be encoded with different quality factors. For example, in the cubemap format, the front, back, left, and right sides can be encoded with higher quality, while the top and bottom surfaces can be encoded with lower quality. This is because the viewer is more likely to see the area near the horizon than the area near the poles. In this way, 360 video can be encoded more efficiently.

In some embodiments, systems and methods are described for specifying the rotation of a geometric coordinate system relative to an absolute coordinate system. These systems and methods can be used to rotate the 3D geometry such that objects or regions of interest are projected into a collection of faces or faces that can be encoded with higher quality. Similarly, if the object or region of interest is split on several faces, which can reduce redundancy within each face, the geometric rotation can be used to define different orientations so that one or more important objects can be placed in one In-plane, making it possible to achieve better compression efficiency. In some cases, if this is not possible, for example, if the object is large and/or close enough to span 90 degrees in one or both of the horizontal and vertical directions, then these faces can be Rotate to place parts of important objects in one plane as much as possible. Due to the inherent nature of 3D geometry, when an object spans more than one face, its geometry "distorts" as it transitions from one face to the other, reducing correlation and coding efficiency. The ability to specify the projection direction to maximize the continuity of objects within a plane can improve coding efficiency. 360- degree video attribute messaging at the video level

Different projection geometries have different characteristics. For example, only one face is used for equidistant columnar projection and equal area projection. There is no problem with the face boundary, although the image is stretched. The cube map has six faces and has many face boundaries in the frame package image. Each image can be encoded with a different projection geometry, or encoded with the same geometry, but with a different face layout, size or quality. To this end, as shown in Table 1, in some embodiments a new set of parameters can be introduced for 360 video. Table 1 Video Parameter Set RBSP

In an exemplary embodiment, the flag vps_360_extension_flag may have the following semantics.

Vps_360_extension_flag: Specifies whether the video is 360-degree video. In this case, specific parameters and tools for efficient representation and compression of 360 video can be used. When not present, the value of vps_360_extension_flag can be inferred to be equal to zero.

At the video level, according to Table 2, in some embodiments the total number of projection formats used in different sequences and/or layers can be communicated. Table 2. Video Parameter Set 360 Extended Syntax

In an exemplary embodiment, the parameters of Table 2 may have the following semantics.

Vps_num_360_formats_minus1: Specifies the number of projection formats used in different sequences and/or layers (minus 1). When not present, the value of vps_num_360_formats_minus1 can be inferred to be equal to 0, indicating that only one projection format is used.

360_format_idx_present_flag: Specifies whether the syntax element vps_360_format_idx[i] exists. When not present, it can be inferred that the value of 360_format_idx_present_flag is equal to zero.

Vps_360_format_idx [i]: Specifies an index of the 360_format() syntax structure applicable to the layer of nuh_layer_id equal to layer_id_in_nuh [i] in the 360_format() syntax structure list in the VPS. When not present, the value of vps_rep_format_idx[i] can be inferred to be equal to Min(i, vps_num_rep_formats_minus1).

For this proposed grammatical structure, the projection format of each layer may be different in a multi-layer video stream. For example, rate distortion optimization can be used to determine the projection format of each layer at the encoder. The encoder can encode the current layer with all available projection formats and then measure the rate distortion cost. If the current layer is an enhancement layer, not only intra-frame and inter-frame prediction may be used for encoding in the same layer, but inter-layer prediction of another layer (eg, reference layer) of the same or different projection format may be used. When the projection format from the reference layer is different from the projection format of the current layer, the inter-layer prediction process may also include a projection format conversion. Finally, the final encoding can be done by selecting the projection format that results in the lowest rate distortion cost.

In some embodiments, the attributes of each projection format and associated parameters can be communicated according to Table 3. Table 3. 360 degree representation format syntax

In an exemplary embodiment, the parameters of Table 3 may have the following semantics.

Projection_geometry: Specifies the map index in Table 4 of the projection geometry used.

Geometry_rotation_param_present_flag: Specifies whether the syntax elements geometry_rotation_yaw, geometry_rotation_pitch, and geometry_rotation_roll exist. When not present, the value of geometry_rotation_param_present_flag can be inferred to be equal to zero.

Geometry_rotation_yaw: Specifies the rotation around the Y coordinate of the geometric coordinate system relative to the absolute coordinate system (see Figure 2A). When not present, the value of geometry_rotation_yaw can be inferred to be equal to zero.

Geometry_rotation_pitch: Specifies the rotation around the Z coordinate of the geometric coordinate system relative to the absolute coordinate system (see Figure 2A). When not present, the value of geometry_rotation_pitch can be inferred to be equal to zero.

Geometry_rotation_roll: Specifies the rotation of the X-axis (see Figure 2A) around the geometric coordinate system relative to the absolute coordinate system. When not present, the value of geometry_rotation_roll can be inferred to be equal to zero.

Compact_representation_enabled_flag: Specifies whether the sample or block used to fill the frame encapsulation image into the rectangular image is skipped by the encoder. When not present, the value of compact_representation_enabled_flag can be inferred to be equal to zero.

Loop_filter_across_faces_enabled_flag: Specifies whether loop filtering can be performed across the boundary. When not present, it can be inferred that the value of loop_filter_across_faces_enabled_flag is equal to one.

Num_face_rows: Specifies the number of faces in the frame wrapper image. When not present, the value of num_face_rows can be inferred to be equal to one.

Num_face_columns: Specifies the number of polygon lines in the frame encapsulation image. When not present, the value of num_face_columns can be inferred to be equal to one.

Note that num_face_rows_minus1 and num_face_columns_minus1 can be signaled instead of num_face_rows and num_face_columns to reduce the number of bits needed to encode these syntax elements.

Equal_face_size_flag: Specifies whether all faces share the same size (width and height are the same). When not present, the value of equal_face_size_flag can be inferred to be equal to zero. When equal_face_size_flag is set to 1, the width and height of all faces in the frame encapsulation image can be inferred based on the projection geometry. For example, for a cubemap projection, it can be inferred that the width in the luma samples of all faces in the frame encapsulation image is equal to pic_width_in_luma_samples / num_face_columns, and the height of the luma samples of all faces in the frame encapsulation image can be inferred to be equal to pic_height_in_luma_samples / num_face_rows . Note that the width and height of the luma samples for all faces in the frame encapsulation image should not be equal to 0 and should be an integer multiple of MinCbSizeY.

Face_qp_offset_enabled_flag specifies whether different QPs are used for different faces. When not present, it can be inferred that the value of face_qp_offset_enabled_flag is equal to zero.

Face_idx [i] [j]: Specifies the index of the faces of the i-th and j-th rows in the frame encapsulation image. For simple geometries with only a single face, such as equidistant columns or equal areas, the only face is face #0. For other geometries, you can use the preset number and positioning of the faces, as shown in Table 5 for cube and octahedral geometry.

Face_width_in_luma_samples [i] [j]: Specifies the width of the luma samples of the faces of the i-th and j-th rows in the frame encapsulation image. Techniques can be employed to prevent ambiguity regarding the width of the frame package image. For example, the sum of the different face widths along each column can be forcibly set equal to the image width of the frame package. Face_width_in_luma_samples [i] [j] must not be equal to 0 and should be an integer multiple of MinCbSizeY.

Face_height_in_luma_samples [i] [j]: Specifies the height of the luma samples of the faces of the i-th and j-th rows in the frame encapsulation image. Techniques can be employed to prevent ambiguity regarding the height of the frame package image. For example, the sum of the heights along different faces of each row can be forced to be equal to the frame package image height. Face_height_in_luma_samples [i] [j] must not be equal to 0 and should be an integer multiple of MinCbSizeY.

Face_rotation_idc [i] [j]: A mapping index in Table 6 that specifies the rotation between the image coordinate system and the area coordinate system of the faces of the i-th column and the j-th row in the frame encapsulation image. When not present, the value of face_rotation_idc [i] [j] can be inferred to be equal to zero.

Face_rotation [i] [j]: Specifies the degree of rotation between the image coordinate system and the area coordinate system of the faces of the i-th column and the j-th row in the frame encapsulation image.

Face_vertical_flip_flag [i] [j]: Specifies whether the faces in the i-th column and the j-th row in the frame encapsulation image are vertically flipped after being rotated. When not present, the value of face_vertical_flip_flag [i] [j] can be inferred to be equal to zero.

Face_qp_offset [i] [j]: Specifies the difference to be added to the sequence level QP when determining the QP value of the faces of the i-th column and the j-th row in the frame encapsulation image. Table 4. Projection Geometry Index Table 5. Preset surface definitions Table 6. Rotation index

Considering the frame-packaged image as a polygon mesh, these parameters can be used for very flexible and powerful messaging in geometric formats. For projection geometries that result in one-sided (eg, equidistant columns, equal areas, or cylinders), the parameters num_face_rows, num_face_columns, face_idx, face_width_in_luma_samples, face_height_in_luma_samples, and face_rotation can be inferred from geometry and image size. However, for other geometries such as cubemaps, octahedrons or icosahedrons, it is best to specify these parameters because the faces can be arranged differently or have different sizes. For example, as shown in Figures 9A-9C, the same cubemap projection can be packaged in different ways, such as (a) a 3x4 grid (Fig. 9A) or (b) a 2x3 grid (Fig. 9B). In the case of a 3x4 grid, face_idx can be set to a value higher than the actual number of faces, which can be inferred from the geometry to indicate that the position of the actual face is not included in the mesh. For example, we can set the parameters as follows: projection_geometry = 1 // cubemap face_idx[0][0] = 2 // face #2 face_idx[0][1] = 6 // invalid face_idx[0][2] = 7 // Invalid face_idx[0][3] = 8 // Invalid face_idx[1][0] = 1 // Face #1 face_idx[1][1] = 4 // Face #4 face_idx[1] [2] = 0 // face #0 face_idx[1][3] = 5 // face #5 face_idx[2][0] = 3 // face #3 face_idx[2][1] = 9 //invalid Face_idx[2][2] = 10 //invalid face_idx[2][3] = 11 //invalid face

In order to provide better detail in certain directions, certain faces can be encoded with higher resolution. This is because viewers are more likely to view certain areas than other areas, especially those near the front. In this way, 360 degree video can be encoded more efficiently. To do this, you can use the face_width_in_luma_samples and face_height_in_luma_samples parameters to specify different sizes for different faces. For example, in the cubemap format, the front can be encoded with a higher resolution than the other faces. As shown in Figure 9C, we can set the parameters as follows: projection_geometry = 1

Where W is the face width of all other faces except the face 0 (front) in the luma sample, and H is the face height of all other faces except the face 0 (front) in the luma sample.

It can be inferred from these parameters that the front spans 4 grid positions, because the size is twice that of other faces, and the information can be correctly retrieved.

The faces can be arranged to have different orientations. For example, as shown by the cube map projection, when compared to the 3x4 grid of Figure 9A, the faces "2", "1", and "3" are rotated 90 degrees counterclockwise in the 2x3 grid of Figure 9B. The face_rottion_idc parameter can be used to specify the rotation between the area coordinate system and the frame-packaged image coordinate system.

Grid systems can also be used for geometries with non-square faces (eg, triangular faces), as shown in Figures 11 and 12, respectively, as octahedrons and icosahedrons. Since some triangular faces are divided into two parts for compact representation (see Figures 11B and 12B), two right triangles can be used instead of one isosceles or an equilateral triangle to define a triangular face. The basic right triangle can be defined as shown in Figure 10A. Since the rotation is not sufficient to construct an isosceles triangle or an equilateral triangle using two right triangles, the rotation can be combined with a vertical flip (or, in some embodiments, a horizontal flip). Through this representation, the same syntax can be used for compact and non-compact representations with great flexibility. For example, to signal the compact octahedron shown in Figure 11B, the parameters can be set as follows:

The face_qp_delta parameter can be used to specify whether a particular face is encoded with a higher or lower quality. For example, similar results can be obtained by adjusting the quality at the slice or coding unit level. However, a slice can cover several faces, and the face is likely to contain multiple coding units, so it may be more efficient to directly communicate the quality difference for each face.

For regular frame encapsulation grids consisting of faces that have the same width (but different rows of different widths) along each row and along the same height of each column (but different columns of different heights), fewer parameters can be used. The communication plane properties are shown in Table 7. Table 7. 360 degree representation format alternative syntax

In an exemplary embodiment, the parameters of Table 7 may have the following semantics.

Num_face_rows_minus1: Specifies the number of facets (minus 1) in the frame encapsulation image. When not present, it can be inferred that the value of num_face_rows_minus1 is equal to zero.

Num_face_columns_minus1: Specifies the number of face lines (minus 1) in the frame encapsulation image. When not present, it can be inferred that the value of num_face_columns_minus1 is equal to zero.

Row_height_in_luma_samples[i]: Specifies the height in the luma samples of the face of the i-th column in the frame encapsulation image. For the last column, the height can be inferred to be equal to pic_height_in_luma_samples - . Row_height_in_luma_samples[i] shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.

Column_width_in_luma_samples[j]: Specifies the width in the luma sample of the face of the jth line in the frame encapsulation image. For the last line, the width can be inferred to be equal to pic_width_in_luma_samples - . Column_width_in_luma_samples[j] must not be equal to 0 and should be an integer multiple of MinCbSizeY.

The face attributes can also be routed in the face index order for irregular face shapes. Table 8 shows an example. Table 8. 360 degree representation format alternative syntax

In an exemplary embodiment, the parameters of Table 8 may have the following semantics.

Num_faces: Specifies the number of faces in the frame encapsulation image. When not present, it can be inferred that the value of num_faces is equal to one.

Note that instead of the signal num_faces, num_faces_minus1 can be signaled to reduce the number of bits needed to encode the syntax element.

Num_face_vertices[i]: Specifies the number of vertices on the i-th face. When not present, the value of num_face_vertices[i] can be inferred to be equal to 4 because the quadrilateral is the most common polygon type.

vertex_2D_pos_x[i][j]: specifies the x coordinate in the frame encapsulation image of the jth vertex of the i-th face.

Vertex_2D_pos_y[i][j]: Specifies the y coordinate in the frame encapsulation image of the jth vertex of the i-th face.

vertex_3D_pos_x[i][j]: specifies the x coordinate in the 3D coordinate system of the jth vertex of the i-th face.

vertex_3D_pos_y[i][j]: specifies the y coordinate in the 3D coordinate system of the jth vertex of the i-th face.

vertex_3D_pos_z[i][j]: specifies the z coordinate in the 3D coordinate system of the jth vertex of the i-th face.

You can use the vertex_3D_pos_x[i][j], vertex_3D_pos_y[i][j], and vertex_3D_pos_z[i][j] parameters to define user-specified polygon-based geometry in 3D space. These parameters can be used to map a sample from its position in the frame-packaged image to a corresponding position in the 3D geometry. This information may be utilized by advanced 360 video coding for better compression efficiency. For example, the codec can utilize redundant information in the 3D representation that is not collocated between adjacent faces in the frame encapsulated image. Sequence level 360 degree video attribute communication

At the sequence level, the projection format used can be signaled. To this end, as shown in Table 9, a new parameter set can be introduced for 360 video. Table 9. General Sequence Parameter Set RBSP Syntax

In an exemplary embodiment, the parameters of Table 9 may have the following semantics.

Sps_360_extension_flag: Specifies whether the sequence is 360-Video, in which case specific parameters and tools for efficient compression of 360 video can be used.

The projection format used can be communicated according to Table 10. Table 10. Sequence parameter set 360 extended syntax

In an exemplary embodiment, the parameters of Table 10 may have the following semantics.

Sps_num_360_formats_minus1: Specifies the number of projection formats used in the sequence (minus 1). When not present, the value of sps_num_360_formats_minus1 can be inferred to be equal to 0, indicating that only one projection format is used.

Sps_360_format_idx[i]: Specifies the index list in the 360_format() syntax structure list in the VPS of the 360_format() syntax structure used in the sequence. The value of sps_360_format_idx [i] can range from 0 to vps_num_360_formats_minus1, including the end value.

Note that all 360 video related parameters defined at the VPS level can be changed at the SPS level. Although not shown in Table 10, instead of using sps_360_format_idx to index the set of 360 video formats sent in the VPS, syntax elements similar to the syntax elements defined in Table 3 (eg, projection_geometry, face dimension parameters, face QP offset, etc.) ) can be directly communicated as part of the SPS extension to indicate the 360 video parameters of the video sequence referenced to this SPS. Image level 360 degree video attribute messaging

In some embodiments, to provide greater coding optimization, sequences can be encoded using different projection formats for different frames. In this case, the projection format can be communicated at the image level via an index in the set of projection formats that have been communicated at the VPS or SPS level. To this end, in some embodiments, as shown in Table 11, a new set of parameters can be introduced for 360 video. Table 11. General Image Parameter Set RBSP Syntax

In an exemplary embodiment, the parameters of Table 11 may have the following semantics.

Pps_360_extension_flag: Specifies whether the image referenced to this PPS contains specific parameters related to 360-degree video encoding. When not present, it can be inferred that the value of pps_360_extension_flag is equal to sps_360_extension_flag.

An example of a PPS extension of 360 video is provided in Tables 12 and 13. Table 12. Image Parameter Set 360 Extended Syntax Table 13. Code Region Table Syntax

In an exemplary embodiment, the parameters of Tables 12 and 13 may have the following semantics.

Pps_360_format_idx: Specifies the index in the set of projection geometries defined by the SPS referenced by this PPS. The value of pps_360_format_idx should be in the range of 0 to sps_num_360_formats_minus1, including the endpoint value. When not present, the value of pps_360_format_idx can be inferred to be equal to zero.

The pps_360_format_idx parameter is used to specify the projection format of the current image in the available projection formats listed at the sequence level. For example, if only equidistant bars and equal areas are available in the SPS sps_360_format_idx list, we use the index "0" to represent the equal area, and "1" to represent the equidistant bars, then the parameter can be set as follows: pps_360_format_idx = 0 // All images involving this PPS will be encoded in an equal area format pps_360_format_idx = 1 //All images involving this PPS are encoded in an equidistant column format.

Within the same video sequence, if different images are allowed to have different projection geometry formats, use a translational motion model with 2 motion parameters (horizontal and vertical displacement parameters, respectively) or use 4 or 6 motion parameters The affine-based motion model performs time motion compensation prediction and may not work very efficiently. Conversely, if the projection geometry of the current image is different from the projection geometry of its temporal reference image, geometric transformations can be performed to align the projection geometry between the current image and its temporal reference before applying the existing temporal motion compensated prediction. This can increase the efficiency of time prediction, albeit at the expense of higher computational complexity. When more than one time reference image is used in motion compensated prediction (eg, bi-prediction), the projection geometry can be aligned between the current image and all of its reference images before performing motion compensated prediction.

In an exemplary embodiment, the semantics of the coding_region_table() syntax structure can be as follows:

Full_sphere_range_coding_flag: Specifies whether the entire sphere range is encoded, or whether only a part of it is encoded. When not present, it can be inferred that the value of full_sphere_range_coding_flag is equal to one.

Pos_x_in_360_packed_frame: specifies the x coordinate of the upper left corner of the encoded image in the frame encapsulation image.

Pos_y_in_360_packed_frame: Specifies the y coordinate of the upper left corner of the encoded image in the frame encapsulation image.

Due to different limitations, such as bandwidth or memory limitation or decoding capabilities, only a portion of the entire sphere can be encoded. This information can be signaled using the full_sphere_range_coding_flag and associated pos_x_in_360_packed_frame and pos_y_in_360_packed_frame parameters. When full_sphere_range_coding_flag is set to 0, only the rectangular portion of the entire frame encapsulation image is encoded. The associated pos_x_in_360_packed_frame and pos_y_in_360_packed_frame parameters are then used to signal the upper left corner of the encoded image within the frame encapsulation image.

Figures 13A through 13B illustrate the use of finite sphere range encoding for cube map (Fig. 13A) and equidistant column (Fig. 13B) projections. In these examples, only the front area is encoded. Note that when using limited sphere range encoding, the constraints of link face width/height and coded image width/height should be disabled. As shown in FIG. 13A, the entire image represents a frame encapsulation image, and a rectangle 1305 defines an encoding region. As shown in Fig. 13B, the entire image represents a frame encapsulation image, and the rectangle 1310 defines an encoding area.

The coding_region_table() can also be transmitted at the VPS and/or PPS level for each projection format.

Note that some parameters defined at the SPS and/or VPS level may alternatively or additionally be communicated at the PPS level. For example, it is particularly advantageous to communicate the face QP offset parameter at the PPS level rather than at the VPS or SPS level as it allows for more flexibility to adjust the face quality of each individual face on the image level. For example, it allows for flexible adjustment of the face quality of each individual face based on the temporal level of the current frame encapsulation image in the hierarchy B prediction structure. For example, at a higher time level, the face QP offset can be set to a larger value for the non-front, and the face QP offset can be set to a smaller value (eg, 0) for the front. This ensures that regardless of the temporal level of the current image, the front is always encoded with a relatively high constant quality, while the other faces of the higher temporal image can be more quantized to save the bits.

Similarly, geometric rotation parameters (eg, geometry_rotation_yaw, geometry_rotation_pitch, and geometry_rotation_roll) can be defined and communicated at the PPS level rather than at the VPS or SPS level because it allows more flexibility to adjust the geometric rotation at the image level. In some embodiments, for content being encoded (eg, guided selection by video content), the recommended viewing direction is selected, wherein the recommended viewing direction can be changed during the video session. In such an embodiment, the geometric rotation parameter can be set according to the recommended viewing direction and coupled to the face QP offset parameter such that the object or region of interest is projected to the face encoded with the highest quality.

Figures 14A and 14B illustrate an exemplary alternative arrangement of faces in a frame-packaged image. Figures 14A and 14B each illustrate an arrangement of six faces, such as may be used in conjunction with cubemap projection. The arrangement of the faces in Figures 14A and 14B can be used as a user-specified geometry using the embodiments disclosed herein.

The exemplary embodiments disclosed herein are implemented using one or more wired and/or wireless network nodes, such as a wireless transmit/receive unit (WTRU) or other network entity.

Figure 15 is a system diagram of an exemplary WTRU 1502 that may be used as an encoder or decoder in the embodiments described herein. As shown in FIG. 15, the WTRU 1502 may include a processor 1518, a communication interface 1519 including a transceiver 1520, a transmission/reception component 1522, a speaker/microphone 1524, a keypad 1526, a display/touchpad 1528, and a non-removable Memory 1530, removable memory 1532, power supply 1534, global positioning system (GPS) chipset 1536, and sensor 1538. It is to be understood that the WTRU 1502 can include any subset of the above-described elements while remaining consistent with the above embodiments.

The processor 1518 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with the DSP core, a controller, a micro control , dedicated integrated circuit (ASIC), field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), state machine, etc. The processor 1518 can perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 1502 to operate in a wireless environment. The processor 1518 can be coupled to a transceiver 1520 that can be coupled to the transmit/receive element 1522. Although processor 1518 and transceiver 1520 are depicted as separate components in FIG. 15, it will be appreciated that processor 1518 and transceiver 1520 can be integrated together into an electronic package or wafer.

The transmit/receive element 1522 can be configured to transmit signals to, or receive signals from, the base station via the null plane 1516. For example, in one embodiment, the transmit/receive element 1522 can be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 1522 can be an emitter/detector configured to transmit and/or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit/receive element 1522 can be configured to transmit and receive both RF signals and optical signals. It is to be understood that the transmit/receive element 1522 can be configured to transmit and/or receive any combination of wireless signals.

Although the transmit/receive element 1522 is depicted as a single element in FIG. 15, the WTRU 1502 may include any number of transmit/receive elements 1522. More specifically, the WTRU 1502 may use MIMO technology. Thus, in one embodiment, the WTRU 1502 may include two or more transmit/receive elements 1522 (eg, multiple antennas) for transmitting and receiving wireless signals via the null intermediate plane 1516.

The transceiver 1520 can be configured to modulate a signal to be transmitted by the transmit/receive element 1522 and configured to demodulate a signal received by the transmit/receive element 1522. As noted above, the WTRU 1502 may have multi-mode capabilities. Thus, transceiver 1520 can include multiple transceivers for enabling WTRU 1502 to communicate via multiple RATs, such as UTRA and IEEE 802.11.

The processor 1518 of the WTRU 1502 can be coupled to a speaker/microphone 1524, a keypad 1526, and/or a display/touchpad 1528 (eg, a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), And the user input data can be received from the above device. The processor 1518 can also output user profiles to the speaker/microphone 1524, the keypad 1526, and/or the display/touchpad 1528. In addition, the processor 1518 can access information from any type of suitable memory and store the data in any type of suitable memory, such as non-removable memory 1530 and/or removable memory. Body 1532. Non-removable memory 1530 can include random access memory (RAM), readable memory (ROM), hard disk, or any other type of memory storage device. The removable memory 1532 may include a user identification module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 1518 can access data from memory that is not physically located on the WTRU 1502, such as on a server or a home computer (not shown), and store data in the memory.

The processor 1518 can receive power from the power source 1534 and can be configured to distribute power to other elements in the WTRU 1502 and/or to control power to other elements in the WTRU 1502. Power source 1534 can be any device suitable for powering WTRU 1502. For example, the power source 1534 can include one or more dry cells (NiCd, NiZn, NiMH, Li-ion, etc.), solar cells, fuel cells, and the like.

The processor 1518 can also be coupled to a GPS die set 1536 that can be configured to provide location information (eg, longitude and latitude) regarding the current location of the WTRU 1502. Additionally or alternatively to the information from the GPS chipset 1536, the WTRU 1502 may determine location information from the base station via the null plane 1516 and/or based on signal timing received from two or more neighboring base stations. position. It is to be understood that the WTRU 1502 can obtain location information using any suitable location determination method while consistent with the embodiments.

The processor 1518 can also be coupled to other peripheral devices 1538, which can include one or more software and/or hardware modules that provide additional features, functionality, and/or wireless or wired connections. For example, peripheral device 1538 can include sensors such as accelerometers, e-compass, satellite transceivers, digital cameras (for photos and/or video), universal serial bus (USB) ports, vibrating devices , TV transceivers, hands-free headsets, Bluetooth® modules, radio frequency (FM) radios, digital music players, media players, video game player modules, Internet browsers, and more.

Figure 16 depicts an exemplary network entity 1590 that may be used in embodiments of the present disclosure, for example as an encoder or decoder. As shown in FIG. 16, network entity 1590 includes communication interface 1592, processor 1594, and non-transitory data store 1596, all of which are communicatively coupled via bus, network, or other communication path 1598.

Communication interface 1592 can include one or more wired communication interfaces and/or one or more wireless communication interfaces. Regarding wired communication, as an example, communication interface 1592 can include one or more interfaces, such as an Ethernet interface. With respect to wireless communication, communication interface 1592 can include elements such as one or more antennas, one or more transceivers/chip sets designed and configured for one or more types of wireless (eg, LTE) communications, and/or related fields Any other component that the technician considers appropriate. Moreover, with regard to wireless communication, the communication interface 1592 can be configured with a scale and configuration suitable for execution on the network side (as opposed to the client side) of wireless communication (eg, LTE communication, Wi Fi communication, etc.). Thus, communication interface 1592 can include suitable devices and circuitry (possibly including multiple transceivers) for serving multiple mobile stations, UEs, or other access terminals in the coverage area.

Processor 1594 can include any type of one or more processors as deemed suitable by those skilled in the relevant art, some examples including general purpose microprocessors and dedicated DSPs.

The data store 1596 can take the form of any non-transitory computer readable medium or a combination of such media, some examples including flash memory, read only memory (ROM), and random access memory (RAM), to name a few. However, any one or more types of non-transitory data storage that are deemed suitable by those skilled in the relevant art can be used. As shown in FIG. 16, data store 1596 includes program instructions 1597 executable by processor 1594 for performing various combinations of the various network entity functions described herein.

It is noted that the various hardware components of one or more of the described embodiments are referred to as "modules" that are implemented (ie, executed, implemented, etc.) with the various modules and with the various functions described herein. As used herein, a given implementation module includes hardware as deemed suitable by those skilled in the relevant art (eg, one or more processors, one or more microprocessors, one or more microcontrollers, One or more microchips, one or more dedicated integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each of the described modules can also include instructions executable to perform one or more functions performed by the respective modules, and note that the instructions can take or include hardware (ie, hardwired) instructions, firmware instructions. The software instructions and/or the like may be stored in any suitable non-transitory computer readable medium or medium, such as commonly referred to as RAM, ROM, and the like. Although the features and elements of the present invention have been described above in terms of specific combinations, those skilled in the art can understand that each feature or element can be used alone or in the absence of other features and elements. Any other combination of features and elements of the invention is used in various situations. Moreover, the methods described herein can be implemented in a computer program, software or firmware executed by a computer or processor, where the computer program, software or firmware is embodied in a computer readable storage medium. Examples of computer readable storage media include, but are not limited to, read only memory (ROM), random access memory (RAM), scratchpad, cache memory, semiconductor storage devices, magnetic media (eg, internal hard drives) Or removable disk), magneto-optical media, and optical media such as CD-ROMs and digital versatile discs (DVDs). The software related processor can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

20‧‧‧Blank area

22‧‧‧ line

102‧‧‧Enter video signal

104‧‧‧Transformation

106‧‧‧Quantification

108, 208‧‧ Entropy coding unit

110, 210‧‧‧ inverse quantization

112, 212‧‧‧ inverse transformation

116‧‧‧Video Block

120, 202‧‧‧ bit flow

126‧‧‧ forecast block

160, 260‧‧‧ space forecast

162, 262‧‧‧ Time Forecast

164, 264‧‧‧ reference image storage

166‧‧•Video Block Application

180‧‧‧ mode decision block

402‧‧‧ Doorway

502‧‧‧360 degree video capture

508‧‧‧ Frame Assembly

510‧‧‧ code

520‧‧‧ head mounted display

1305, 1310‧‧‧ rectangle

1502‧‧‧Transmission/receiving unit (WTRU)

1516‧‧‧Intermediate mediation

1522‧‧‧Transmission/receiving components

1518, 1594‧‧‧ processor

1520‧‧‧ transceiver

1519, 1592‧‧‧Communication interface

1524‧‧‧Speaker/Microphone

1526‧‧‧Keypad

1528‧‧‧Display/Touchpad

1530‧‧‧ Non-removable memory

1532‧‧‧Removable memory

1534‧‧‧Power supply

1536‧‧‧Global Positioning System (GPS) chipset

1538‧‧‧Sensor, peripheral device

1590‧‧‧Network entities

1596‧‧‧Non-temporary data storage

1597‧‧‧ directive

A0, A1, A2, A3, A4, A5, A6‧‧‧ horizontal lines

FOV‧‧ ‧ field of view

P, Pf, Ps, q‧‧‧ points

NX, NXNY, NXPY, NY, NYNZ, NYPZ, NZ, PX, PXNY, PXPY, PY, PYNZ, PYPZ, PZ‧‧

φ‧‧‧Longitude

Θ‧‧‧ latitude

The description provided by the examples in conjunction with the following figures can be understood in more detail, wherein: Figure 1A shows an equidistant columnar projection of the spherical geometry using the longitude and latitude sampling in the sphere. Figure 1B shows a 2D planar equidistant columnar projection for the sample in Figure 1A, where the point P on the sphere in Figure 1A is projected onto the point q in the 2D plane. Figure 1C is a schematic diagram of an example image with equidistant columnar projections. Figure 2A shows a cubemap projection on a 3D geometry with faces PX(0), NX(1), PY(2), NY(3), PZ(4), NZ(5). Figure 2B shows the 2D plane of the six faces defined in Figure 2A. Figure 2C schematically shows an example diagram with a cubemap projection. Figure 3A shows sphere sampling in an equal area manner for equal area projection. Fig. 3B shows the 2D plane of the equal area projection of Fig. 3A, in which the point p on the sphere is projected to the point q in the 2D plane, and the latitudes of the horizontal lines (A0, A1, A2, etc.) are unequal intervals. Figure 3C schematically shows an example diagram with an equal area projection. Figure 4A shows an octahedral projection with a 3D geometry. Figure 4B shows a 2D planar package of the 3D structure of Figure 4A. Figure 4C schematically shows an example diagram with an octahedral projection. Figure 5 shows an embodiment of a 360 degree video processing workflow. Figure 6 shows an embodiment of a functional block diagram of a block based video encoder. Figure 7 shows an embodiment of a functional block diagram of a video decoder. Figure 8A illustrates one embodiment of a physical layout of a cubemap projection format. Figure 8B illustrates one embodiment of a physical layout of an octahedral projection format. Figure 9A shows a cube map in a 4 x 3 format. Figure 9B shows a cube map in a 3 x 2 format. Figure 9C shows a cube map in a 3 x 3 format with twice the size of the other faces (four times the area) (in this case, two columns and two rows are expanded in front). Figures 10A through 10H show the definition of the surface rotation of the triangular face: Figure 10A: 0° rotation; Figure 10B: 90° rotation; Figure 10C: 180° rotation; Figure 10D: 270° rotation; Figure 10E: 0° rotation, then vertical flip; 10F: 90° rotation, then vertical flip; 10G: 180° rotation, then vertical flip; 10H: 270° rotation, then vertical flip. Figure 11A shows a non-compact frame encapsulation format for octahedrons. Figure 11B shows a compact frame encapsulation format for octahedrons. Figure 12A shows a non-compact frame encapsulation format for an icosahedron. Figure 12B shows a compact frame packing format for icosahedrons. Figure 13A shows a finite sphere range encoding for a cube map where the full image represents the frame encapsulated image and the rectangle defines the encoded region. Figure 13B shows a finite sphere range encoding for equilateral edges, where the full image represents the frame encapsulation image, and the rectangle defines the encoded region. Figures 14A and 14B illustrate exemplary alternative arrangements of faces in a frame-packaged image, each showing an arrangement of six faces, such as may be used in conjunction with cubemap projection. Figure 15 illustrates an exemplary wireless transmit/receive unit (WTRU) that may be used as an encoder or decoder in some embodiments. Figure 16 illustrates an exemplary network entity that may be used as an encoder or decoder in some embodiments.

Claims (20)

A method for decoding 360-degree video encoded in a bit stream, the method comprising: receiving a bit stream encoding a 2D planar video, the bit stream including parameters identifying a projection geometry format; The identified projection geometry formats the 2D planar video to a 360 degree video. The method of claim 1, wherein the bit stream further includes a parameter indicating whether the bit stream encodes 360-degree video, wherein the parameter is executed as long as the parameter indicates that the bit stream represents 360-degree video. The 2D plane video is to the mapping of the 360 degree video. The method of claim 1, wherein the projection format comprises a projection geometry type, and wherein the parameter identifying the projection format comprises a parameter identifying the projection geometry type. The method of claim 3, wherein the parameter identifying the projection geometry type comprises an index of the identified projection geometry type. The method of claim 3, wherein the parameter identifying the projection geometry type identifies a geometric type selected from one or more of the following: equidistant columnar, cube map, equal area, eight sides Body, icosahedron, cylinder, and user-specified polygons. The method of claim 3, wherein the identified projection geometry type has a plurality of faces, and wherein the parameter identifying the projection geometry type includes an indication of the number of faces. The method of claim 3, wherein the identified projection geometry type has a plurality of faces, and wherein identifying the parameter of the projection geometry type comprises identifying an arrangement of the plurality of faces in the 2D planar video Frame encapsulation parameters. The method of claim 1, wherein the identified projection format has a plurality of faces, and wherein the bitstream further comprises parameters identifying quality levels of the plurality of faces in the 2D planar video. The method of claim 1, wherein the projection geometry comprises a geometric orientation of a projection geometry, and wherein the parameter identifying the projection geometry comprises a parameter identifying the geometric orientation. The method of claim 9, wherein the parameter identifying the geometric orientation comprises at least one of: a yaw parameter, a pitch parameter, and a rolling parameter. The method of claim 9, wherein the parameter identifying the geometric orientation comprises a parameter identifying the geometric orientation of an equidistant columnar projection, and wherein mapping the 2D planar video to a 360 degree video is Execution is performed using an equidistant columnar projection with the identified geometric orientation. The method of any one of clauses 1 to 11, wherein the parameter identifying the projection format is received in at least one video parameter set of the bitstream. The method of any one of clauses 1 to 11, wherein the parameter identifying the projection format is received in at least one sequence parameter set of the bitstream. A method for encoding 360-degree video, the method comprising: selecting a projection geometry format; mapping the 360-degree video to a 2D planar video using the selected projection geometry format; and imaging the 2D plane in a bit stream Encoding; and in the bitstream, the communication identifies parameters of the projected geometric format. The method of claim 14, further comprising transmitting, in the bitstream, a parameter indicating that the bitstream encodes 360-degree video. The method of claim 14, wherein selecting the projection geometry comprises selecting a geometric orientation of a projection geometry, and wherein the parameter communicated in the bitstream includes a one identifying the selected geometric orientation. parameter. The method of claim 16, wherein the parameter identifying the projected geometric orientation comprises a parameter identifying the geometric orientation of an equidistant cylindrical projection, and wherein the mapping of the 360-degree video to the 2D planar video It is performed using an equidistant columnar projection with the identified geometric orientation. The method of claim 16, wherein the projection geometry comprises a plurality of faces, and wherein the geometric orientation of the projection geometry is selected such that it falls within one of the plurality of faces within the 360 degree video A portion of a selected region of interest is substantially maximized. The method of claim 16, wherein the projection geometry comprises a plurality of faces comprising at least one face encoded with a higher quality level lower than at least one other face, and The geometric orientation of the projection geometry is selected such that a selected region of interest within the 360 degree video falls substantially over a portion of the face having the higher quality level. The method of claim 14, wherein selecting the projection format comprises selecting a geometry type, and wherein the parameter communicated in the bitstream includes a parameter identifying the selected geometry type.