TWI637356B - Method and apparatus for mapping omnidirectional image to a layout output format - Google Patents

Abstract

Methods and apparatus for processing omnidirectional images are disclosed. According to one method, a current set of omnidirectional images converted from each of the 360 degree panoramic video sequences using the selected projection format is received, wherein the selected projection format belongs to a projection format group including an aspect format, And the current omnidirectional image set having the format of the aspect is composed of six aspects. If the selected projection format corresponds to the aspect format, one or more mapping syntax elements that map the current omnidirectional image collection to the current cubemap image belonging to the output layout format set are identified. A codec material is provided in the bitstream, the bitstream including the one or more mapping syntax elements for the current omnidirectional image set.

Description

Method and apparatus for mapping an omnidirectional image to a layout output format 【cross reference】

The present application claims priority to U.S. Provisional Application No. 62/332,505, filed on Jan. 6, s., and U.S. Provisional Application Serial No. 62/356,571, filed on Jun. Quoted here.

The present invention relates to 360 degree video. In particular, the present invention relates to mapping an omnidirectional image converted from each spherical image in a 360 degree panoramic video sequence into an output format using the selected projection format.

Advances in digital video coding standards have led to the success of multimedia systems such as smart phones, digital televisions and digital cameras over the past decade. After the standardization activities of H.261, MPEG-1, MPEG-2, H.263, MPEG-4 and H.264/AVC, due to the need for larger image resolution and higher frame rate (frame rate) ), better video quality, the demand for improved video compression performance is still strong. Therefore, the development of new video codec technology with better codec efficiency than H.264/AVC has not ended. HEVC is based on mixing The motion compensation transform codec architecture of the block.

Three-dimensional (3D) televisions have been a technology trend in recent years, aiming to make the audience a sensational viewing experience. Various technologies have been developed to achieve 3D viewing, and multi-view video is one of the key technologies for 3D TV applications. For example, the video can be a two-dimensional (2D) medium that provides a single view of the scene to the viewer only from the perspective of the camera. However, multi-view video can provide any viewpoint of a dynamic scene and provide a realistic feeling to the viewer. The 3D video format may also include a depth map associated with the corresponding texture picture. The depth map must also be coded to render a 3D view or multiple views. Due to the strong need for encoding and decoding multi-view data, large image resolution and better quality, and increasing the efficiency of 3D and multi-view video coding efficiency, various techniques have been proposed.

As an extension of HEVC and next-generation 3D video codec standards, the standardization of 3D-HEVC video codec standards is developed by the Joint Collaborative Team on 3D video coding extension development (JCT-3V) joint collaboration group. It was officially launched in July 2012 and was confirmed after the 11th JCT-3V meeting in February 2015. In order to more practically support autostereoscopic multi-view display, a multi-view video plus depth (MVD) format is introduced as a new 3D video format for 3D-HEVC. The MVD format consists of a texture image and its associated depth map. Unlike texture images that represent the brightness and chrominance information of an object, a depth map is an image of a message that contains the distance from the object of the camera's shooting plane, and is typically used as a virtual non-visual information. View rendering.

Virtual reality (VR) with a head-mounted display (HMD) is associated with various applications. The ability to display wide field of view content to the user can be used to provide an immersive visual experience. The real world environment must be captured in all directions to produce omnidirectional video corresponding to the viewing sphere. With the advancement of camera rigs and HMDs, the delivery of VR content will quickly become a bottleneck due to the high bit rate required to represent such content. Since omnidirectional video usually has a resolution of 4K or higher, compression is critical to reducing the bit rate. The omnidirectional video provided is an equitctangular projection. FIG. 1 shows an example of an image of an omnidirectional video (referred to as "Hangpai_2") of an equidistant rectangular projection format. When the original image is full color, the black and white version is shown in Figure 1 (because the black and white image is sufficient to illustrate the invention).

The equidistant rectangular projection format can be converted to different formats such as shown in Figures 2a through 2k: (a) cubemap, (b) Cubemap_32, (c) Cubemap_180, (d) Plane_poles, (e ) P1ane_poles_6, (f) Plane_poles_cubemap, (g) Plane_cubemap, (h) Plane_cubemap_32, (i) Flat_fixed, (j) 180degree 3D video (ie, 180-3D), and (k) Cylindermap/Cylindrical . The images in FIGS. 2a to 2i are based on the image of FIG. 1.

FIG. 3 shows an example of converting an equidistant rectangular projection format into a cubic format using a projection conversion 310, wherein the images labeled 1 to 6 correspond to those for representing 360-degree video. An image of the six faces of the cube. Four common cloths are shown in Figure 4. Bureau (ie 1x6-layout 410, 2x3-layout 420, 3x2-layout 430 and 6x1-layout 440). In each layout, images from 6 faces are assembled into a single rectangular image. Figure 5 shows the geometric comparison between the equidistant rectangular format and the cubic format. An equidistant rectangular geometry 510 and a cubic geometry 520 are shown in FIG. Image 512 is an example of an equidistant rectangular format, and image 522 is an example of a cubic format.

In existing methods for converting an aspect to an output format, the same selected output layout format is always used, and six faces are assigned to the output layout format in a fixed manner. Although the fixed mapping is simple, it prevents users from using other layout formats to meet the needs of the user. Furthermore, after converting the aspect to the output layout format, the converted output image is often compressed to reduce the space required. The selected output layout format and fixed mapping may not be valid for compression.

Methods and apparatus for processing omnidirectional images are disclosed. According to one method, a current set of omnidirectional images converted from each of the 360 degree panoramic video sequences using the selected projection format is received, wherein the selected projection format belongs to a projection format group including an aspect format, And the current omnidirectional image set having the format of the aspect is composed of six aspects. If the selected projection format corresponds to the aspect format, one or more mapping syntax elements that map the current omnidirectional image collection to the current cubemap image belonging to the output layout format set are identified. A codec material is provided in the bitstream, the bitstream including the one or more mapping syntax elements for the current omnidirectional image set.

The projection format group may further include an equidistant rectangular format, a 180-3D format, and a cylindrical chart format. If the current omnidirectional image set is the equidistant rectangular projection format, converting the current omnidirectional image set into the vertical aspect format, and by considering the converted current omnidirectional image set as having the aspect Format, the one or more mapping syntax elements are identified for the current omnidirectional image collection of the transformation.

The mapping syntax element can include a current cube type associated with the current cubemap image, and the current cube type belongs to a current output layout consisting of a 1x6 cubemap layout, a 2x3 cubemap layout, a 3x2 cubemap layout, and a 6x1 cubemap layout. Format collection. The mapping syntax element can further include a layout mapping index, wherein each layout mapping index associates an aspect of the current omnidirectional image collection with a location of the current cubemap image. For each aspect of the current omnidirectional image collection, a layout map index is identified in addition to the last aspect of the current omnidirectional image collection. Each layout map index is coded using a code table having an entry number equal to the number of remaining facades to be mapped. In another embodiment, each layout map index is coded using a code table having an entry number equal to the number of entries to be mapped.

In another embodiment, the mapping syntax element further includes a rotation index, wherein each rotation index indicates a rotation angle of an aspect of the current omnidirectional image collection at the one location of the current cubemap image. A rotation index is identified for each aspect of the current omnidirectional image collection. Each rotation index is coded using a code table to indicate from {0° and 90°}, {0°, +90°, -90° and 180°} or {0°, 90°, 180° and 270°}selection angle A rotation angle selected in the collection.

The mapping syntax element also includes a default layout flag for indicating whether the current omnidirectional image collection having the current cube type uses a default cubemap image, and wherein only the default layout flag indicates that the default cubemap image is not When used for the current omnidirectional image collection, the layout map index and the rotation index are identified for the current omnidirectional image collection. If the default layout flag indicates that the default cubemap image is used for the current omnidirectional image collection, the default layout mapping index and the default rotation index are used for the current omnidirectional image collection.

The output layout format set for the current omnidirectional image collection may be identified in a 360 degree panoramic video sequence at a sequence level, a view level, an image level, a slice level, a sequence parameter set, a video parameter set, or an application parameter set. In the bit stream. The mapping syntax elements are identified in a bit stream of a 360 degree panoramic video sequence at a sequence level, view level, image level, slice level, sequence parameter set, video parameter set, or application parameter set.

The mapping syntax elements may be predictively identified based on one or more reference mapping syntax elements. In one embodiment, multiple sets of one or more reference mapping syntax elements are identified in a bitstream for a 360 degree panoramic video sequence at a sequence level, view level, or image level, and at a slice level or graph Like the level identification flag, the one or more mapping syntax elements are selected from the plurality of sets of one or more reference mapping syntax elements of the current omnidirectional image collection. In another embodiment, the reference mapping syntax element is predicted by one or more first mapping syntax elements from a previous picture, slice or frame.

410, 1210, 1510‧‧1x6-layout

420, 1220, 1520‧‧‧2×3-layout

430, 1230, 1530‧‧3×2-layout

440, 1240, 1540‧‧‧6×1-layout

510‧‧‧isometric rectangular geometry

520‧‧‧Cubic Geometry

512, 522, 1010, 1020, 1030‧‧‧ images

1310‧‧ ‧ default layout

1320‧‧‧3×2 cube layout

1410, 1110‧‧‧ Index

1420, 1120‧‧‧ equidistant rectangular

1610, 1611‧‧2x3 cube layout

1620~1625‧‧1x6 cube layout

1710, 1720, 1730, 1740‧‧ steps

Figure 1 shows an example of an image of omnidirectional video (referred to as "Hangpai_2") in an equidistant rectangular projection format.

Figure 2a-2k shows the inclusion of (Fig. 2a) cube map, (2b) Cubemap_32, (2c) Cubemap_180, (2d) Plane_poles, (2e) Plane_poles_6, (2f) Plane_poles_cubemap (2g) Plane_cubemap, (2h) Plane_cubemap_32, (2i) Flat_fixed, (2j) 180 degree 3D video (ie, 180-3D) and (2k) cylindrical/cylindrical Output layout format for different formats.

Figure 3 shows an example of converting an equidistant rectangular projection format to a cubic format using projection transformation, where images labeled 1 through 6 correspond to images on six faces of a cube representing 360 degree video. .

Four common layouts are shown in FIG. 4: 1x6-layout 410, 2x3-layout 420, 3x2-layout 430, and 6x1-layout 440.

A geometric comparison of the equidistant rectangular format and the cubic format is shown in Figure 5.

Figure 6 shows an example of mapping six vertical aspects to a 1x6 layout.

Figure 7 shows an example of mapping six vertical aspects to a 2x3 layout.

Figures 8a-8c illustrate an example of assigning six vertical aspects to a 1x6 layout, where (Fig. 8a), the face #5 in step 1 is assigned to the first position of the layout, (Fig. 8b), step 2, face# 4 is assigned to the second position of the layout, (Fig. 8c), in step 3, face #6 is assigned to the third position of the layout.

Fig. 9 shows an example of assigning six vertical aspects to a 1x6 layout in which an aspect #1 having a (-90 degree rotation) is assigned to an example of a fourth position in the layout.

Figure 10 shows an example of rotation prediction in which a reference layout is used to predict a target output layout.

Figure 11 shows an example of the representation of six vertical aspects of the facade index and the corresponding equidistant rectangular image.

The positional order in each of the cube layouts for the 1x6 layout, the 2x3 layout, the 3x2 layout, and the 6x1 layout is shown in FIG.

Figure 13 shows an example of a default layout for six vertical and target 3x2 cube layouts.

Figure 14 shows the default relative position of the facade and the index of the aspect index and the corresponding equidistant rectangle.

An exemplary sequence of locations in each of the 1x6 layout, 2x3 layout, 3x2 layout, and 6x1 layout is shown in FIG.

Figures 16a-16d show (6a) different 6x1 cube layouts, (16b) different 3x2 cube layouts, (16c) different 2x3 cube layouts, and (16d) different Other predefined layouts for the 1x6 cube layout.

Figure 17 shows an exemplary flow chart of a system processing an omnidirectional image in accordance with an embodiment of the present invention.

The following description is the best mode of concept for carrying out the invention. This description is made to illustrate the general principles of the invention and should not be considered as limiting. The scope of the invention is preferably determined by the appended claims.

In one aspect of the invention, a different set of output layouts is predefined, and in the bitstream, at the sequence level, view level, picture level, slice Level, sequence parameter set (short for SPS), video parameter set (VPS) or application parameter set (abstract for APS) to send an explicit flag to Select the output layout from a different set of output layouts. For example, a collection of different output layouts may include from stereoscopic layout, cubemap_32 layout, cubemap_180 layout, plane_poles layout, plane_poles_6 layout, plane_poles_cubemap layout, plane_cubemap layout, plane_cubemap_32 layout, flat_fixed layout, cubemap_1x6 layout, cubemap_2x3 layout, cubemap_3x2 layout, and cubmap_6x1 layout At least two output layout formats selected in the group.

In another example, the set of output layout formats includes only the cubemap_1x6 layout, the cubemap_2x3 layout, the cubemap_3x2 layout, and the cubmap_6x1 layout. Transfer flags to select the output layout in the cubemap_1x6 layout, cubemap_2x3 layout, cubemap_3x2 layout, and cubmap_6x1 layout.

Figures 6 and 7 show an example of mapping six vertical aspects to two possible layouts, where six vertical aspects map to the 1x6 layout in Figure 6, and six vertical aspects map to Figure 7. 2x3 layout in the middle.

According to one method of the present invention, six aspects can be assigned to possible layouts according to the following steps:

Step 1: Send (on the encoder side) / receive (on the decoder side) flag to assign one of the six vertical aspects to the first position in the layout. Figure 8a shows an example of assigning face #5 to the first position of the 1x6 layout.

Step 2: Send/receive the flag to the remaining five aspects One is assigned to the second position in the layout. Figure 8b shows an example of assigning face #4 to the second position of the 1x6 layout.

Step 3: Send/receive flags to assign one of the remaining four aspects to the third position in the layout. Figure 8c shows an example of assigning face #6 to the third position of the 1x6 layout.

Step 4: Send/receive flags to assign one of the remaining three aspects to the fourth position in the layout.

Step 5: Send/receive flags to assign one of the remaining two aspects to the fifth position in the layout.

Step 6: Assign the last remaining faces to the final position in the layout. Since there is only one remaining aspect, there is no need to identify the last one.

In the method proposed above, the truncated Unary code as shown in Table 1 can be used to transmit the flag. In step 1, the first marker is encoded using a truncated unary code of size 6 to select one of the six aspects. In step 2, the second token is encoded using a truncated unary code having a size of 5 to select one of the remaining five cubes, and so on.

In another embodiment, the rotation of the face is also defined to map the aspect to an output layout. For example, assign six vertical aspects to a possible layout according to the following steps: Step 1: Send (on the encoder side) / Receive (on the decoder side) flags to assign one of the six vertical aspects to the layout First position. Send/receive another flag to define the rotation of the selected face.

Step n (n from 2 to 5): Send/receive flags to assign one of the remaining (6-n+1) aspects to the nth position in the layout. Send/receive another flag to define the rotation of the selected face.

Step 6: Assign the last remaining faces to the final position in the layout. Send/receive another flag to define the rotation of the selected face.

The rotation of these six faces can also be transmitted after the six faces are assigned, as shown in the following steps: Step 1-a: Transmit (on the encoder side) / receive (on the decoder side) flag to assign one of the six vertical aspects to the first position in the layout.

Step n-a (n from 2 to 5): Send/receive flags to assign one of the remaining (6-n+1) aspects to the nth position in the layout.

Step 6-a: Assign the last remaining faces to the final position in the layout.

Step n-b (n from 1 to 6): Send/receive flags to define the rotation of the nth face.

On the other hand, the assignment of the face and the rotation of the face can also be combined into A flag, as shown in the following steps: Step 1: Send (on the encoder side) / Receive (on the decoder side) flag to assign one of the six vertical aspects, and its rotation (direction) to the first in the layout position.

Step n (n from 2 to 5): Send/receive flags to assign one of the remaining (6-n+1) vertical aspects and its rotation (direction) to the nth position in its layout.

Step 6: Assign the last remaining faces to the final position in the layout. Send/receive a flag to define the rotation of the sixth face.

The rotation of the face can be selected from the set {0 degrees, 90 degrees}. In another embodiment, the rotation of the face can also be selected from the set {0 degrees, 90 degrees, 180 degrees and 270 degrees} or {0 degrees, 90 degrees, -90 degrees, 180 degrees}. For example, in step 4, a flag is sent or received to assign the rotated aspect #1 to the fourth position in the layout. In this example, after steps 1 to 3, the remaining faces are {face #1, face #2, and face #3}. Based on the following truncated unary table, index 0 (non-negative) / index 1 (strictly positive) is sent using code "0" to assign face #1 to the fourth position shown in Table 2, and as shown in Table 3. The rotation code "10" (or the rotation code "1" shown in Table 4) is assigned to the defined -90 degree rotation.

Fig. 9 shows an example of assigning the aspect #1 having (-90 degree rotation) to the fourth position in the layout.

The proposed peers may be sent in the bitstream at the sequence level, view level, image level, slice level, SPS (sequence parameter set), VPS (video parameter set) or APS (adaptive parameter set) level. The method of mapping from the rectangular format to the layout. In another embodiment, as shown below, the mapping format of the N sets of equidistant rectangles to the output layout is marked in the sequence level, view level, image level, SPS, VPS or APS:

Mapping format:

Index 1: {face #1 (rotation angle), face #2 (rotation angle), face #3 (rotation angle), face #4 (rotation angle), face #5 (rotation angle), face #6 (rotation angle) )}

Index 2: {face #1 (rotation angle), face #2 (rotation angle), face #3 (rotation) Angle), face #5 (rotation angle), face #4 (rotation angle), face #6 (rotation angle)}

......

Index N: {face #2 (rotation angle), face #3 (rotation angle), face #1 (rotation angle), face #5 (rotation angle), face #4 (rotation angle), face #6 (rotation angle) )}

The flag is then further transmitted in the slice or image level to select a mapping format from {index 1, index 2, ..., index N}.

In another embodiment, the mapping format can be predicted from another mapping format. For example, a subsequent reference mapping format can be selected from N sets of mapping formats sent at the sequence level, view level, or image level. In yet another embodiment, the mapping format from the previous picture/slice/frame can be used as a reference mapping format to predict the current mapping format. The reference mapping format {face #1, face #2, face #3, face #4, face #5, face #6} is known for predicting the current mapping format.

Suppose the target mapping format is {face #1, face#2, face#4, face#3, face#5, face#6}, from {face#1, face#2, face#3, face#4, Face # 5, face # 6) predict the target mapping format, the prediction algorithm can be explained as follows: Step 1: The remaining faces in order are: {face # 1, face # 2, face # 3, face # 4, face # 5 , face # 6}. Index 0 is transmitted to determine the selected face #1 in the first position in the layout {face # 1,---,---,---,---,---}.

Step 2: The remaining faces in order are: {face # 2, face # 3, face # 4, face # 5, face # 6}. Index 0 is transmitted to determine the selected face #2 in the second position in the layout {face #1, face #2, ---, ---, ---, ---}.

Step 3: The remaining faces in order are: {face #3, face #4, face #5, face # 6}. Index 1 is transmitted to determine the selected face #4 in the third position in the layout {face #1, face #2, face #4, ---, ---, ---}.

Step 4: The remaining faces in order are: {face #3, face #5, face #6}. Index 0 is transmitted to determine the selected face #3 in the fourth position in the layout {face #1, face #2, face #4, face #3, ---, ---}.

Step 5: The remaining faces in order are: {face #5, face #6}. Index 0 is transmitted to determine the selected face #5 in the fifth position in the layout {face #1, face #2, face #4, face #3, face #5, ---}.

Step 6: The remaining face is {face #6}, which is selected as the final face of the layout {face #1, face #2, face #4, face #3, face #5, face #6}.

It is also possible to predict the rotation from the reference layout. Figure 10 shows an example of rotation prediction, where image 1010 corresponds to six elevations, image 1020 corresponds to a reference layout, and image 1030 corresponds to a target layout. In Figure 10, the first three positions in the target layout are rotated.

In the above example, the following steps can be used to predict the target layout from the reference layout. In this example, the reference layout is {face # 1 (0), face # 2 (0), face # 3 (0), face # 4 (0), face # 5 (0), face # 6 (0) }, the target layout is {face # 1 (-90), face # 2 (-90), face # 3 (-90), face # 5 (0), face # 4 (0), face # 6 (0) )}, where (0) represents 0 degrees and (-90) represents -90 degrees.

Step 1: The remaining faces in order are: {face # 1 (0), face # 2 (0), face # 3 (0), face # 4 (0), face # 5 (0), face # 6 ( 0)}. Index 0 is passed to the selected face #1 in the first position in the layout {face # 1,---,---,---,---,---}. As shown in Table 5, a truncated one of size 6 The metacode can be used to encode index 0, where the lines of the selected code are represented in bold italic font. The angular difference (-90) is further transmitted, and the final angle is reconstructed as: reference angle + angle difference = (0) + (-90) = (-90). The angular difference (-90) can be encoded using Rotation Encoding Type 1 in Table 6 or Rotation Encoding Type 2 (Table 4) in Table 7, where the selected line of code is represented in bold italic font.

Step 2: The set of remaining faces in order is: {face # 2 (0), face # 3 (0), face # 4 (0), face # 5 (0), face # 6 (0)}. Index 0 is passed to selected face #2 in the second position in the layout {face #1, face #2, ---, ---, ---, ---}. The angular difference (-90) is further passed. In this case, index 0 can be encoded using a truncated unary code of size 5 as shown in Table 8, where the lines of the selected code are represented in bold italic font.

Step 3: The set of remaining faces arranged in order is: {face # 3 (0), face # 4 (0), face # 5 (0), face # 6 (0)}. The index 0 is transmitted to the selected face #3 of the third position in the layout {face #1, face #2, face #3, ---, ---, ---}. Further pass the angle difference (-90). In this case, index 0 can be encoded using a truncated unary code of size 4 as shown in Table 9, which The selected line of code is represented in bold italic font.

Step 4: The set of remaining faces in order is: {face # 4 (0), face # 5 (0), face # 6 (0)}. Index 1 is sent to the selected face #5 of the fourth position in the layout {face #1, face #2, face #3, face #5, ---, ---}. Further send the angle difference (0). In this case, index 1 can be encoded using a truncated unary code of size 3 as shown in Table 10, where the selected line of code is represented in bold italic font.

Step 5: The set of remaining faces in order is: {face # 4(0), face # 6(0)}. Index 0 is sent to the selected face #4 of the 5th position in the layout {face #1, face #2, face #3, face #5, face #4, ...}. Further send the angle difference (0). In this case, a cut of size 2 as shown in Table 11 can be used. The broken unary code encodes index 0, where the lines of the selected code are represented in bold italic font.

Step 6: The remaining faces are {face #6} and are selected as the last position in the layout {face #1, face #2, face #3, face #5, face #4, face #6}. Further send the angle difference (0).

For example, you can first send an index in the picture/slice/frame level to select a mapping layout as a reference layout. The prediction method is then further used to predict the target layout of the current picture/slice/frame from the reference layout. In another example, the reference layout may be inherited from the previously encoded image/slice/frame and then the prediction method is applied to predict the target layout of the current image/slice/frame.

In the above proposed method, the truncated unary code can also be replaced by other entropy coding and decoding methods. For example, one of the following entropy codec methods can be applied to the present invention.

- ae(v): context adaptive arithmetic entropy codec syntax element.

- b(8): A bit string with any bit string (8-bit) mode. The parsing process of this descriptor is specified by the return value of the function read_bits(8).

- f(n): Uses a fixed-mode bit string written with n-bits written to the left bit first (from left to right). The parsing process for this descriptor is specified by the return value of the function read_bits(n).

- se(v): a 0th-order exponential Columbus codec syntax element with a signed integer, the left bit is the first.

- u(n): Use an unsigned integer of n bits. When n is "v" in the syntax table, the number of bits varies in a manner that depends on the values of other syntax elements. The parsing process of this descriptor is specified by the return value of the function read_bits(n), interpreted as a binary representation of the unsigned integer, with the most significant bit first written.

- ue(v): Unsigned integer 0th-order exponential Columbus codec syntax element, the left bit is the first.

An exemplary syntax design for layout signaling in accordance with an embodiment of the present invention is as follows:

In the above syntax table, the syntax num_of_layout_faces specifies the total number of faces in the layout. The syntax layout_face[i] specifies the index of the remaining (num_of_layout_faces-i) face as the corresponding face of the i-th position in the layout, and the value of layout_face[i] should be in the range of 0 to (num_of_layout_faces-i-1), including 0. And (num_of_layout_faces-i-1). The grammar layout_face[num_of_layout_faces-1] is inferred to be equal to the last remaining face.

An exemplary syntax design for layout signaling with rotation in accordance with an embodiment of the present invention is as follows:

In the above syntax table, the syntax num_of_layout_faces specifies the index in the remaining num_of_layout_faces-i faces as the corresponding face of the i-th position in the layout, and the value of layout_face[i] should be in the range of 0 to (num_of_layout_faces-i-1) , including 0 and (num_of_layout_faces-i-1). The grammar layout_face[num_of_layout_faces-1] is inferred to be equal to the last remaining face. The grammar layout_face[i] specifies the corresponding face rotation of the i-th position in the layout specified in Table 14 or Table 15, and the value of layout_face[i] should be 0 to 3 (or 1), including 0 and 3 (or 1 ).

In accordance with another embodiment, an exemplary syntax design for layout signaling is shown in Table 16, with syntax added for these six-part replacement and rotated message flags. The relative position of the default aspect. The initial vertical aspect array is equal to {Left, Front, Right, Rear, Top, Bottom}. The vertical index and the corresponding equidistant rectangular are shown in Figure 11.

In the above syntax table, num_of_layout_face_minus1 specifies the total number of faces in the layout. The syntax num_of_layout_faces is inferred to be num_of_layout_face_rminus1+1. The syntax layout_face[i] specifies the index in the remaining (num_of_layout_faces-i) plane as the corresponding face of the i-th position in the layout, and the value of layout_face[i] should be 0 to (num_of_layout_face_minus1-i) (including 0 and num_of_layout_face_minus1- Within the scope of i). The grammar layout_face[num_of_layout_face_minus1] is presumed to be the last remaining face. The grammar layout_rotation[i] specifies the corresponding face rotation of the i-th position in the layout specified in Table 17, and the value of layout_rotation[i] should be in the range of 0 to 3, including 0 and 3.

The positional order (i.e., layout_face[i] ) in each of the cube layouts in the 1x6 layout (1210), the 2x3 layout (1220), the 3x2 layout (1230), and the 6x1 layout (1240) is shown in FIG. Figure 13 shows an example of a default layout (1310) and a target 3x2 cube layout (1320) for six aspects.

An example of the landmark aspect layout according to the above embodiment is shown in the following table. For i=0, 1 and 2, the first three layout face selections correspond to the first position in the list (ie 0{Left}, 0{Front} and 0{Right}). Therefore, for i=0, 1 and 2, index 0 is selected. For i=3, select the second position in the list (ie, 1{Top}). Therefore, the flag index is 1. For i=4, select the second position in the list (ie 0{Rear}). Therefore, the flag index is 1. Since there is only one aspect left, there is no need to index the flag for the last aspect. However, the last aspect (ie i=5) still requires layout_rotation[i].

In another embodiment, the syntax design for layout signaling includes an additional syntax for transmitting the replacement and rotation of the six aspects. The default relative position of the aspect and its index are shown in Figure 14, which shows the aspect index (1410) and the corresponding equidistant rectangle (1420).

Exemplary syntax designs in accordance with the above embodiments are shown in Tables 19a and 19b. In Table 19a, mapping_extension() is defined by italicized text. The detailed information of mapping_extension() is shown in Table 19b.

In the above table, the syntax num_of_layout_face_minus1 specifies the total number of faces in the layout. The syntax num_of_layout_face is inferred to be num_of_layout_face_minus1+1. The grammar layout_face[i] specifies the vertical aspect index as the corresponding face of the i-th position in the layout, and the value [i] of the layout_face ranges from 0 to (num_of_layout_face_minus1) (including 0 and (num_of_layout_face_minusl)). The grammar layout_face[num_of_layout_face_minus1] is presumed to be the last remaining face. The grammar layout_rotation[i] specifies the corresponding face rotation of the i-th position in the layout specified in Table 20, and the value of layout_rotation[i] should be in the range of 0 to 3, including 0 and 3.

Figure 15 shows the 1x6 layout (1510), 2x3 layout (1520) Another exemplary order of positions in each cube layout (ie, layout_face [i]) in the 3x2 layout (1530) and the 6x1 layout (1540).

Another exemplary syntax design according to the above embodiment is shown in Table 21.

In the above table, mapping_format specifies the input mapping format of the 360-degree video as shown in the following table, where the value of mapping_format should be in the range of 0 to 3, including 0 and 3.

In Table 21, cubic_type specifies the cube type of the cube format as shown in Table 23, and the value of cubic_type should be in the range of 0 to 3, including 0 and 3.

The syntax default_layout_flag equal to 1 indicates that the cube layout follows the default layout specified in Table 24. The syntax default_layout_flag equal to 0 indicates that the default cube layout does not exist and explicitly marks the corresponding face and rotation of each position. Table 24 specifies the default layout. According to the sequence shown in Fig. 12, the respective faces and rotations of each position in the three-dimensional layout are designated as follows. A map from equidistant rectangular to vertical aspects is defined in Fig. 11, showing the vertical aspect index (1110) and the corresponding equidistant rectangular (1120).

In Table 21, layout_face[i] specifies the index in the remaining faces as the corresponding face of the i-th position in the layout. According to the order shown in Figure 12, each bit The faces and rotations (ie, layout_face[i] and layout_rotation[i]) are signaled iteratively as follows. The initial (face-select) array includes all six faces of the order of {Left, Front, Right, Rear, Top, Bottom}. For the first position, the layout_face[0] is flagged to indicate the index of the selected face in the array (ie {Left, Front, Right, Rear, Top, Bottom}) as the corresponding face of position 0. The value of layout_face[0] should be in the range 0 to 5, including 0 and 5. The face of the selected position 0 is then removed from the array of remaining candidates. For the ith position, the layout_face[i] is flagged to indicate the index in the updated array consisting of the remaining unselected faces as the corresponding face of position i. The value of layout_face[i] should be in the range 0 to 5-i, including 0 and 5-i. The corresponding selected face of position i is then removed from the remaining candidate array. For the last position, it is inferred to be the last remaining unselected face.

In Table 21, layout_rotation[i] specifies the corresponding face rotation of the i-th position in the layout specified in Table 20, and the value of layout_rotation[i] should be in the range of 0 to 3, including 0 and 3.

In accordance with another embodiment, an exemplary syntax design for layout signaling is shown in Table 25, with syntax added for the replacement and rotated message flags for the six aspects.

In another embodiment of the present invention, if the aspect is selected as the layout format, in the sequence level, view level, image level, slice level, tile level, SPS, VPS or APS of the bit stream Sending a second flag selects a cube type from a set of cube type sets including a 1 x 6 cube format, a 2 x 3 cube format, a 3 x 2 cube format, and 6 At least two of the ×1 cube formats.

In another embodiment, a flag is sent at the sequence level of the bitstream, view level, picture level, slice level, tile level, SPS, VPS or APS to indicate whether a predefined default of cube or other format is used. layout. If you do not use a predefined default layout of cubes or other formats, the layout is then displayed and transferred to the bitstream. The predefined layout of the aspect can be defined as Table 24.

Other predefined layouts are also available. For example, a different 6x1 cube layout can also be defined as shown in Figure 16a. A different 3x2 cube layout is shown in Figure 16b. Two different 2x3 cube layouts (1610 and 1611) are shown in Figure 16c. Six different 1x6 cube layouts (1620 to 1625) are shown in Figure 16d.

To explicitly mark the layout, corresponding faces and/or rotations associated with each respective face are transmitted in the bitstream. In one example, the initial (face-select) array includes at least two of {Left, Front, Right, Rear, Top, Bottom} with a given predefined order.

For the first location, the flag syntax indicates the index of the selected face in the given array as the corresponding face of the first location, and the value of the grammar should be between 0 and (N-1), where N represents the total number of faces.

For the ith position, another syntax is indicated to indicate the index in the updated array consisting of the remaining unselected faces as the corresponding face of position i, and the value of the grammar should be in the range of 0 to (N-1) , including 0 and (N-1). For the last position, it is inferred to be the last remaining unselected face.

In another example, the initial (face-select) array includes Given at least two of {Left, Front, Right, Rear, Top, Bottom} in a predefined order.

For the first location, the flag syntax is to indicate the index of the selected face in the given array as the corresponding face of the first location, and the value of the grammar should be in the range of 0 to (N-1), where N represents the total The number of faces. The selected faces of the first position are then removed from the array.

For the ith position, another syntax is indicated to indicate the index in the updated array consisting of the remaining unselected faces as the corresponding face of position i, and the value of the grammar should be 0 to (N-1-i) (including 0) And (N-1-i)) within the scope. The corresponding selected face of position i is then removed from the array. For the last position, it is inferred to be the last remaining unselected face.

To specify the corresponding face rotation for each position in the layout, the syntax for each position is sent to indicate at least one of the set of rotation candidates (which includes (-90 ⁰ , +90 ⁰ , +180 ⁰ , 0 ⁰ ) The corresponding face rotation is indicated in both). The specifications of the rotation index can be defined as shown in Table 20.

Figure 17 shows an exemplary flow chart of a system processing an omnidirectional image in accordance with an embodiment of the present invention. The steps shown in the flowcharts and other flowcharts of the present disclosure may be implemented as program code executable on one or more processors (eg, one or more CPUs) on the encoder side and/or the decoder side. . The steps shown in the flowcharts can also be implemented based on hardware such as one or more electronic devices or processors configured to perform the steps in the flowcharts. According to the method, a current omnidirectional image set for each spherical image transformation in a 360 degree panoramic video sequence using the selected projection format is received in step 1710, wherein the selected projection format belongs to a projection format including an aspect format Group format The current set of omnidirectional images consists of six vertical aspects. It is checked in step 1720 whether the selected projection format corresponds to the aspect format. If the selected projection format corresponds to the aspect format, steps 1730 and 1740 are performed. Otherwise (ie, the "no" path of the non-formal format), steps 1730 and 1740 are skipped. In step 1730, the flag maps the current omnidirectional image set to one or more mapping syntax elements in a current cube map image belonging to the output layout format set, wherein the output layout format set includes from the including cube map At least two output layout formats selected in the layout and layout groups of the equi-rectangular format. In step 1740, the current omnidirectional image group is provided with codec material in the bitstream containing the one or more mapping syntax elements.

The flowcharts shown above are intended to be illustrative of embodiments of the invention. The present invention can be practiced by a person skilled in the art by modifying various steps, separate or combining steps without departing from the spirit of the invention.

The above description is presented to enable a person of ordinary skill in the art to practice the invention in the context of the particular application. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the invention is not to be limited to the specific embodiments shown and described. In the above detailed description, various specific details are shown However, it will be understood by those skilled in the art that the present invention may be practiced.

Embodiments of the invention as described above may be implemented in various hardware, software codes, or a combination of both. For example, embodiments of the invention may be integrated into one or more electronic circuits in a video compression chip or integrated into video compression Program code in the software to perform the processing described herein. Embodiments of the invention may also be program code to be executed on a digital signal processor (DSP) to perform the processes described herein. The invention may also relate to a plurality of functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors may be configured to perform particular tasks in accordance with the present invention by executing machine readable software code or firmware code that defines a particular method embodied by the present invention. Software code or firmware can be developed in different programming languages and in different formats or styles. It is also possible to compile software code for different target platforms. However, the different code formats, the style and language of the software code, and other ways of configuring the code in accordance with the present invention will not depart from the spirit and scope of the present invention.

The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. The described examples are to be considered in all respects illustrative illustrative Therefore, the scope of the invention is applied for The scope of the patent is not indicated by the foregoing description. All changes that come within the meaning and range of equivalents of the claims of the claims are intended to be included.

Claims (19)

An omnidirectional image processing method, the method comprising: receiving, using a selected projection format, a current omnidirectional image set converted from each spherical image in a 360 degree panoramic video sequence, wherein the selected projection format belongs to a a projection format group comprising a facade format, and the current omnidirectional image collection having the facade format is composed of six aspects; and if the selected projection format corresponds to the aspect format: Identifying one or more mapping syntax elements to map the current omnidirectional image collection to a current cubemap image, the current cubemap image belonging to an output layout format set, the output layout format set including from a cube At least two output layout formats selected from a layout group and a layout group of an equidistant rectangular projection format; and providing codec data in a one-bit stream, the bitstream including the current omnidirectional image collection The one or more mapping syntax elements. The omnidirectional image processing method according to claim 1, wherein the projection format group further comprises a 180-3D format, a cylinder format, a cubemap_32 layout, a cubemap_180 layout, a plane_poles layout, a plane_poles_6 layout, a plane_poles_cubemap layout, and a plane_cubemap layout. , plane_cubemap_32 layout, flat_fixed layout, cubemap_1x6 layout, cubemap_2x3 layout, cubemap_3x2 layout, and cubmap_6x1 layout. The omnidirectional image processing method of claim 2, wherein if the current omnidirectional image set is the equidistant rectangular projection format, The front omnidirectional image set is converted to the vertical aspect format, and by considering the converted current omnidirectional image set as having the vertical aspect format, the one or more mapping syntax elements are identified for the current omnidirectional direction of the transformation Image collection. The omnidirectional image processing method of claim 1, wherein the one or more mapping syntax elements include a current cube type associated with the current cubemap image, and the current cube type belongs to a 1x6 cube A collection of current output layout formats consisting of a texture layout, a 2x3 cubemap layout, a 3x2 cubemap layout, and a 6x1 cubemap layout. The omnidirectional image processing method of claim 4, wherein the one or more mapping syntax elements further comprise a plurality of layout mapping indexes, wherein each layout mapping index is one of the current omnidirectional image collections The aspect is associated with a location of the current cubemap image. The omnidirectional image processing method of claim 5, wherein each of the layout map indexes is coded using a code table having an entry number equal to the number of vertical points to be mapped. The omnidirectional image processing method of claim 5, wherein for each aspect of the current omnidirectional image set, in addition to the last aspect of the current omnidirectional image set, the identification A layout map index. An omnidirectional image processing method according to claim 7, wherein each layout map index is coded using a code table having an entry number equal to the number of remaining facades to be mapped. An omnidirectional image processing method according to claim 5, wherein the one The one or more mapping syntax elements further include a plurality of rotation indices, wherein each rotation index indicates a rotation angle of an aspect of the current omnidirectional image collection at the one location of the current cubemap image. The omnidirectional image processing method of claim 9, wherein a rotation index is identified for each aspect of the current omnidirectional image collection. An omnidirectional image processing method according to claim 10, wherein each of the rotation indexes is coded and decoded using a code table to indicate that the correspondence corresponds to {0° and 90°}, {0°, +90°, -90° and 180°} or {0°, 90°, 180° and 270°} select one of the rotation angles selected in the set of angles. The omnidirectional image processing method of claim 9, wherein the one or more mapping syntax elements further comprise a default layout flag for indicating whether the current omnidirectional image set having the current cube type is Using a default cubemap image, and wherein the layout map index and the rotation index are identified for the current full only if the default layout flag indicates that the default cubemap image is not used for the current omnidirectional image collection Directional image collection. The omnidirectional image processing method of claim 9, wherein the one or more mapping syntax elements further comprise a default layout flag for indicating whether the current omnidirectional image set having the current cube type is Using a default cubemap image, and wherein if the default layout flag indicates that the default cubemap image is used for the current omnidirectional image collection, the default layout map index and the default spin index are used for the current omnidirectional image set. The omnidirectional image processing method of claim 1, wherein the output layout format set for the current omnidirectional image set is at a sequence level, a view level, an image level, a slice level, and a sequence parameter set. The set of video parameters, or the set of application parameters, is identified in the bitstream of the 360-degree panoramic video sequence. The omnidirectional image processing method of claim 1, wherein the one or more mapping syntax elements are at a sequence level, a view level, an image level, a slice level, a sequence parameter set, a video parameter set, or an application. The parameter set is identified in the bit stream of the 360 degree panoramic video sequence. The omnidirectional image processing method of claim 1, wherein the one or more mapping syntax elements are predictively identified based on one or more reference mapping syntax elements. The omnidirectional image processing method of claim 16, wherein the plurality of sets of one or more reference mapping syntax elements are identified at a sequence level, a view level, or an image level for use in 360 degree panoramic video. Identifying a flag in a stream of sequences and at a slice level or an image level to select the one or more of the plurality of sets of one or more reference mapping syntax elements of the current omnidirectional image set Map syntax elements. The omnidirectional image processing method of claim 16, wherein the one or more reference mapping syntax elements are predicted by one or more first mapping syntax elements from a previous picture, slice or frame. An omnidirectional image processing apparatus comprising one or more electronic circuits or processors arranged to receive each of a 360 degree panoramic video sequence using the selected projection format a current omnidirectional image set of spherical image transformations, wherein the selected projection format belongs to a projection format group, the projection format includes a vertical aspect format, and the current omnidirectional image set having the vertical aspect format is composed of six a personal aspect composition; and if the selected projection format corresponds to the aspect format: identifying one or more mapping syntax elements to map the current omnidirectional image collection to a current cubemap image, the current cubemap The image belongs to an output layout format set including from cube map layout, cubemap_32 layout, cubemap_180 layout, plane_poles layout, plane_poles_6 layout, plane_poles_cubemap layout, plane_cubemap layout, plane_cubemap_32 layout, flat_fixed layout, cubemap_1x6 layout, cubemap_2x3 layout At least two output layout formats selected from a layout set of cubemap_3x2, and a layout of the cubmap_6x1 layout; and a codec material in a one-bit stream, the bitstream including the one or for the current omnidirectional image collection Multiple images Syntax elements.