WO2018036112A1

WO2018036112A1 - Method and system for encoding/decoding panoramic video

Info

Publication number: WO2018036112A1
Application number: PCT/CN2017/073979
Authority: WO
Inventors: 孙其民; 李炜
Original assignee: 深圳市掌网科技股份有限公司
Priority date: 2016-08-23
Filing date: 2017-02-17
Publication date: 2018-03-01
Also published as: CN108322763A

Abstract

Provided are a method and system for encoding/decoding panoramic video, said method comprising: a panoramic camera photographing a panoramic video, then sending the panoramic video to a video-data caching unit; according to a viewing-angle division rule, said video-data caching unit dividing the panoramic video into a plurality of viewing-angle video sequences by means of an encoder; encoding each of the viewing-angle video sequences to obtain multiple sets of encoded data; a user equipment decoding said encoded data by means of a decoder and playing the obtained viewing-angle video sequences; the system comprises a panoramic camera, a communications unit, and a user equipment; the solution provided not only accurately determines the trend of a user's movement, but also ensures, according to a preprocessing unit, real-time decoding of the panoramic video corresponding to the effective field of view (FOV) in front of the viewing angle, reducing the decoding overhead.

Description

Method and system for encoding and decoding panoramic video

[0001] The present invention relates to the field of panoramic video codec technology, and more particularly to a method and system for encoding and decoding panoramic video.

Background technique

[0002] The panoramic video is the main content carrier of the virtual reality image display, and the panoramic video is a video composed of a series of panoramic image frames, and the intuitive image is dynamically displayed in the panoramic view. The panoramic image frame is an image frame formed by recording a scene of a predetermined viewing angle, for example, a predetermined viewing angle of 360° horizontally, 180° vertical, or a predetermined viewing angle of 360° horizontal, 270° vertical, and the like. After obtaining the panoramic video through the panoramic camera, the encoder needs to encode the panoramic video, and send the encoded panoramic video to the decoder for decoding. After decoding, the user can watch the panoramic video.

[0003] In the prior art, the encoder acquires all the panoramic image frames in the panoramic video, and encodes each panoramic image frame, using the encoding defined by the highly compressed digital video encoder standard H.264/AVC. The rules are encoded, or encoded according to the encoding rules defined by the high-efficiency video coding standard H.265; after encoding each panoramic image frame of the panoramic video, we obtain a set of rectangular regions in units of blocks, that is, the image frames. Encoding data; then transmitting each panoramic image frame to the decoder, and the decoder in the user equipment decodes the transmitted encoded panoramic image frame, and decodes all the panoramic image frames to form a complete video for the user to watch.

[0004] However, in practice, the user's viewing angle is usually around 150°, so that the encoded data of only 150° in the panoramic image frame is valid, and the remaining encoded data is invalid, and receiving and encoding the data may cause Unnecessary decoding credits.

technical problem

[0005] An object of the present invention is to provide a method and system for encoding and decoding panoramic video, which aims to solve the problem of redundant decoding and crediting of panoramic image frames in the existing virtual reality image, and how to ensure the real range of viewing angles. The image frame coding and decoding problem of the panoramic video corresponding to the FOV, and the difficulty in judging the trend of the irregular head movement are difficult. Problem solution

Technical solution

The present invention solves the technical problem thereof, and adopts the technical means: providing a method for encoding and decoding panoramic video, including using a panoramic camera, a communication unit, a user equipment, a camera group, an image data buffer unit, an encoder, and a positioning device The preprocessing unit and the decoder include the following steps:

[0007] Sl, sending a panoramic video to obtain a panoramic video to the image data buffer unit;

[0008] S2, the image data buffer unit divides each panoramic image frame of the panoramic video into A regions according to a horizontal perspective according to a view division rule, and A is an integer, and the panoramic video is divided according to a head motion trend. Multiple valid view video sequences;

[0009] S3. The encoder encodes an effective view video sequence composed of panoramic image frames of a specific viewing angle to obtain a plurality of sets of encoded data.

[0010] S4. Send corresponding encoded data to the user equipment according to the viewing angle of the user, and the user equipment decodes the encoded data by using a decoder, and plays the decoded video sequence of the view.

[0011] Preferably, step S3 comprises the following sub-steps:

[0012] S31. The panoramic camera captures video through cameras of different orientations;

[0013] S32, splicing each image frame captured by the camera group in the same engraving, and splicing into a panoramic image frame;

[0014] S33. According to the head motion trend data and the view angle division rule, the panoramic image frame of the specific view constitutes a valid panoramic video and is provided to the encoder.

[0015] Preferably, step S4 comprises the following sub-steps:

[0016] S41: collecting a trend of a user's head movement by using a positioning device in the user equipment;

[0017] S42. The encoder determines, according to the head motion trend data, a valid view video sequence corresponding to the predetermined area in the foregoing A regions.

[0018] S43. The pre-processing unit determines, according to the head motion trend data of the user collected by the positioning device, the size of the next full-frame image frame display gap.

[0019] Preferably, step S41 comprises the following sub-steps:

[0020] S411. Determine a reference location point, where the location of the point is different from a specific location of the user's field of view in the VR environment. Direction association

[0021] S412: Track each motion of the user's head in each direction from the reference position and each rotation motion to determine how to display the panoramic image frame to maintain a continuous and true effect of the VR environment;

[0022] S413. The user equipment sends the collected motion trend data to the panoramic camera.

[0023] According to the method of encoding and decoding panoramic video according to the present invention, the motion trend data includes angle and amplitude data of a detected head rotation of the user.

[0024] According to the method for encoding and decoding panoramic video according to the present invention, the crevice is determined by a certain perspective and a panoramic image, and is determined by the motion trend data of the user's head collected by the pre-processing unit.

[0025] According to the method for encoding and decoding panoramic video according to the present invention, the viewing angle division rules of all the image frames in one panoramic video video are the same, and the sum of the horizontal viewing angles of the A regional viewing angle video sequences is the total level of the panoramic image frames. The angle of view, and the regions in a panoramic image frame do not overlap each other.

[0026] A system for encoding and decoding panoramic video, including a panoramic camera, a communication unit, and a user equipment; the panoramic camera includes a camera group composed of cameras for capturing various orientation videos and a panoramic view for storing the camera group An image data buffer unit of the video, the image data buffer unit including an encoder for encoding a panoramic image frame of the panoramic video of a predetermined angle of view.

[0027] the user equipment includes positioning means for collecting user head motion trend data, a pre-processing unit for pre-determining the size of the next panoramic image frame in the user equipment, and for encoding The panoramic image frame is decoded by the decoder.

[0028] the panoramic camera captures a video through a camera of different orientations, and sends the captured panoramic video to the image data buffer unit, where the encoder in the image data buffer unit divides the panoramic video into a plurality of viewing angle video sequences, the encoder encoding each of the viewing angle video sequences to obtain a plurality of sets of encoded data; and using the motion trend data of the user's head collected by the positioning device to pass the corresponding panoramic encoded data through the communication The unit is sent to the user equipment; the pre-processing unit in the user equipment determines, according to the motion trend data of the user's head collected by the positioning device, the next partial range of the panoramic image frame display gap, and the control instruction And transmitting to the encoder and the decoder, where the encoder and the decoder encode and decode the panoramic image frame of the specific view according to the motion trend data and the control instruction.

Advantageous effects of the invention Beneficial effect

The method and system for implementing the codec panoramic video of the present invention have the following beneficial effects: The encoder in the image data buffer unit divides the panoramic video into multiple view video sequences according to the view division rule, by pairing each view The video sequence is encoded to obtain a plurality of sets of encoded data, and the positioning device is used to collect the motion trend data of the user's head, which not only can accurately determine the user's motion trend, but also ensures that the real-time view is effective according to the pre-processing unit. The panoramic video corresponding to the FOV range is decoded to reduce the decoding credit.

Brief description of the drawing

DRAWINGS

1 is a schematic diagram of a system embodiment of a codec panoramic video according to the present invention;

2 is a logic diagram of a panoramic camera embodiment of a system for encoding and decoding panoramic video according to the present invention;

3 is a schematic diagram of a user equipment of a system for encoding and decoding panoramic video according to the present invention;

4 is a schematic flowchart of a method for encoding and decoding panoramic video according to the present invention;

[0034] FIG. 5 is a schematic flowchart of step S3 in an embodiment of a method for encoding and decoding panoramic video according to the present invention;

6 is a schematic flowchart of step S4 in an embodiment of a method for encoding and decoding panoramic video according to the present invention;

7 is a schematic flowchart of step S41 in an embodiment of a method for encoding and decoding panoramic video according to the present invention.

Embodiments of the invention

[0037] The present invention will be further explained below in conjunction with the accompanying drawings and embodiments.

1 is a logic diagram of a system embodiment of a codec panoramic video according to the present invention. FIG. 2 is a schematic diagram of a panoramic camera embodiment of a system for encoding and decoding panoramic video according to the present invention, and FIG. 3 is a schematic diagram of a panoramic camera embodiment of the present invention. A schematic diagram of a user equipment of a system for encoding and decoding panoramic video. As shown in FIG. 1-3, the system 100 of the present invention includes a panoramic camera 110, a communication unit 120, and a user equipment 130. The panoramic camera 110 includes a camera group 111 composed of cameras of the azimuth video and an image data buffer unit 112 for storing the panoramic video captured by the camera group, the image data buffer unit 112 including a panoramic image frame for encoding the panoramic video of the predetermined angle of view. The user equipment 130 includes a positioning device 131 for collecting user head motion trend data, a pre-processing unit 132 for pre-determining the size of the next panoramic image frame in the user equipment, and For the encoded panoramic image frame A decoder 133 that performs decoding. In the prior art, the encoder acquires all the panoramic image frames in the panoramic video, and encodes each of the panoramic image frames, and encodes the encoding rules defined by the highly compressed digital video encoder standard H.264/A VC, or The encoding is performed according to the encoding rule defined by the high-efficiency video coding standard H.265; the decoder in the user equipment decodes the transmitted encoded panoramic image frame, and decodes all the panoramic image frames to form a complete video for the user to watch. However, in practice, the user's viewing angle is usually around 150°, so the encoded data of only 150° in the panoramic image frame is valid, and the rest of the encoded data is invalid, and receiving and encoding and decoding the data may cause unnecessary Decode the credit.

[0039] The technical solution proposed by the present invention is: providing a positioning device 131 in the user equipment 130, the positioning device 131 is provided with a gravity sensor and a gyro sensor to detect the movement trend data of the user's head, specifically reflecting the head movement The parameters of the six degrees of freedom (three spatial coordinates and three angular coordinates) accurately track the motion of the user's head, and the encoder 112a performs each panoramic image frame in the panoramic video according to a preset viewing angle division rule. Dividing, dividing it into A areas, and determining a valid view video sequence in the A areas according to the user head motion trend data sent by the positioning device 131, the encoder 11 2a encoding the same, and setting in the user equipment There is a pre-processing unit 132, which determines, according to the motion trend data of the user's head collected by the positioning device 1 31, the next partial-range panoramic image frame display gap, and sends a control command to the encoder 112a and The decoder 133, the decoder 133 decodes the received panoramic image frame of the effective view video, and the video Presented to the user to watch.

[0040] The technical solution proposed by the present invention effectively solves two technical problems: 1. The panoramic video encoding and decoding operation corresponding to the effective viewing angle range FOV before the viewing angle is ensured by the positioning device 131 and the angle of view division rule of the encoder; Through the positioning device 131 and the pre-processing unit 132, the cloud head movement tendency can be accurately determined, and the encoded video frame is selected in advance.

[0041] Specifically, the panoramic camera 110 captures video through the camera group 111 of different orientations, and transmits the captured panoramic video to the image data buffer unit 112. The encoder 112a in the image data buffer unit 112 will panorama according to the viewing angle division rule. The video is divided into a plurality of video sequences, and the encoder 112a encodes each video sequence to obtain a plurality of sets of encoded data. The motion trend data of the user's head collected by the positioning device 131 passes the corresponding panoramic encoded data through the communication unit. 120 is sent to the user equipment 130; the pre-processing unit 132 in the user equipment 130 determines the next partial range of panoramic image frame display gap according to the motion trend data of the user's head collected by the positioning device 131, and sends a control command to the Encoder 112a and solution The encoder 133, the encoder 112a and the decoder 133 encode and decode the panoramic image frame of a specific angle of view according to the motion trend data and the control command.

4 is a schematic flowchart of a method for encoding and decoding a panoramic video according to the present invention. FIG. 5 is a schematic flowchart of a step S3 in a method for encoding and decoding a panoramic video according to an embodiment of the present invention. FIG. FIG. 7 is a schematic flowchart of step S4 in a method for encoding and decoding panoramic video according to an embodiment of a method for encoding and decoding a panoramic video; FIG.

[0043] Please refer to FIG. 4-7. In the method for dynamically selecting and decoding panoramic video based on the perspective of the present invention, the method includes the following steps:

[0044] After the panoramic camera captures the panoramic video, the panoramic camera sends the panoramic video to the image data buffer unit 11 2; the camera group 111 of the panoramic camera may be distributed on the mechanism of the head mounted display device around the head, It may be separated from the user's VR display device; the panoramic video captured by the camera group 111 of the panoramic camera is stored in the image data buffer unit 112;

[0045] S2, the image data buffering unit 112 divides the panoramic video into a plurality of viewing angle video sequences according to the viewing angle division rule by the encoder 112a; the image data buffering unit 112 sends the received panoramic video to the encoder 112a for processing;

[0046] S3, encoding each view video sequence to obtain multiple sets of encoded data; the encoder 112a encodes each of the panoramic video frames in the panoramic video;

[0047] S4. The corresponding encoded data is sent to the user equipment 130 according to the viewing angle of the user, and the user equipment 130 decodes the encoded data through the decoder 133, and plays the decoded view video sequence.

[0048] Specifically, the view division rule is used to instruct the encoder 112a to divide the panoramic image frame according to the perspective of the viewer, and the view division rules of all the panoramic image frames in one panoramic video are the same. The sum of the horizontal viewing angles of the A video frames is the total horizontal viewing angle of the panoramic image frames, and the regions in one panoramic image frame do not overlap each other. For example, when the total horizontal angle of view of the panoramic image frame is 360°, the sum of the horizontal angles of the A view video sequences is 360°. For example, the range of the horizontal angle of view of each region in a panoramic image frame may be the same. It can also be different and is not limited. If the rule of view division is divided into 4 horizontal views on average, the range of each horizontal view is 90°, that is, 0°~90° is taken as a horizontal view. 90°~180° as a horizontal viewing angle, 180°~270° as a horizontal viewing angle, 270°~360° as a horizontal viewing angle; or in realizing 吋, the viewing angle division rule can be determined according to the viewing angle commonly used by users to improve The accuracy of the area division, for example, the viewing angle commonly used by users is 20°~175°, then the viewing angle division rule is to use 0°~20° as a horizontal angle of view, and 20°~175° as a horizontal angle of view, 175°~ 3 60° as a horizontal viewing angle.

[0049] Preferably, step S31 includes the following sub-steps: S31, the panoramic camera captures video through cameras of different orientations; S32, splicing and splicing each image frame captured by all cameras in the same engraving into a panoramic image frame; S33 Each panoramic video frame constitutes a panoramic video and is provided to the encoder 112a.

[0050] After performing encoding, it is necessary to determine the area number of the viewing angle range of each image frame. For example, if the viewing angle dividing rule is divided into three horizontal viewing angles, the horizontal viewing angle is 0° for the first frame of the panoramic image. The area of 12 0° is used as the viewing area 1, the area with the horizontal viewing angle of 120°~240° is taken as the viewing area 2, and the area with the horizontal viewing angle of 240°~360° is taken as the viewing area 3; for the second frame of the panoramic image, the horizontal viewing angle is used. The area of 0°~120° is used as the viewing area 1, the area with the horizontal viewing angle of 120°~240° is taken as the viewing area 2, and the area with the horizontal viewing angle of 240°~360° is taken as the viewing area 3; and so on, for the panoramic image In the N frame, a region having a horizontal viewing angle of 0° to 120° is regarded as a viewing zone 1, a region having a horizontal viewing angle of 120° to 240° is regarded as a viewing zone 2, and a region having a horizontal viewing angle of 240° to 360° is referred to as a viewing zone 3.

[0051] Preferably, step S4 includes the following sub-step S41, collecting the head motion trend of the user by using the positioning device 131 in the user equipment 130; S42. The encoder 112a determines the predetermined one of the A areas according to the head motion trend data. An effective view video sequence corresponding to the area; S43. The pre-processing unit 132 determines the size of the next panoramic image frame display gap according to the user's head motion trend data collected by the positioning device 131.

[0052] It should be noted that the gap proposed by the present invention does not display the scanning scan time used for displaying a picture in a full screen generally recognized in the field. The length of the gap in the present proposal is that the user equipment 130 is based on the positioning device 131. The actual monitoring of the collected motion data is determined; for example, in a certain interval, the user looks at the partial panoramic image displayed on the screen with a certain fixed viewing angle, and divides the rules according to the preset viewing angle, encoding and decoding A fixed horizontal viewing angle is presented. In another more active image interaction session, the user perceives the panoramic video in a volatility manner. Similarly, the encoder 112a and the decoder 133 display the images corresponding to the varying horizontal angles of view.

[0053] Step S41 includes the following sub-steps: S411, determining a reference location point, the location of the point and the VR environment The specific position and orientation of the user's field of view are associated; S412, tracking each movement of the user's head in each direction from the reference position and each rotation motion to determine how to display the panoramic image frame to maintain the VR environment continuously and The real effect; S413. The user equipment 130 sends the collected motion trend data to the panoramic camera.

[0054] In the present embodiment, the latest parameter data of six degrees of freedom (three spatial coordinates and three angular coordinates) is acquired by the gyro sensor and the gravity sensor provided in the positioning device 131, thereby ensuring display by graphic data. The field of view for the user is consistent with the field of view that should actually be displayed.

The method and system for encoding and decoding panoramic video of the present invention have the following beneficial effects: The encoder 112a in the image data buffering unit 112 divides the panoramic video into a plurality of viewing angle video sequences according to the viewing angle division rule, by pairing each The video sequence of the view is encoded to obtain a plurality of sets of coded data, and the positioning device 131 is used to collect the motion trend data of the user's head, which not only accurately determines the user's motion trend, but also ensures the actual pair according to the pre-processing unit 132. The panoramic video corresponding to the effective viewing angle range FOV before the viewing angle is decoded, which reduces the decoding credit.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. All modifications, equivalents, improvements, etc., made within the spirit and scope of the invention are intended to be included within the scope of the appended claims.

Claims

Claim

[Claim 1] A method of encoding and decoding panoramic video, using a panoramic camera (110), a communication unit

(120), user equipment (130), camera group (111), image data buffer unit (112), encoder (112a), positioning device (131), pre-processing unit (132), and decoder (133), It is characterized by the following steps:

51. The panoramic camera (110) captures the panoramic video and sends it to the image data buffer unit (112);

52. The image data buffer unit (112) divides each panoramic image frame of the panoramic video into A regions according to a horizontal perspective according to a view division rule by the encoder (112a), where A is an integer, and the panorama is based on the head motion trend. The video is divided into a plurality of valid viewing angle video sequences;

53. The encoder (112a) encodes an effective view video sequence composed of panoramic image frames of a specific view to obtain a plurality of sets of coded data;

54. The corresponding encoded data is sent to the user equipment (130) according to the viewing angle of the user, and the user equipment (130) decodes the encoded data by using a decoder (133), and the decoded perspective is obtained. The video sequence is played.

[Claim 2] A method for encoding and decoding panoramic video according to claim 1, wherein step S

3 includes the following substeps:

531, panoramic camera (110) shooting video through cameras of different orientations;

532. splicing and splicing the image frames captured by the camera group (111) in the same moment into a panoramic image frame;

533. According to the head motion trend data and the view division rule, the panoramic image frame of the specific view constitutes an effective panoramic video and is provided to the encoder (112a).

[Claim 3] A method for encoding and decoding panoramic video according to claim 1, wherein step S

4 includes the following substeps:

541. Collecting a trend of a user's head movement by using a positioning device in the user equipment (130);

542. The encoder (112a) determines, according to the head motion trend data, a valid view video sequence corresponding to the predetermined area in the foregoing A areas. S43. The pre-processing unit (132) determines, according to the user's head motion trend data collected by the positioning device (131), the size of the next panoramic image frame display gap.

[Claim 4] A method for encoding and decoding panoramic video according to claim 3, wherein: step S

41 includes the following substeps:

5411. Determine a reference location point, where the location is associated with a specific location and direction of a user's field of view in the VR environment;

5412. Track each motion of the user's head in each direction from the reference position and each rotation motion to determine how to display the panoramic image frame to maintain a continuous and true effect of the VR environment;

5413, User equipment (130) Send the collected motion trend data to the panoramic camera

[Claim 5] A method of encoding and decoding a panoramic video according to claim 1, wherein the motion trend data includes angle and amplitude data of a detected head rotation of the user.

[Claim 6] A method for encoding and decoding a panoramic video according to claim 1, wherein the crevice is a view angle, a panoramic image display length, and a user's head collected by the preprocessing unit The trend data of the movement is determined.

[Claim 7] A method for encoding and decoding panoramic video according to claim 1, wherein a view dividing rule of all image frames in one panoramic video video is the same, and a sum of horizontal viewing angles of A regional viewing angle video sequences It is the total horizontal viewing angle of the panoramic image frame, and the regions in one panoramic image frame do not overlap each other.

[Claim 8] A system for encoding and decoding panoramic video, comprising: a panoramic camera (110)

a communication unit (120) and a user equipment (130); the panoramic camera (110) includes a camera group (111) composed of cameras for capturing various orientation videos and an image for storing panoramic video captured by the camera group A data buffer unit (112), the image data buffer unit (112) including an encoder (112a) for encoding a panoramic image frame of a panoramic video of a predetermined angle of view.

[Claim 9] A system for encoding and decoding panoramic video according to claim 8, wherein the user equipment (130) includes positioning means for collecting user head motion trend data (1) 31), a pre-processing unit (132) for pre-determining the size of the gap in the user equipment for the next panoramic image frame, and a decoder (133) for decoding the encoded panoramic image frame.

[Claim 10] A system for encoding and decoding panoramic video according to claim 9, wherein the panoramic camera captures video through cameras of different orientations, and transmits the captured panoramic video to the image data buffer. The unit (112), the encoder (112a) in the image data buffer unit (112) divides the panoramic video into a plurality of view video sequences according to a view division rule, and the encoder (112a) performs a video sequence for each view. Encoding, obtaining a plurality of sets of encoded data; using the motion trend data of the user's head collected by the positioning device (131) to transmit corresponding panoramic encoded data to the user equipment (130) through the communication unit (120) The pre-processing unit (132) in the user equipment (130) determines, according to the motion trend data of the user's head collected by the positioning device (131), the next partial range of the panoramic image frame to display the gap, and controls An instruction is sent to the encoder (112a) and the decoder (133), the encoder (112a) and the decoder (133) according to Said movement trend data and the control instruction to the codec particular view panorama image frame.