WO2022135133A1

WO2022135133A1 - Vr playback synchronization method and apparatus, storage medium, and electronic device

Info

Publication number: WO2022135133A1
Application number: PCT/CN2021/135864
Authority: WO
Inventors: 梁国柱; 朱兴昌; 张忠威
Original assignee: 中兴通讯股份有限公司
Priority date: 2020-12-24
Filing date: 2021-12-06
Publication date: 2022-06-30
Also published as: CN114679611A

Abstract

A VR playback synchronization method and apparatus, a storage medium and an electronic device. Said method comprises: a set-top box receiving a screen synchronization message sent by a VR device to perform VR synchronization playback between the set-top box and the VR device (S102); the set-top box acquiring corresponding media data from a media server according to the screen synchronization message, and performing VR playback (S104); the set-top box receiving synchronization parameter information sent by the VR device (S106); and the set-top box maintaining synchronization of the VR playback with the VR device according to the synchronization parameter information (S108).

Description

Synchronization method, device, storage medium and electronic device for VR playback

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number of 202011555793.5 and the filing date of December 24, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is incorporated herein by reference.

technical field

The embodiments of the present application relate to the field of VR video playback, and in particular, to a synchronization method, device, storage medium, and electronic device for VR playback.

Background technique

VR is the abbreviation of Virtual Reality (Virtual Reality) technology, and its basic implementation is to simulate a virtual environment through a computer to give people an immersive experience. With the advent of the 5G era and the gradual development of VR technology, VR headsets are gradually gaining popularity.

When users use VR headsets to watch VR movies or play VR games, usually only the wearer can see the picture, and there is a lack of interaction between multiple people. The VR headset is synchronized with the large screen of the TV, which can satisfy the simultaneous viewing of multiple people. At present, the synchronization is realized by connecting the transmission between the terminal head-mounted display and the large screen through a wired connection, so as to realize the picture synchronization between the two. However, the way of wired connection greatly limits the range of activities of the VR wearer, which is very inconvenient to use.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a method, device, storage medium, and electronic device for synchronizing VR playback, so as to solve one of the related technical problems at least to a certain extent, including VR caused by VR synchronous playback through wired connection The problem of limited range of user activities.

According to an embodiment of the present application, a method for synchronizing VR playback is provided, the method comprising: a set-top box receiving a same-screen message sent by a VR device to perform VR synchronous playback between the set-top box and the VR device; The set-top box obtains corresponding media data from the media server according to the same-screen message and performs VR playback; the set-top box receives the synchronization parameter information sent by the VR device; the set-top box maintains the synchronization parameter information with the VR device according to the synchronization parameter information. Synchronization of the VR playback.

According to another embodiment of the present application, a method for synchronizing VR playback is provided, the method comprising: a VR device sending a same-screen message to a set-top box to perform VR synchronous playback between the VR device and the set-top box, so that The set-top box obtains the corresponding media data from the media server and performs VR playback; the synchronization parameter information sent by the VR device to the set-top box, so that the set-top box can maintain the synchronization parameter information with the VR device according to the synchronization parameter information. Synchronization of VR playback.

According to yet another embodiment of the present application, a synchronization device for VR playback is provided, the synchronization device is located in a set-top box, and the device includes: a first receiving module configured to receive a data sent by a VR device for displaying between the set-top box and the set-top box. The same-screen message for performing VR synchronous playback between the VR devices; the acquiring module is configured to acquire corresponding media data from the media server according to the same-screen message and perform VR playback; the second receiving module is configured to receive the Synchronization parameter information sent by the VR device; a synchronization module configured to maintain the synchronization of the VR playback with the VR device according to the synchronization parameter information.

According to yet another embodiment of the present application, there is provided a synchronization device for VR playback, located in a VR device, the device includes: a first sending module configured to send to a set-top box to be displayed between the VR device and the set-top box The same-screen message for performing VR synchronous playback between the set-top boxes, so that the set-top box obtains corresponding media data from the media server and performs VR playback; the second sending module is set to the synchronization parameter information sent to the set-top box, so as to facilitate the said set-top box. The set-top box maintains the synchronization of the VR playback with the VR device according to the synchronization parameter information.

According to yet another embodiment of the present application, a computer-readable storage medium is also provided, and a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute any one of the above methods when running steps in the examples.

According to yet another embodiment of the present application, an electronic device is also provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any one of the above Steps in Method Examples.

Description of drawings

1 is a flowchart of a method for synchronizing VR playback according to an embodiment of the present application;

2 is a flowchart of a method for synchronizing VR playback according to another embodiment of the present application;

3 is a structural block diagram of a synchronization device for VR playback according to an embodiment of the present application;

4 is a structural block diagram of a synchronization device for VR playback according to another embodiment of the present application;

5 is a schematic structural diagram of a system for synchronizing pictures between a VR head-mounted display and a set-top box according to an embodiment of the present application;

FIG. 6 is a flowchart of screen synchronization between a VR head-mounted display and a set-top box according to an embodiment of the present application.

Detailed ways

Hereinafter, the embodiments of the present application will be described in detail with reference to the accompanying drawings and in conjunction with the embodiments.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence.

A method for synchronizing VR playback is provided in this embodiment. FIG. 1 is a flowchart of a method for synchronizing VR playback according to an embodiment of the present application. As shown in FIG. 1 , the process includes the following steps:

Step S102, the set-top box receives the same-screen message sent by the VR device to perform VR synchronous playback between the set-top box and the VR device;

Step S104, the set-top box obtains corresponding media data from the media server according to the on-screen message and performs VR playback;

Step S106, the set-top box receives the synchronization parameter information sent by the VR device;

Step S108, the set-top box maintains the synchronization of the VR playback with the VR device according to the synchronization parameter information.

Before step S102 in this embodiment, the method may further include: establishing a TCP connection between the set-top box and the VR device through DLNA or a transit server.

In this embodiment, the synchronization parameter information may include at least one of the following: posture information of the VR device, position information of the VR device, and playback progress information.

In step S108 of this embodiment, the set-top box calculates a posture matrix according to the posture information of the VR device, and maintains posture synchronization with the VR device according to the posture matrix; the set-top box calculates the posture matrix according to the posture information of the VR device; The playback progress information is synchronized with the playback progress of the VR device.

After step S108 in this embodiment, the method may further include: the set-top box receives the synchronization parameter information updated by the VR device at a predetermined frequency, and when the difference between the playback progress of the set-top box and the VR device reaches a preset value, Re-synchronize the playback progress.

After step S108 in this embodiment, the method may further include: after the set-top box receives a message for ending the same screen sent by the VR device, exiting the synchronization of the VR playback.

An embodiment of the present application also provides a synchronization method for VR playback. FIG. 2 is a flowchart of a method for synchronizing VR playback according to an embodiment of the present application. As shown in FIG. 2 , the flowchart includes the following steps:

Step S202, the VR device sends the same-screen message to the set-top box to perform VR synchronous playback between the VR device and the set-top box, so that the set-top box obtains corresponding media data from the media server and performs VR playback;

Step S204, the VR device sends the synchronization parameter information to the set-top box, so that the set-top box can maintain the synchronization of the VR playback with the VR device according to the synchronization parameter information.

Before step S202 in this embodiment, the method may further include: establishing a TCP connection between the VR device and the set-top box through DLNA or a transit server.

In this embodiment, the synchronization parameter information includes at least one of the following: posture information of the VR device, position information of the VR device, and playback progress information.

After step S204 in this embodiment, the method may further include: the VR device sending an end same-screen message to the set-top box instructing the set-top box to quit the VR playback synchronization.

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in the various embodiments of this application.

In this embodiment, a synchronization device for VR playback is also provided, and the device is used to implement the above-mentioned embodiment and other implementation manners, and what has been described will not be repeated. As used below, the term "module" or "unit" may be a combination of software and/or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

FIG. 3 is a structural block diagram of a synchronization apparatus for VR playback according to an embodiment of the present application. The synchronization device is located in the set-top box, which may be a functional component in the set-top box, or be co-located with the set-top box. As shown in Figure 3, the synchronization device includes:

The first receiving module 10 is configured to receive the same-screen message sent by the VR device to perform VR synchronous playback between the set-top box and the VR device;

The acquiring module 20 is configured to acquire corresponding media data from the media server according to the same-screen message and perform VR playback;

The second receiving module 30 is configured to receive the synchronization parameter information sent by the VR device;

The synchronization module 40 is configured to maintain the synchronization of the VR playback with the VR device according to the synchronization parameter information.

In this embodiment, the synchronization module 40 may further include: a first synchronization unit 41, configured to calculate an attitude matrix according to the attitude information of the VR device, and maintain attitude synchronization with the VR device according to the attitude matrix; The second synchronization unit 42 is configured to keep the playback progress synchronized with the VR device according to the position information of the VR device and the playback progress information.

FIG. 4 is a structural block diagram of a synchronization apparatus for VR playback according to an embodiment of the present application. The synchronization device is located in the VR device, and the synchronization device may be a functional component in the VR device. As shown in Figure 4, the synchronization device includes:

The first sending module 50 is configured to send the same-screen message to the set-top box to perform VR synchronous playback between the VR device and the set-top box, so that the set-top box obtains corresponding media data from the media server and performs VR playback. ;

The second sending module 60 is configured to send synchronization parameter information to the set-top box, so that the set-top box can maintain the synchronization of the VR playback with the VR device according to the synchronization parameter information.

It should be noted that the above modules can be implemented by software or hardware, and the latter can be implemented in the following ways, but not limited to this: the above modules are all located in the same processor; or, the above modules can be combined in any combination The forms are located in different processors.

In order to facilitate the understanding of the present application, the present application will be described in detail below with reference to specific application scenario embodiments.

The existing wired way to realize the real-time synchronization of VR pictures cannot meet the needs of users, so a more convenient and wireless technical solution for realizing the synchronization of the large-screen pictures of the VR terminal equipment and the set-top box is required.

To this end, this embodiment provides a system and method for real-time synchronization of a VR head-mounted display and a set-top box. In this embodiment, a link is established between the VR head-mounted display and the set-top box through DLNA or a relay server. If the user performs the same-screen operation during the VR head-mounted display and the game, the same-screen command is sent to the set-top box, and the set-top box synchronously requests the media server. data flow. At the same time, the VR head-mounted display synchronizes the posture, position, playback status and other information to the set-top box in real time through DLNA or a transit server. After the set-top box performs corresponding analysis and conversion, it is set to the player or game application on the set-top box to maintain the screen synchronization.

FIG. 5 is a structural diagram of a system for real-time synchronization of images between a VR head-mounted display and a set-top box according to an embodiment of the present application. As shown in Figure 5, the system includes four parts: VR head display, set-top box, DLNA or transit server, and media server.

In this embodiment, the VR head-mounted display terminal is responsible for VR playback and manipulation. A TCP connection is established between the VR head display and the set-top box. The VR head display obtains the media stream from the media server for VR video playback or games. When the VR head display performs the same screen operation, the same screen is sent through the established TCP link. message to the set-top box. The VR head display synchronizes attitude and position information to the set-top box in real time through the above link. After the same screen is completed, the VR head display terminal will actively disconnect the above link or automatically disconnect the link after timeout.

In this embodiment, the set-top box is responsible for receiving the real-time command of the screen of the VR head display terminal, parsing and responding to the execution, and outputting it to the large screen of the TV. The set-top box establishes a TCP connection with the VR head-mounted display. When receiving the same-screen command, it actively requests the media server to pull the stream, and receives the attitude and position information sent by the VR head-mounted display terminal in real time. Note that in this embodiment, the set-top box may be other functional devices, including smart TVs and the like.

In this embodiment, the DLNA or the transit server is responsible for the communication between the VR head-mounted display and the set-top box, and transmits the real-time picture synchronization command from the head-mounted display to the set-top box.

In this embodiment, a media server (CDN) is responsible for media content storage and distribution, and provides media streams for downloading and playing by terminal devices.

FIG. 6 is a flowchart of the real-time synchronization of the screen of the VR head display and the set-top box according to an embodiment of the present application. As shown in FIG. 6 , the real-time synchronization process of the screen of the VR head-mounted display and the set-top box may include the following steps:

Step S602, the set-top box terminal receives the DLNA broadcast and reports the device; the VR head-mounted display obtains a list of online set-top boxes and establishes a TCP link with the set-top box. When the VR head-mounted display and the set-top box are in the same local area network, it is realized by DLNA. When the VR head-mounted display and the set-top box are not in the local area network, Realized through a transit server;

Step S604, the VR head display terminal pulls the stream from the media server to play the VR video or play the game;

In step S606, when the VR terminal performs the same-screen operation, the same-screen message is sent to the set-top box through the established TCP link in step S602;

Step S608, after receiving the same-screen message, the set-top box requests media data from the media server, and performs VR video playback or game;

In step S610, the VR head-mounted display sends the posture of the head-mounted display unidirectionally through the link established in step S602, and the position message is sent to the set-top box, and the posture information is described by Euler angles or quaternions;

Step S612, the set-top box calculates the attitude matrix after receiving the on-screen message,

When the passed parameter is a quaternion: q=[λ P ₁ P ₂ P ₃ ];

The corresponding rotation matrix is:

In this embodiment, when the passed parameter is Euler angle: e=[α, β, γ]; the rotation order of the VR headset is external rotation Z-X-Y, and the rotation matrix is calculated according to this order:

After obtaining the attitude rotation matrix, and then rotate the matrix T _VR-STB according to the VR head-mounted display to the set-top box coordinate system to obtain the final rotation matrix: T _final =T _VR-STB ·T _rot , the set-top box side can rotate according to T _final and then The VR side maintains the posture synchronization, and the VR side needs to send the headset position and playback progress information synchronously, so that the video screen of the set-top box side can keep the progress synchronization;

In this embodiment, in order to ensure the same screen effect, the VR head display side needs to send attitude and position information at a sampling rate of, for example, 20 times or more per second, and when the playback progress is synchronized, in order to avoid multiple screen jumps, it is agreed that when When the playback progress difference between the VR terminal and the set-top box terminal reaches more than 2 seconds, the screen will be synchronized; when the VR terminal performs operations such as play, pause, time-shift, etc., it will synchronously send corresponding instructions to the set-top box terminal to ensure the screen synchronization;

Step S614, when it is necessary to end the same screen, the VR terminal sends an exit instruction to the set-top box, and the process ends.

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the steps in any one of the above method embodiments when running.

In one embodiment, the above-mentioned computer-readable storage medium may include, but is not limited to, a USB flash drive, a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), mobile Various media that can store computer programs, such as hard disks, magnetic disks, or optical disks.

Embodiments of the present application further provide an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above method embodiments.

In one embodiment, the electronic device may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

For specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiment and other implementation manners, and details are not described herein again in this embodiment.

In the above-mentioned embodiments of the present application, a technical solution for realizing real-time screen synchronization between a VR head-mounted display and a set-top box in a near-field within a local area network or in a far-field between different networks is provided, allowing the wearer of the VR head-mounted display to Simultaneously experience the fun of watching VR videos with onlookers.

In the above-mentioned embodiment of the present application, through the information interaction between the VR device and the set-top box, the set-top box can maintain the synchronization of the VR playback with the VR device according to the obtained synchronization parameter information, thereby solving the problem that the VR user's range of activities is limited. It enables VR users and bystanders to experience the viewing pleasure of VR videos simultaneously.

Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented by a general-purpose computing device, and they can be centralized on a single computing device, or distributed in a network composed of multiple computing devices On the other hand, they can be implemented in program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, can be performed in a different order than shown here. Or the described steps, or they are respectively made into individual integrated circuit modules, or a plurality of modules or steps in them are made into a single integrated circuit module to realize. As such, the present application is not limited to any particular combination of hardware and software.

The above descriptions are only some embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the principles of this application shall be included within the protection scope of this application.

Claims

A synchronization method for VR playback, including:

The set-top box receives the same-screen message sent by the VR device to perform VR synchronous playback between the set-top box and the VR device;

The set-top box obtains corresponding media data from the media server according to the same-screen message and performs VR playback;

receiving, by the set-top box, synchronization parameter information sent by the VR device;

The set-top box maintains the synchronization of the VR playback with the VR device according to the synchronization parameter information.
The method according to claim 1, wherein before the set-top box receives the same-screen message sent by the VR device to perform VR synchronous playback between the set-top box and the VR device, the method further comprises:

A TCP connection is established between the set-top box and the VR device through DLNA or a transit server.
The method according to claim 1, wherein the synchronization parameter information includes at least one of the following: posture information of the VR device, position information of the VR device, and playback progress information.
The method according to claim 3, wherein maintaining the synchronization of the VR playback by the set-top box with the VR device according to the synchronization parameter information comprises:

The set-top box calculates a posture matrix according to the posture information of the VR device, and maintains posture synchronization with the VR device according to the posture matrix;

The set-top box keeps the playback progress synchronized with the VR device according to the location information of the VR device and the playback progress information.
The method according to claim 4, wherein after the set-top box maintains the synchronization of the VR playback with the VR device according to the synchronization parameter information, the method further comprises:

The set-top box receives the synchronization parameter information updated by the VR device at a predetermined frequency, and re-synchronizes the playback progress when the playback progress difference between the set-top box and the VR device reaches a preset value.
The method according to claim 1, wherein after the set-top box maintains the synchronization of the VR playback with the VR device according to the synchronization parameter information, the method further comprises:

The set-top box exits the synchronization of the VR playback after receiving the end of the same-screen message sent by the VR device.
A synchronization method for VR playback, including:

The VR device sends the same-screen message to the set-top box to perform VR synchronous playback between the VR device and the set-top box, so that the set-top box can obtain corresponding media data from the media server and perform VR playback;

The synchronization parameter information sent by the VR device to the set-top box, so that the set-top box can maintain the synchronization of the VR playback with the VR device according to the synchronization parameter information.
The method according to claim 7, wherein before the VR device sends the same-screen message to the set-top box to perform VR synchronous playback between the VR device and the set-top box, the method further comprises:

A TCP connection is established between the VR device and the set-top box through DLNA or a transit server.
The method according to claim 7, wherein the synchronization parameter information includes at least one of the following: posture information of the VR device, position information of the VR device, and playback progress information.
The method according to claim 7, wherein after the set-top box and the VR device maintain the VR to play synchronously, the method further comprises:

The VR device sends to the set-top box an end same-screen message instructing the set-top box to quit the VR playback synchronization.
A synchronization device for VR playback, located in a set-top box, comprising:

The first receiving module is configured to receive the same-screen message sent by the VR device to perform VR synchronous playback between the set-top box and the VR device;

an acquisition module, configured to acquire corresponding media data from the media server according to the same-screen message and perform VR playback;

The second receiving module is configured to receive the synchronization parameter information sent by the VR device;

A synchronization module configured to maintain the synchronization of the VR playback with the VR device according to the synchronization parameter information.
The synchronization apparatus according to claim 11, wherein the synchronization parameter information includes at least one of the following: posture information of the VR device, position information of the VR device, and playback progress information.
The synchronization apparatus according to claim 12, wherein the synchronization module comprises:

a first synchronization unit, configured to calculate a posture matrix according to the posture information of the VR device, and maintain posture synchronization with the VR device according to the posture matrix;

The second synchronization unit is configured to maintain the playback progress synchronization with the VR device according to the position information of the VR device and the playback progress information.
A synchronization device for VR playback, located in a VR device, comprising:

The first sending module is configured to send the same-screen message to the set-top box to perform VR synchronous playback between the VR device and the set-top box, so that the set-top box obtains corresponding media data from the media server and performs VR playback;

The second sending module is configured to send synchronization parameter information to the set-top box, so that the set-top box can maintain the synchronization of the VR playback with the VR device according to the synchronization parameter information.
A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 10.
An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the claims 1 to 10 when executing the computer program The steps of the method of any one.