CN114900508B - Method for transmitting VR application data based on webrtc - Google Patents

Method for transmitting VR application data based on webrtc Download PDF

Info

Publication number
CN114900508B
CN114900508B CN202210530055.8A CN202210530055A CN114900508B CN 114900508 B CN114900508 B CN 114900508B CN 202210530055 A CN202210530055 A CN 202210530055A CN 114900508 B CN114900508 B CN 114900508B
Authority
CN
China
Prior art keywords
data
rtp
server
client
header
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210530055.8A
Other languages
Chinese (zh)
Other versions
CN114900508A (en
Inventor
罗俊华
邹琼
周双全
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Ruiyun Technology Co ltd
Original Assignee
Shenzhen Rayvision Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Rayvision Technology Co ltd filed Critical Shenzhen Rayvision Technology Co ltd
Priority to CN202210530055.8A priority Critical patent/CN114900508B/en
Publication of CN114900508A publication Critical patent/CN114900508A/en
Application granted granted Critical
Publication of CN114900508B publication Critical patent/CN114900508B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80—Responding to QoS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
    • H04L69/163—In-band adaptation of TCP data exchange; In-band control procedures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

The invention discloses a method for transmitting VR application data based on webrtc, which comprises the steps of pre-expanding RTP head, wherein the expanded content comprises the following steps: URI is added to rtp_parameters.cc; adding an extension type in rtp_rtcp_defined. Defining extension data and a data structure in rtp_header_extensions.h; the data is packed and unpacked in rtp_header_extensions. According to the invention, the RTP header is expanded, the required VR control information is transmitted in the media packet, the control information and the media data are achieved at the same time, and the processing logic of the client is reduced; the control information of the data channel does not need to be waited, and the delay is reduced; the IP header and the UDP header added by the transmission layer are wasted without independently sending the control information block, so that the bandwidth is reduced; the smoothness of the image is obviously improved in the project use process.

Description

Method for transmitting VR application data based on webrtc
Technical Field
The invention relates to the technical field of VR (virtual reality), in particular to a method for transmitting VR application data based on webtc.
Background
In the conventional VR data transmission mode, UDP or TCP is generally used for transmission, and some users use webtc to transmit video through a media channel, and the data channel transmits VR control information, gesture data, keys and the like. In the prior art, if UDP or TCP is directly used, a screen is jumped or blocked when the packet is lost, so that the user experience is not smooth enough; the webrtc is directly used, the media channel and the data channel are different in transmission strategy, VR control information is not timely arrived, or subsequent processing logic is complex, so that the blocking and delay are increased to a certain extent, and the user experience is also influenced.
Accordingly, the prior art has drawbacks and needs improvement.
Disclosure of Invention
The invention aims to solve the technical problems that: the method for transmitting VR application data based on webrtc is provided, the processing difficulty of a client is reduced, the blocking and delay of the client are reduced, and the user experience is improved.
The technical scheme of the invention is as follows: there is provided a method of transmitting VR application data based on webtc, comprising the following steps.
S1: expanding the RTP header; the extended content includes: URI is added to rtp_parameters.cc; adding an extension type in rtp_rtcp_defined. Defining extension data and a data structure in rtp_header_extensions.h; the data is packed and unpacked in rtp_header_extensions.
S2: the VR client connects to the signaling server through TCP.
S3: the VR server is connected to the signaling server through TCP.
S4: the VR client calls the CreateOffer of webrtc to create an offer SDP; SDP, session Description Protocol, is a control protocol used to describe multimedia sessions.
S5: sendOffer1: the VR client communicates the offer SDP to the signaling server over TCP.
S6: sendOffer2: the signaling server forwards the offer SDP to the VR server.
S7: createAnswer: the VR server receives the offer and invokes the createensiwer method of webrtc to create an answer SDP.
S8: sendAnswer1: the VR server communicates the answer SDP to the signaling server over TCP.
S9: sendAnswer2: the signaling server forwards the answer SDP to the VR client.
S10: P2P media communication establishment procedure.
S11: and the VR client sends the gesture index and gesture data of the hardware provided with the VR client to the VR server.
S12: the VR server side sends VR video data to the VR client side, wherein the RTP extension header contains newly defined extension data: attitude index and frame transmission time.
The scheme is based on the webrtc transmission, can fully utilize the existing transmission technology of the webrtc, such as packet loss retransmission, FEC, transmission bandwidth evaluation and the like, and is superior to the direct use of UDP or TCP in the aspect of real-time transmission; meanwhile, by expanding the RTP header, information in the aspect of VR control is added into the RTP expansion header and is transmitted together with media data in a media channel, so that the problems of blocking and time delay caused by inconsistent transmission strategies of the media channel and the data channel are solved, and user experience is improved.
Further, the gesture data includes: position information and direction information of hardware.
Further, the URI is: rayvision: rayxr: vr-control-info.
Further, the extension type is: RTPExtensionType kRtpExtensionVRConrtolInfo.
Further, the data structure includes: an ID; len: a data length; pomeid: posture index, VR customer end produces; framesettime: frame transmission time, VR server side generation.
Further, the hardware is a head-mounted VR display device.
Further, the VR server receives the gesture data of the hardware, renders the image according to the gesture data, and returns the rendered gesture data to the VR client.
By adopting the scheme, the invention provides a method for transmitting VR application data based on webtc, which transmits required VR control information in a media packet by expanding an RTP header, and the control information and the media data are achieved simultaneously, so that the processing logic of a client is reduced; the control information of the data channel does not need to be waited, and the delay is reduced; the IP header and the UDP header added by the transmission layer are wasted without independently sending the control information block, so that the bandwidth is reduced; the smoothness of the image is obviously improved in the project use process.
Drawings
Fig. 1 is a flowchart illustrating steps between a VR client, a signaling server, and a VR server in accordance with the present invention;
fig. 2 is a schematic diagram of a data structure.
Detailed Description
The invention will be described in detail below with reference to the drawings and the specific embodiments.
Referring to fig. 1 and 2, the present invention provides a method for transmitting VR application data based on webtc, which includes the following steps.
S1: expanding the RTP header; the extended content includes: URI is added to rtp_parameters.cc; adding an extension type in rtp_rtcp_defined. Defining extension data and a data structure in rtp_header_extensions.h; the data is packed and unpacked in rtp_header_extensions. The URI is as follows: rayvision: rayxr: vr-control-info. The extension type is as follows: RTPExtensionType kRtpExtensionVRConrtolInfo. The data structure comprises: an ID; len: data length (len); pomeid: posture index, VR customer end produces; framesettime: frame transmission time, VR server side generation.
S2: the VR client connects to the signaling server through TCP.
S3: the VR server is connected to the signaling server through TCP.
S4: the VR client calls the CreateOffer of webrtc to create an offer SDP; SDP, session Description Protocol, is a control protocol used to describe multimedia sessions.
S5: sendOffer1: the VR client communicates the offer SDP to the signaling server over TCP.
S6: sendOffer2: the signaling server forwards the offer SDP to the VR server.
S7: createAnswer: the VR server receives the offer and invokes the createensiwer method of webrtc to create an answer SDP.
S8: sendAnswer1: the VR server communicates the answer SDP to the signaling server over TCP.
S9: sendAnswer2: the signaling server forwards the answer SDP to the VR client.
S10: P2P media communication establishment procedure.
S11: and the VR client sends the gesture index and gesture data of the hardware provided with the VR client to the VR server. The hardware is a head-mounted VR display device. The gesture data includes: position information and direction information of hardware.
S12: the VR server side sends VR video data to the VR client side, wherein the RTP extension header contains newly defined extension data: attitude index and frame transmission time. And the VR server receives the gesture data of the hardware, renders the image according to the gesture data, and returns the image to the VR client together with the gesture data used for rendering. The VR server renders the VR image according to the gesture data of the hardware, and returns the rendered VR image to the VR client together with the gesture data, so that the hardware and the VR client do not need to do excessive calculation processing, and play the VR image directly according to the gesture data, thereby effectively delaying time and improving smoothness; meanwhile, the requirements on hardware can be effectively reduced, and the cost is saved.
The scheme is based on the webrtc transmission, can fully utilize the existing transmission technology of the webrtc, such as packet loss retransmission, FEC, transmission bandwidth evaluation and the like, and is superior to the direct use of UDP or TCP in the aspect of real-time transmission; meanwhile, by expanding the RTP header, information in the aspect of VR control is added into the RTP expansion header and is transmitted together with media data in a media channel, so that the problems of blocking and time delay caused by inconsistent transmission strategies of the media channel and the data channel are solved, and user experience is improved.
In summary, the present invention provides a method for transmitting VR application data based on webtc, by expanding the RTP header, transmitting the required VR control information in the media packet, the control information and the media data are achieved simultaneously, and the processing logic of the client is reduced; the control information of the data channel does not need to be waited, and the delay is reduced; the IP header and the UDP header added by the transmission layer are wasted without independently sending the control information block, so that the bandwidth is reduced; the smoothness of the image is obviously improved in the project use process.
The foregoing description of the preferred embodiment of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.

Claims (6)

1. A method for transmitting VR application data based on webrtc, comprising the steps of:
s1: expanding the RTP header; the extended content includes: URI is added to rtp_parameters.cc; adding an extension type in rtp_rtcp_defined. Defining extension data and a data structure in rtp_header_extensions.h; packaging and unpacking data in rtp_header_extensions.cc;
s2: the VR client is connected to the signaling server through TCP;
s3: the VR server is connected to the signaling server through TCP;
s4: the VR client calls the CreateOffer of webrtc to create an offer SDP; SDP, session Description Protocol, is a control protocol used to describe multimedia sessions;
s5: sendOffer1: the VR client transmits the offer SDP to a signaling server through TCP;
s6: sendOffer2: the signaling server forwards the offer SDP to the VR server;
s7: createAnswer: the VR server receives the offer and calls the createensiwer method of webrtc to create an answer SDP;
s8: sendAnswer1: the VR server transmits the answer SDP to the signaling server through TCP;
s9: sendAnswer2: the signaling server forwards the answer SDP to the VR client;
s10: a P2P media communication establishment procedure;
s11: the VR client sends the gesture index and gesture data of the hardware provided with the VR client to the VR server;
s12: the VR server side sends VR video data to the VR client side, wherein the RTP extension header contains newly defined extension data: attitude index and frame transmission time;
the data structure comprises: an ID; len: a data length; pomeid: posture index, VR customer end produces; framesettime: frame transmission time, VR server side generation.
2. The method of claim 1, wherein the gesture data comprises: position information and direction information of hardware.
3. The method of claim 1, wherein the URI is: rayvision: rayxr: vr-control-info.
4. The method for transmitting VR application data based on webtc of claim 1, wherein the extension type is: RTPExtensionType kRtpExtensionVRConrtolInfo.
5. The method of claim 1, wherein the hardware is a head-mounted VR display device.
6. The method of claim 1, wherein the VR server receives pose data of the hardware and renders the image according to the pose data, and then returns the rendered pose data to the VR client.
CN202210530055.8A 2022-05-16 2022-05-16 Method for transmitting VR application data based on webrtc Active CN114900508B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210530055.8A CN114900508B (en) 2022-05-16 2022-05-16 Method for transmitting VR application data based on webrtc

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210530055.8A CN114900508B (en) 2022-05-16 2022-05-16 Method for transmitting VR application data based on webrtc

Publications (2)

Publication Number Publication Date
CN114900508A CN114900508A (en) 2022-08-12
CN114900508B true CN114900508B (en) 2023-08-29

Family

ID=82724802

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210530055.8A Active CN114900508B (en) 2022-05-16 2022-05-16 Method for transmitting VR application data based on webrtc

Country Status (1)

Country Link
CN (1) CN114900508B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024075092A1 (en) * 2022-10-31 2024-04-11 Lenovo (Singapore) Pte. Ltd. User interaction data transportation using real-time transport protocol header extension

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101008753B1 (en) * 2010-01-14 2011-01-14 주식회사 씨앤드디큐브 Multimedia data streaming system
GB201709619D0 (en) * 2017-06-16 2017-08-02 Canon Kk Methods, devices, and computer programs for improving streaming of portions of media data
WO2018041780A1 (en) * 2016-08-29 2018-03-08 Koninklijke Kpn N.V. Communicating in a virtual reality environment
WO2018126702A1 (en) * 2017-01-06 2018-07-12 华为技术有限公司 Streaming media transmission method applied to virtual reality technology and client
WO2019157803A1 (en) * 2018-02-13 2019-08-22 华为技术有限公司 Transmission control method
CN111479121A (en) * 2020-04-08 2020-07-31 北京智能工场科技有限公司 Live broadcasting method and system based on streaming media server
CN113315823A (en) * 2021-05-21 2021-08-27 广州赞赏信息科技有限公司 Low-delay audio and video transmission method
CN114036044A (en) * 2021-10-28 2022-02-11 深圳市瑞云科技有限公司 Openvr development drive-based debugging method

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102540459B1 (en) * 2016-12-22 2023-06-05 한화비전 주식회사 Method and Device for media streaming between server and client using RTP/RTSP standard protocol

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101008753B1 (en) * 2010-01-14 2011-01-14 주식회사 씨앤드디큐브 Multimedia data streaming system
WO2018041780A1 (en) * 2016-08-29 2018-03-08 Koninklijke Kpn N.V. Communicating in a virtual reality environment
WO2018126702A1 (en) * 2017-01-06 2018-07-12 华为技术有限公司 Streaming media transmission method applied to virtual reality technology and client
GB201709619D0 (en) * 2017-06-16 2017-08-02 Canon Kk Methods, devices, and computer programs for improving streaming of portions of media data
WO2019157803A1 (en) * 2018-02-13 2019-08-22 华为技术有限公司 Transmission control method
CN111479121A (en) * 2020-04-08 2020-07-31 北京智能工场科技有限公司 Live broadcasting method and system based on streaming media server
CN113315823A (en) * 2021-05-21 2021-08-27 广州赞赏信息科技有限公司 Low-delay audio and video transmission method
CN114036044A (en) * 2021-10-28 2022-02-11 深圳市瑞云科技有限公司 Openvr development drive-based debugging method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
面向Web AR的微服务化可定制业务系统的设计与实现;耿超;《中国优秀硕士学位论文全文数据库 信息科技辑》;I138-1242页 *

Also Published As

Publication number Publication date
CN114900508A (en) 2022-08-12

Similar Documents

Publication Publication Date Title
US7447775B1 (en) Methods and apparatus for supporting transmission of streaming data
JP5545295B2 (en) How to establish a thin client session
US9143547B2 (en) Streaming video over a wireless network
US20060168291A1 (en) Interactive multichannel data distribution system
EP1395014B1 (en) A method of transmitting data streams with data segments of variable length
US8521901B2 (en) TCP burst avoidance
CN101656747A (en) Method and system for transmitting streaming media data
CN114900508B (en) Method for transmitting VR application data based on webrtc
WO2011088711A1 (en) Method and system for updating web page data
WO2013116975A1 (en) Stream media playing method, device and system
CN110505532A (en) Low delay image transfer method, device, equipment and storage medium
CN107872471A (en) Remote desktop image command processing method and system
US7770211B2 (en) Unauthorized access prevention method, unauthorized access prevention apparatus and unauthorized access prevention program
KR20070024183A (en) Method for controlling data transmission and network apparatus transmitting data by using the same
JP2011512073A (en) Efficient interworking between circuit-switched and packet-switched multimedia services that specifies a maximum packet size attribute
US20070115945A1 (en) Communication protocol and method for operating a volp communication with additional data transmission
CN109361671A (en) A kind of streaming media framework method based on Session Initiation Protocol
CN114710568B (en) Audio and video data communication method, device and storage medium
KR20120015037A (en) System and method for compensating consecutive palyback delay of video playback service based on real-time streaming protocol
EP2034664A1 (en) Method and device of congrolling media resource, method and system of establishing calling
WO2011000291A1 (en) Method, device and system for associating real-time transport protocol (rtp) packets in session initiation protocol (sip) session
CN114422425B (en) Method, system, device and storage medium for transmitting multilink aggregated data
WO2006070341A3 (en) Universal port user agent capable of caching route information among sessions and associated method
US20070153828A1 (en) System and method to negotiate the addition or deletion of a PPP link without data loss
JP4359019B2 (en) How to send a video stream

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP01 Change in the name or title of a patent holder
CP01 Change in the name or title of a patent holder

Address after: 518000 17th floor, block B, Sunshine Technology Innovation Center, No.2 Shanghua Road, Nanshan street, Nanshan District, Shenzhen City, Guangdong Province

Patentee after: Shenzhen Ruiyun Technology Co.,Ltd.

Address before: 518000 17th floor, block B, Sunshine Technology Innovation Center, No.2 Shanghua Road, Nanshan street, Nanshan District, Shenzhen City, Guangdong Province

Patentee before: SHENZHEN RAYVISION TECHNOLOGY CO.,LTD.