CN116320439A

CN116320439A - Cloud game video stream weak network transmission optimization method and system based on RS (Reed-Solomon) coding

Info

Publication number: CN116320439A
Application number: CN202310278057.7A
Authority: CN
Inventors: 李兴平; 陈旺科; 潘光炜
Original assignee: 4399 NETWORK CO LTD
Current assignee: 4399 NETWORK CO LTD
Priority date: 2023-03-21
Filing date: 2023-03-21
Publication date: 2023-06-23

Abstract

The invention provides a cloud game video stream weak network transmission optimization method and a system based on RS coding, wherein a sending end analyzes and processes a received network state parameter of a receiving end to obtain a sending end control parameter, and the method comprises the following steps: code rate optimum, redundancy optimum, and transmission rate optimum; the transmitting end processes the original media data packet to be transmitted according to the control parameters of the transmitting end, and transmits the processed data packet to the receiving end through a network. Has the following advantages: 1) The invention optimizes the phenomena of video stream decoding failure caused by packet loss in the traditional UDP transmission process, such as video jamming, screen display and the like of the receiving end, and can enable cloud game players to obtain better game experience in a complex network environment. 2) Compared with the traditional error correction algorithm, the method has wider weak network adaptability, and can process the decoding recovery under the conditions of burst packet loss and continuous packet loss.

Description

Cloud game video stream weak network transmission optimization method and system based on RS (Reed-Solomon) coding

Technical Field

The invention belongs to the technical field of cloud games, and particularly relates to a method and a system for optimizing video stream weak network transmission of a cloud game based on RS (reed-solomon) coding.

Background

With the continuous upgrade of technology, people are more and more familiar with the word "cloud computing", and companies and individuals are focusing on various fields of cloud computing, such as cloud storage, cloud identity, cloud internet of things, cloud security and the like. The advent of the "cloud" changed people's knowledge of the internet. Without surprise, "cloud gaming" has also emerged, in the era of games becoming more popular, where the gaming experience is limited by low-profile devices, game developers have turned attention to "cloud", combining cloud computing with games, and "cloud gaming" has resulted therefrom. "cloud games" may also be referred to as demand games, which are a new field of cloud computing based. A desired game is literally understood to be available whenever desired, which is a service that may be used to improve game performance. In the cloud game platform, all game logics and rendering are operated at a server side, and then the game pictures after being rendered are compressed and then transmitted to a user through a network, so that game equipment of a player does not need any high-end processor and display card, and the only requirement is basic video decoding software and a reliable network. It will be appreciated that cloud gaming essentially breaks away from reliance on device hardware. From the perspective of the server, no new host needs to be developed, and only the performance of the server needs to be improved. From the perspective of users, the user can obtain higher image quality feeling without purchasing a high-performance computer, and even a mobile phone can experience a computer game or a host game, and the characteristic is also successful in attracting the favor of more players.

Therefore, the transmission quality of the video picture becomes the most important index of whether the cloud game can attract users, the network environment of the users, especially the access method of the mobile wireless network, has great network fluctuation, various network environments cause the conditions of high delay, high jitter and high packet loss at the receiving end, and once the situation of weak network occurs, the game experience of the users is influenced greatly, so that a video transmission method is required to be designed, and the implementation of remedial measures is required to be carried out in the specific weak network environment.

Disclosure of Invention

Aiming at the defects existing in the prior art, the invention provides a cloud game video stream weak network transmission optimization method and system based on RS coding, which can effectively solve the problems.

The technical scheme adopted by the invention is as follows:

the invention provides a cloud game video stream weak network transmission optimization method based on RS coding, which comprises the following steps:

step 1, a transmitting end receives receiving end network state parameters fed back by a receiving end in real time, including: the method comprises the steps of receiving a real-time code rate value of a receiving end, a real-time maximum receiving code rate estimated value of the receiving end and a real-time packet loss rate of the receiving end;

the sending end analyzes and processes the received network state parameters of the receiving end to obtain sending end control parameters, and the method comprises the following steps: code rate optimum, redundancy optimum, and transmission rate optimum;

step 2, the transmitting end processes the original media data packet to be transmitted according to the control parameter of the transmitting end, and transmits the processed data packet to the receiving end through the network, specifically comprising:

step 2.1, a sending end acquires an original media data packet to be transmitted;

step 2.2, the transmitting end encodes the original media data packet according to the optimal value of the encoding code rate to obtain an encoded media data packet;

step 2.3, the transmitting end generates a set number of redundant data packets according to the optimal redundancy value, and encodes the redundant data packets to obtain encoded redundant data packets;

step 2.4, the sending end transmits the coded media data packet and the coded redundant data packet to the receiving end through a network by adopting a UDP protocol according to the optimal sending rate value;

step 3, the receiving end receives the coded media data packet and the coded redundant data packet sent by the sending end, and detects whether the packet loss phenomenon occurs, if yes, the lost data packet is recovered; then, decoding the coded media data packet and the coded redundant data packet to decode the original media data packet;

the receiving end detects the network state parameter of the receiving end in real time and feeds the network state parameter back to the sending end.

Preferably, in step 2.4, the encoded media data packet and the encoded redundant data packet are RTP data packets encoded by RTP.

The invention also provides a cloud game video stream weak network transmission optimization system based on RS coding, which comprises a transmitting end and a receiving end;

the transmitting end comprises: the system comprises a data acquisition module, a coding module, a redundant data packet generation coding module (RSFEC), a packaging module, a smooth sending module, a UDP sending end transmission module, a sending end congestion control module, a code rate controller and a receiving end feedback receiving module;

the receiving end comprises: the device comprises a UDP receiving module, an identification module, an RSFEC decoding module, an anti-shake buffer module, a decoding module, a data consumption module and a receiving end congestion control module;

the receiving end feedback receiving module is used for receiving the RTCP RTPFB feedback message, the RTCP REMB feedback message, the RTCP RR feedback message and the RTCP NACK feedback message transmitted by the receiving end congestion control module;

the sending end congestion control module is used for analyzing the RTCP RTPFB feedback message to obtain a receiving end real-time code rate value, and transmitting the receiving end real-time code rate value to the code rate controller;

the code rate controller is used for analyzing the RTCP REMB feedback message to obtain a real-time maximum receiving code rate estimated value of the receiving end; analyzing the RTCP RR feedback message to obtain the real-time packet loss rate of the receiving end; then, analyzing and processing the receiving end real-time code rate value, the receiving end real-time maximum receiving code rate estimated value and the receiving end real-time packet loss rate to obtain a transmitting end control parameter, wherein the method comprises the following steps: the method comprises the steps of transmitting an optimal coding rate value, an optimal redundancy value and an optimal sending rate value to a coding module, transmitting the optimal redundancy value to a redundant data packet generation coding module (RSFEC), and transmitting the optimal sending rate value to a smooth sending module;

the data acquisition module is used for acquiring an original media data packet to be transmitted;

the coding module is used for coding the original media data packet according to the optimal value of the coding code rate to obtain a coded media data packet;

the redundant data packet generation coding module RSFEC is used for generating a set number of redundant data packets according to the optimal redundancy value, and coding the redundant data packets to obtain coded redundant data packets;

the packaging module is used for packaging the coded media data packet and the coded redundant data packet;

the smooth sending module is configured to perform smooth sending processing on the packed coded media data packet and the coded redundant data packet, where the smooth sending processing is: controlling the sending rate to be the optimal value of the sending rate;

the UDP transmitting end transmission module is used for transmitting the optimal value of the transmitting rate to the receiving end through a network by adopting a UDP protocol;

the UDP receiving module is used for monitoring the relevant port and receiving the coded media data packet and the coded redundant data packet which are sent by the sending end;

the identification module is configured to identify a type of the data packet received by the UDP receiving module, namely: identifying whether a received data packet is the encoded media data packet or the encoded redundant data packet; when packet loss is detected, transmitting the coded redundant data packet and the packet sequence number to be recovered to the RSFEC decoding module; meanwhile, the coded media data packets are ordered and enter the anti-shake buffer module;

the RSFEC decoding module is used for recovering lost data packets according to the coded redundant data packets and the packet sequence numbers to be recovered transmitted by the identification module, and sequencing the recovered data packets into the anti-shake buffer module;

the decoding module is used for regularly reading the coded media data packet in the anti-shake buffer module and decoding the original media data packet;

the data consumption module is used for using the original media data packet decoded by the decoding module;

the receiving end congestion control module is used for detecting the network state parameters of the receiving end in real time and feeding back the network state parameters to the sending end.

The cloud game video stream weak network transmission optimization method and system based on RS coding provided by the invention have the following advantages:

1) The invention optimizes the phenomena of video stream decoding failure caused by packet loss in the traditional UDP transmission process, such as video jamming, screen display and the like of the receiving end, and can enable cloud game players to obtain better game experience in a complex network environment.

2) Compared with the traditional error correction algorithm, the method has wider weak network adaptability, and can process the decoding recovery under the conditions of burst packet loss and continuous packet loss.

Drawings

FIG. 1 is a block diagram of a system for optimizing the transmission of a cloud game video stream weak network based on RS coding;

FIG. 2 is a schematic diagram of a sender redundant data packet generation encoding module provided by the present invention;

fig. 3 is a schematic diagram of a receiving-end RSFEC decoding module according to the present invention.

Detailed Description

In order to make the technical problems, technical schemes and beneficial effects solved by the invention more clear, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

The invention provides a cloud game video stream weak network transmission optimization method and system based on RS (Reed-Solomon) coding, which are characterized in that error correction code coding is carried out to generate redundant video packets before video transmission, and a receiving end reversely decodes a lost video data packet to obtain a complete video according to a corresponding algorithm, so that the remedy after the video packet loss is realized in a weak network high packet loss environment; in addition, according to the network state parameters of the receiving end, the coding rate, redundancy and sending rate of the sending end are subjected to real-time feedback regulation control, the weak network transmission effect of the cloud game video is optimized, and the user experience effect is improved in the weak network environment in the cloud game video streaming transmission process.

Referring to fig. 1, the method for optimizing the weak network transmission of the cloud game video stream based on the RS coding provided by the invention comprises the following steps:

in this step, the encoded media data packet and the encoded redundancy data packet are RTP data packets encoded by RTP.

Referring to fig. 1, the invention also provides a cloud game video stream weak network transmission optimization system based on RS coding, which comprises a transmitting end and a receiving end;

as a specific implementation manner, the encoded media data packet and the encoded redundant data packet are both FEC data packets, and the packing module is configured to pack the FEC data packets into an RTP format for transmission.

The RTP format of the FEC data packet is as follows:

in the RTP header of the FEC packet, the padding bit, extension bit, CC, and marker bit are obtained by calculation, and the value of the SSRC is generally the same as the SSRC value of the protected media packet. CSRC list is absent, extension is absent, and both values are not affected by CC and X. The sequence number is monotonically increased, and the timestamp is consistent with the timestamp of the media data packet sent at the time, and is monotonically increased. The PT value of the FEC packet is dynamically determined and is finished through out-of-band negotiation. Unrecognized PT value packets need to be discarded, and flexible processing mechanisms can be realized.

The 12 bytes of the FEC header head, as indicated above, contain the fields SN base, length recovery, E, PT recovery, mask and TS recovery.

The field length recovery is used to recover any associated data packet, and is obtained by performing bit exclusive-or calculation on a plurality of data packet lengths, and is an unsigned 16-bit integer of a network byte order. The packet length contains the media payload, CSRC list, extension and padding. If each packet is of unequal length, an exclusive or operation may still be performed. If the two media packet lengths are 3 (0 b 011) and 5 (0 b 101), respectively, then the length recovery calculation method: 0b011 xor 0b 101=0b 110.

The field mask length 24 bits, if bit i is set to 1, indicates that the media packet with sequence number N + i is associated with this FEC packet, where N is SN base, indicating the smallest RTP sequence number in this group of media packets. Therefore, one FEC packet can be calculated by a group of 24 packets at most, i=0 at the lowest and i=23 at the highest.

The invention can also comprise a data coding module, a data transmitting module, a data receiving module and a data decoding module when aiming at video data transmission.

And a data coding module: and performing error correction coding on the compressed video data coded by the video acquisition equipment, coding according to algorithm parameters and the number of acquired video data packets, generating a certain number of redundant video data packets, wherein the generation parameters of the redundant video data packets can be dynamically adjusted according to the real-time network state of the receiving end, and the redundancy of the RS algorithm is adjusted in real time according to the network packet loss and code rate conditions of the receiving end so as to achieve the optimal network bandwidth consumption and packet loss recovery effect.

And a data transmitting module: the UDP protocol is adopted for video network transmission, and in view of the self characteristics of UDP, the data transmission module needs to arrange a specific sequence number of a transmitted data packet, and the data receiving module is provided for judging the data integrity and controlling a certain transmission rate so as to avoid transmission congestion caused by overlarge fluctuation of network bandwidth.

And a data receiving module: the method is used for receiving video data sent to the local end by the sending end through a UDP network protocol, verifying the data integrity, and further feeding back network parameters such as packet loss data, the optimal code rate of the receiving party and the like to the sending end so as to optimize the RS algorithm parameters of the sending party.

And a data decoding module: and the system is responsible for carrying out missing decoding on the video data collected by the data receiving module, and adopts an algorithm of the system to decode and recover the lost partial packets in the transmission process according to error correction codes and the non-lost related packets so as to obtain complete video data packets.

As a specific embodiment, the data is sent and received by using RTP to carry data for network transmission, and the standard RTP header structure is as follows:

the meaning of each field is as follows:

1) V: the version number of the RTP protocol occupies 2 bits, and the current protocol version number is 2.

2) P: padding flags, taking up 1 bit, and if p=1, padding one or more additional octets at the end of the message, which are not part of the payload.

3) X: an extension flag, taking 1 bit, if x=1, then there is an extension header followed by an RTP header.

4) CC: the CSRC counter occupies 4 bits and indicates the number of CSRC identifiers.

5) M, marking, namely marking the end of a frame for video, wherein the mark occupies 1 bit, different payloads have different meanings; for audio, the start of the session is marked.

6) PT is the type of payload, takes 7 bits, is used for explaining the type of payload in RTP message, such as GSM audio, JPEM image, etc., and is mostly used for distinguishing audio stream and video stream in streaming media, thus facilitating the analysis of clients.

7) Sequence number: the number of the RTP message occupies 16 bits, and is used for identifying the sequence number of the RTP message sent by the sending end, and the sequence number is increased by 1 when one message is sent. This field can be used to check for packet loss when the underlying bearer protocol uses UDP, when the network conditions are not good. The simultaneous occurrence of network jitter can be used to reorder the data, with the initial value of the sequence number being random, while the sequence of the audio and video packets are counted separately.

8) Timestamp (Timestamp): occupying 32 bits, a clock frequency of 90kHz must be used. The timestamp reflects the sampling instant of the first octet of the RTP packet. The receiving end uses the time stamp to calculate the delay and delay jitter, and performs synchronous control.

9) Synchronization Source (SSRC) identifier: and occupies 32 bits and is used for identifying the synchronous information source. The identifier is randomly selected and two synchronous sources participating in the same video conference cannot have the same SSRC.

10 A Contracted Source (CSRC) identifier: each CSRC identifier occupies 32 bits and may have 0 to 15. Each CSRC identifies all of the offer sources contained in the RTP message payload.

Wherein the x flag bit in the first byte indicates whether the RTP header is extended, the RTP protocol allows for user-defined extension, with the extended field next to the RTP fixed header described above. The scheme carries the following information in the RTP extension header:

(1) The Group number where the current packet is located. The code stream is composed of successive groups, each Group having its own unique sequence number. (2) The Group size (3) where the current packet is located is the position of the current packet within the Group.

The 7bit PT field in the RTP header indicates the type of payload, and its reference value is listed in RFC3551 for standard types such as audio AAC, video H264. In the invention, the RTP load comprises redundant packets besides audio and video data, and a custom FEC redundant packet type is specified for the purpose, thereby facilitating the distinguishing processing of a receiving end. The transmitting end performs FEC encoding on a group of application layer data to be transmitted and RTP packages and transmits the application layer data, and the flow is shown in fig. 2, and includes:

in fig. 2, P1 to P5 represent media packets, and r1 to r3 represent redundant packets. In this example, one Group consists of 5 media packets and 3 redundancy packets. Specifically, when the media data packet P1 is transmitted, the RTP packet is sent out after being encapsulated, and is stored in the internal data queue. When the last media data packet P5 of the Group is sent, FEC encoding is carried out on the P1-P5 stored in the data queue, redundant data packets r 1-r 3 are generated, and RTP packaging and sending are carried out. This is looped until all incoming packets are processed.

The groups are independent, and the number of the packets can be different, for example, the previous Group is a (5, 3) combination, and the next Group can be a (5, 4) combination, that is, the invention provides a dynamic redundancy mode, which supports the sender to dynamically adjust the redundancy according to the network receiving condition of the receiver so as to achieve the best service quality. For example, the channel condition is poor, when the packet loss rate of the receiving party is increased, the sending end can increase the redundancy so as to enhance the packet loss resistance; otherwise, if the packet loss rate of the receiving party is very low, the sending end can properly reduce the redundancy so as to save the network bandwidth.

The receiving end performs FEC decoding to recover the packet loss data, and the processing flow is shown in fig. 3, and includes:

when the receiving end receives the P1 and P4 packets of the Group, the P1 and P4 packets can be directly output to the application layer because the P1 and P4 packets are not lost, and meanwhile, the P1 and P4 packets are cached in the local queue for the subsequent possible FEC decoding, and if no packet is lost in the Group, the cached data is emptied. When the P5 packet is received, whether the packet loss condition occurs can be judged according to the packet sequence number in the Group, if the P4 packet is judged to be lost, the output of the Group data packet is stopped at the moment, and the P5 packet is stored in the local queue. When P5 is received until the redundant packet rr is received, the condition of packet loss recovery is satisfied: the received media packet number+the received redundant packet number > =the Group original media packet number, and therefore, the FEC decoding process is performed on P1, P2, P3, P5, and r1, and P4 packets are recovered. Since P1 to P3 have been previously outputted, only P4 to P5 are outputted here. When the redundant packet r3 continues to be received, it can be discarded directly. If the number of received media packets plus the number of redundant packets is smaller than the number of the original media packets of the Group, packet loss of the Group cannot be recovered, and the system directly outputs the received media packets in sequence.

As a specific embodiment, in conjunction with fig. 1, a sender-to-receiver interaction flow of the present invention is shown, where the overall flow is:

after acquisition, coding and redundancy coding, a transmitting end transmits RTP (Real-time Transport Protocol) data packets and RSFEC redundancy error correction packets to a receiving end, a data receiving end receives data, performs certain interactions such as buffering and packet loss feedback, and then decodes and recovers the RSFEC packets on the data, and decodes the data for use after recovering the original data;

the encoding process of the RSFEC redundancy algorithm needs two parameters N (the number of original data packets) and M (the total number of encoded data packets), M-N RSFEC redundancy error correction packets are generated according to the N original data packets in the RS algorithm, and the redundancy P is M-N and is calculated by combining a video data code rate and a current packet loss rate according to a certain relation.

In the accompanying drawings, 1-N is a transmitting end flow, 2-N is a receiving end flow, and the method specifically comprises the following steps:

and the transmitting end:

1-1: the system flow starts to work, and the acquisition module acquires an original media data packet to be used;

1-2: setting parameters of a coding module, initializing the coding module and starting the coding module;

1-3: initializing and setting RSFEC algorithm parameters, and starting a redundant data packet generation coding module RSFEC;

1-4: the data generated by the coding module and the redundant data packet generation coding module RSFEC are packed into RTP data packets, and the packing examples of the original data and the redundant data are as follows:

R：0：repair packet、1：retransmision。

F：0：flexible mask、1：offset M and N。

p, X, CC, M, PT (Recovery field): and calculating the RTP header of the main stream by using a protection algorithm.

Length recovery: the payload length of the recovered packet is verified, and the TS recovery is obtained by calculating the length of the protected rtp packet by using a protection algorithm: and calculating TS in RTP header of the main stream by using a protection algorithm.

SSRCCount: number of SSRCs protected by fec, 0: and (5) illegal operation.

Reserved, set 0.

Ssrc_i, SN base_i describes SSRC, SN base, respectively, of the packet protected by the fec.

The protected packet information is described using a fixed length if f=1:

M>0and N＝0Row Fec

M>0and N＝1Row Fec+colum FEC follow

M>0and N>1Column FEC

if r=1, f=1 then this means using retransmission

1-5: the RTP packet firstly enters a transmission buffer area to wait for transmission time, and different ssrc are used for identifying the original data and the redundant data;

the data transmission transmitting end is provided with a smooth transmitting module (packed Sender), because the key frames of the video are much larger than the non-key frames, generally one key frame needs to be packed into a plurality of RTP messages, and at this time, if all RTP messages are directly transmitted to the network, network congestion is easily caused. The Paced Sender module has the function of enabling data to be sent on a network more smoothly, and preventing network congestion caused by sudden increase of data volume;

1-6: sequentially sending the data packet and the redundant packet to related network addresses;

1-7: after receiving RTCP data fed back by a receiving end, a sending end calculates the current network packet loss rate; the sending end analyzes the RR message as follows:

synchronization Source (SSRC) identifier: randomly generated character strings mark a path of data sources.

fraction last: and calculating the packet loss rate when the report block is sent.

cumulative number of packets lost: message loss cumulative total.

extended highest sequence number received: the first 16 bits of the received packet sequence number indicate what number of turns, and the last 16 bits indicate the current sequence number.

interarrival jitter: average spacing between packets.

Last SR: time stamp of Last SR packet.

Delay since last SR time interval from last LSR.

The Loss fraction calculation formula is as follows:

fraction lost = this time 255/100 packet loss

Simultaneous loss = accumulated packet loss number

1-8: the code rate control module calculates the optimal code rate and redundancy according to the packet loss rate of the congestion control module, and feeds back the optimal code rate and redundancy to the coding module and the redundancy data packet to generate the coding module RSFEC and update the parameters of the corresponding modules.

The receiving end:

2-1: initializing a receiving end module, and monitoring that a relevant port is ready to receive a transmitted data packet;

2-2: the RTP data packet in the network stream is read, whether the RTP data packet is a redundant packet or an original data packet is judged according to ssrc and pt, and when the packet loss is detected, the redundant packet and the packet sequence number to be recovered are transmitted to an RSFEC decoding module;

2-3: the RSFEC decoding module restores the lost data packet according to the data packet received by the receiving module;

2-4: the RSFEC recovers packet loss and then sequences the packet loss into a buffer module;

2-5: the decoding module acquires complete data at fixed time and decodes the complete data into original data;

2-6: completing data transmission to obtain available complete data packets;

2-7: the receiving end determines a new maximum receiving code rate value according to Remote Rate Controller, and then feeds back the new maximum receiving code rate value to the sending end through an RTCP REMB message, and reports a packet loss rate to the sending end through an RR message, so that the sending end regulates and controls the code rate and the redundancy.

Compared with the prior art, the invention has the beneficial effects that:

The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which is also intended to be covered by the present invention.

Claims

1. A cloud game video stream weak network transmission optimization method based on RS coding is characterized by comprising the following steps:

2. The RS-coding-based cloud video stream weak network transmission optimization method according to claim 1, wherein in step 2.4, the coded media data packet and the coded redundancy data packet are RTP data packets coded by RTP.

3. A cloud game video stream weak network transmission optimization system based on RS coding is characterized by comprising a sending end and a receiving end;