WO2022194140A1

WO2022194140A1 - Remote video transmitting method and transmitting apparatus, storage medium, and electronic device

Info

Publication number: WO2022194140A1
Application number: PCT/CN2022/080907
Authority: WO
Inventors: 万安庆
Original assignee: 上海达龙信息科技有限公司
Priority date: 2021-03-18
Filing date: 2022-03-15
Publication date: 2022-09-22
Also published as: CN113747194B; CN113747194A

Abstract

The present invention provides a remote video transmitting method and transmitting apparatus, a storage medium, and an electronic device. The remote video transmitting method is executed by a server, and comprises steps of: sending video frame data to a client; receiving a packet loss rate and bandwidth occupancy sent by the client every a preset time; predicting a current bandwidth of the server according to the packet loss rate and the bandwidth occupancy, and adjusting an encoding rate of the server on this basis; and sending the video frame data according to the adjusted encoding rate. According to the technical solution of the present invention, a server can reasonably adjust and control encoded transmission and playback of video streams in real time according to constantly changing network conditions, so that better video definition and video fluency can be realized when a client performs remote access.

Description

Remote video transmission method, transmission device, storage medium and electronic equipment

technical field

The present invention relates to the technical field of remote video transmission, and in particular, to a remote video transmission method, a transmission device, a storage medium and an electronic device.

Background technique

With the continuous advancement of technology and technology, remote technology has developed rapidly and has a wide range of applications. For example, QQ remote technology, teamviewer and the remote assistance technology that comes with Windows are currently the most used. Users can use such remote management tools to Realize remote support, remote access, remote management, online office and conference presentations, and remote on-hook upgrades.

Existing remote technologies provide users with real-time remote server and client information. However, the reality is that the video stream transmitted by such remote technologies has various problems after being played on the client. The video stream is unstable and the delay is high. The main influencing factors are that the server's ability to capture and encode, the network transmission ability, and the client's decoding and rendering ability cannot reach a coordinated balance value, and the client's access process is not smooth, resulting in poor visual effects.

Nowadays, various remote desktop connection applications have become an important part of people's information life. At present, various remote connection tools emerge in an endless stream, which provide people with a comprehensive experience integrating audio, video, graphics and text. People expect a faster, smoother, and higher-quality remote video service experience, and how to achieve the above user expectations has become a major difficulty in today's technology.

SUMMARY OF THE INVENTION

In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a remote video transmission method, a transmission device, a storage medium and an electronic device for solving the above problems in the prior art.

In order to achieve the above purpose and other related purposes, the present invention provides a remote video transmission method, which is applied to a server; the method includes: sending video frame data to a client; receiving data sent by the client every preset time; Packet loss rate and bandwidth occupancy; predict its own current bandwidth according to the packet loss rate and the bandwidth occupancy, and adjust its own encoding bit rate accordingly; send video frame data according to the adjusted encoding bit rate.

In an embodiment of the present invention, the step of sending video frame data to a client includes: sending video frame data to the client according to an initial capture code sending frequency; wherein the step of obtaining the initial capture code sending frequency includes: : obtain the retransmission timeout sent by the client; calculate the video buffer size according to the retransmission timeout, and set the video buffer accordingly; determine the initial Capture encoding send frequency.

In an embodiment of the present invention, the step of sending video frame data to the client further includes: when the number of video buffer frames in the video buffer area exceeds the capacity of the video buffer area, reducing the frequency of capturing and encoding sending, Send video frame data to the client at a reduced capture and encoding sending frequency; or, stop capturing video frame data.

In an embodiment of the present invention, the step of sending video frame data to the client includes: when each frame of video data is sent, storing the frame of video data in the video buffer, and waiting for the client Receive the reply sent by the client; wherein, if the reply is successfully received, clear the one frame of video data in the video buffer; if the reply is not received, then The one frame of video data is resent to the client.

In an embodiment of the present invention, the step of adjusting the self-encoding bit rate according to the predicted current bandwidth of the self includes: judging a preset bit rate level corresponding to the current bandwidth; searching for a preset bit rate level corresponding to the preset bit rate level. An adjustment strategy is set, and the self-encoding bit rate is adjusted according to the preset adjustment strategy.

In an embodiment of the present invention, the rules for the preset adjustment policy include: increasing the bit rate as the packet loss rate decreases, and decreasing the code rate as the packet loss rate increases; wherein the preset bit rate The higher the gear, the finer the adjustment granularity of the corresponding preset adjustment strategy.

In an embodiment of the present invention, the method further includes: receiving the decoding and rendering capability sent by the client; comparing its current encoding frame rate with the decoding and rendering capability, and adjusting itself according to the minimum value of the two. encoding frame rate.

In order to achieve the above purpose and other related purposes, the present invention provides a remote video transmission device, comprising: a video frame data sending module for sending video frame data to a client; a client information receiving module for every preset time Receive the packet loss rate and bandwidth occupancy sent by the client; the coding rate adjustment module is used to predict its own current bandwidth according to the packet loss rate and the bandwidth occupancy, and adjust its own coding rate accordingly; The video frame data sending module is further configured to send the video frame data according to the encoding bit rate adjusted by the encoding bit rate adjusting module. .

In order to achieve the above object and other related objects, the present invention provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is loaded and executed by a processor, the remote video transmission method can be realized.

In order to achieve the above object and other related objects, the present invention provides an electronic device, comprising: a processor and a memory; wherein, the memory is used to store a computer program; the processor is used to load and execute the computer program, so that all The electronic device executes the remote video transmission method.

As mentioned above, the remote video transmission method, transmission device, storage medium and electronic equipment of the present invention send video frame data to the client; receive packet loss rate and bandwidth occupancy sent by the client at preset time intervals; according to The packet loss rate and the bandwidth occupancy predict the current bandwidth of the self, so as to adjust the encoding code rate of the self; send video frame data according to the adjusted encoding code rate; thus have the following beneficial effects: through the packet loss rate of the client and bandwidth occupation to predict the server bandwidth, and then adjust and control the video transmission encoding bit rate and encoding frame rate of the server, so that the video stream capture capability, server encoding capability, server network transmission capability, and client decoding and rendering capability on the server can be maximized. Balanced to achieve stable transmission and playback of video streams.

Description of drawings

FIG. 1 is a schematic diagram of an application scenario of a remote video transmission method according to an embodiment of the present invention.

FIG. 2 is a schematic flowchart of a remote video transmission method according to an embodiment of the present invention.

FIG. 3 is a schematic flowchart of a server sending video frame data to a client according to an embodiment of the present invention.

FIG. 4 is a schematic flowchart of a server adjusting an encoding frame rate and an encoding bit rate according to an embodiment of the present invention.

FIG. 5 is a schematic block diagram of a remote video transmission device according to an embodiment of the present invention.

FIG. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other under the condition of no conflict.

It should be noted that the diagrams provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic way, so the diagrams only show the components related to the present invention rather than the number, shape and For dimension drawing, the type, quantity and proportion of each component can be changed at will in actual implementation, and the component layout may also be more complicated.

As shown in FIG. 1 , the server 2 remotely provides a video stream to the mobile terminal 1, and the mobile terminal 1 presents the video content according to the received video stream. Specifically, the mobile terminal 1 is an electronic device such as a smart phone or a tablet computer, the client is a software program mounted on the mobile terminal 1 , and the server is a software program mounted on the server 2 . The user uses the client installed in the mobile terminal 1 to remotely access the server installed in the server 2, thereby obtaining and playing the video stream provided by the server.

When the server 2 is a remote desktop server, the scenario shown in FIG. 1 is a remote desktop connection scenario, and the remote desktop server is used to provide a video stream of the remote desktop to the mobile terminal 1 so that the mobile terminal can display the picture of the remote desktop. In this remote desktop connection scenario, the user has high requirements on the real-time and fluency of the video.

The server-side capture capability, server-side encoding capability, client-side decoding and rendering capability, and network transmission capability are the four major factors that affect the video frame rate. The capture capability of the server can be expressed by the average number of video frames captured by the server per second, which is generally more than 1,000 frames. The display can capture one frame every time it refreshes. If the display does not refresh, it can continue to use the previous frame to make up the frame, such as: The server can capture 1660 frames per second, while the general frame capture requirements are 60 frames/120 frames/144 frames. The server encoding capability can be expressed by the average number of video frames encoded by the server per second, which is generally fixed, but far greater than the required number of frames. For example, a server using an Nvidia GTX1060super graphics card can encode 500-800 frames per second, while the general encoding requirements 60fps/120fps/144fps. The decoding and rendering capability of the client can be represented by the number of video frames that the client decodes and renders per second, generally ranging from a few frames to several hundreds of frames. The network transmission capacity can be measured by the average number of bytes transmitted per second, mainly affected by two parameters RTT and bandwidth. The above four influencing factors are in the life cycle of the video stream from the capture of the server to the playback and display of the client. In order to ensure that the video has a high quality as much as possible, it is necessary to continuously make dynamic adjustments under subsequent network changes.

As shown in FIG. 2, in order to meet the high requirements of users for remote video, the present application provides a remote video transmission method, which is executed by the server, by predicting the bandwidth of the server and intelligently adjusting its encoding frame rate and encoding bit rate, thereby Make the above four influencing factors reach the maximum balance, and realize the stable transmission and playback of video streams. The method includes the following steps:

S21: Send video frame data to the client;

S22: Receive the packet loss rate and bandwidth occupation sent by the client every preset time;

S23: Predict its own current bandwidth according to the packet loss rate and the bandwidth occupancy, so as to adjust its own coding rate;

S24: Send video frame data according to the adjusted coding rate.

Through steps S21 to S24, the present application can adjust and control video stream encoding, transmission and playback in real time and reasonably according to changing network conditions under the reliable transmission of a single long-term connection, so as to achieve better performance when the client remotely accesses video clarity and video fluency.

The remote video transmission method of the present application will be described in detail below with reference to the embodiments.

As shown in FIG. 3 , in this embodiment, the remote desktop server provides the video stream of the remote desktop to the smart phone, and the smart phone restores and displays the remote desktop according to the received video stream.

The process of sending video frame data from the server to the client is as follows:

S31: The client and the server confirm the RTT and RTO by sending data packets to each other, and the client returns the RTO to the server.

Among them, RTT (Round-Trip Time, that is, round-trip delay) means that the data starts from the sender (the last bit is pushed to the data link), and the sender receives the confirmation from the receiver (after the receiver receives the data confirmation is sent immediately) the total elapsed time. RTO (Retransmission Timeout, that is, retransmission timeout) is used for reliable network transmission, and is the result of statistical analysis and prediction of RTT for a continuous period of time.

Specifically, the client periodically sends a heartbeat packet to the server every second, and the server immediately replies to the heartbeat packet after receiving the heartbeat packet. The client will count a period of time (for example, 5s as the statistical period) after receiving a reply. and use the corresponding algorithm (such as the moving average method) to estimate the RTO, and then send the current RTO to the server synchronously when sending the next heartbeat packet.

S32: The server determines the sending frequency of the initial acquisition code according to the RTO.

Specifically, after receiving the RTO sent by the client, the server calculates the number of buffered frames according to the current RTO and sets the video buffer, for example: buffer size S=RTO/(1000/fps)+3 (1000 is 1000 milliseconds, 3 It is error compensation, which is obtained according to actual measurement), and the fps value is the required frame rate, which is generally preset to 60. The server determines the initial capture encoding sending frequency according to the S value and its own encoding capability, so as to send video frame data to the client according to the initial capture encoding sending frequency.

S33: The server sends a video frame to the client.

Specifically, a continuous video consists of a group of continuous still pictures, and these still pictures are called video frames. When sending a frame of video data to the client, the server simultaneously stores the frame of video data in the video buffer, and waits for a reply from the client. If the client's reply is successfully received, the server will clear the one frame of video data in the video buffer; if it does not receive a successful reply from the client, the server will resend the one frame of video data to client.

S34: The client receives the video frame and replies to the server.

Specifically, each time the client receives a frame of video data, it replies an ack (acknowledgement character) to the server. If the frame packet received by the client is incomplete due to packet loss or the client's decoding capability is insufficient, so that the client's buffer area is full and cannot receive new frame data, it will not reply ack to the server, so that the server can receive this frame of video data. Retransmission.

S35: The server adjusts the transmission frequency of the capture code.

Specifically, when the number of video frames entering the video buffer is greater than S (the client's encoding capability is insufficient or the packet loss is serious, causing the server to continuously retransmit the current video frame data), the server reduces the frequency of capture and encoding transmission, so as to reduce the The capture encoding sending frequency sends video frame data to the client; or, the server stops capturing new video frame data.

Referring to Figure 4, during the transmission of the above video frame data, the client and the server also interact as follows:

S41: The client sends its own packet loss rate, bandwidth occupation, decoding and rendering capability to the server at preset time intervals.

Taking the preset time as 1s as an example, the client calculates the number of bytes of packets received (ie bandwidth) and the number of lost packets per second, calculates the average packet loss rate and occupied bandwidth per second, and calculates the decoding and rendering capabilities per second , sent to the server. Preferably, in order to avoid the clutter of information exchange between the client and the server, the client puts the packet loss rate, bandwidth occupancy, decoding and rendering capability in the heartbeat packet, and sends it to the server together with the RTO.

S42: The server adjusts its own encoding frame rate according to the received decoding and rendering capabilities, and adjusts its own encoding bit rate according to the received packet loss rate and bandwidth occupation.

Specifically, after receiving the decoding and rendering capabilities of the client, the server compares its current encoding frame rate with the decoding and rendering capabilities of the client, and adjusts its encoding frame rate according to the minimum of the two, that is: if If the current encoding frame rate of the server is smaller than the decoding and rendering capability of the client, the current encoding frame rate will remain unchanged; if the current encoding frame rate of the server is larger than the decoding and rendering capability of the client, the current encoding frame rate will be changed to the current encoding frame rate. Adjust to the value of this decoded rendering capability.

It should be noted that since the decoding and rendering capabilities of the client will not change much, the client can only send the packet loss rate and bandwidth usage to the server.

Specifically, the server predicts its own current bandwidth according to the received packet loss rate and bandwidth occupation of the client, and adjusts its own encoding bit rate accordingly.

It should be noted that a rule for adjusting the coding rate is preset in the server. The core idea of this rule is: the higher the predicted bandwidth value, the finer the adjustment granularity of the corresponding adjustment strategy. When the packet loss rate of the client decreases, the encoding rate of the client is increased (improving the client image quality), and the encoding rate is reduced (reducing the client image quality) with the increase of the client packet loss rate.

For example, it is assumed that the bandwidth ranges from 2 Mbps to 40 Mbps, and 20 Mbps is used as a grade dividing line, 2 Mbps to 20 Mbps is set as a low grade, and 20 Mbps to 40 Mbps is set as a high grade. If the bit rate of the current predicted bandwidth is at a low level, aggressive strategies should be adopted, including: if the packet loss rate of the client for two consecutive cycles is less than 2%, increase the encoding bit rate by one level, such as increasing 3Mbps; if the packet loss rate of the client is less than 2% If the rate is between 2% and 8%, keep the current encoding rate unchanged; if the packet loss rate of the client is greater than 8%, reduce the encoding rate by one level, such as reducing 1Mbps. If the bit rate of the current predicted bandwidth is at a high level, a conservative strategy should be adopted, including: if the packet loss rate of the client for two consecutive cycles is 0, increase the encoding bit rate by one level, such as 1Mbps; if the packet loss rate of the client is 0 When it is between 0-5%, keep the current encoding bit rate unchanged; if the packet loss rate of the client is greater than 5%, reduce the encoding bit rate by one level, such as reducing 3Mbps. It should be added that, assuming that 3 seconds is a statistical period, the downshift needs to be observed for one period, that is, 1 second, and the upshift needs to be observed for two periods, that is, 2 seconds.

S43: The server sends subsequent video frame data according to the adjusted bit rate.

All or part of the steps for implementing the above method embodiments may be completed by hardware related to computer programs. Based on this understanding, the present invention also provides a computer program product comprising one or more computer instructions. The computer instructions may be stored in a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a server, data center, etc., which includes one or more available media integrated. The usable medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a Solid State Disk (SSD)), and the like.

Referring to FIG. 5 , this embodiment provides a remote video transmission apparatus 50 . Since the technical principle of this embodiment is similar to that of the foregoing method embodiments, the same technical details will not be repeated. The apparatus 50 of this embodiment includes the following modules:

The video frame data sending module 51 is used for sending video frame data to the client.

Specifically, the video frame data sending module 51 sends the video frame data to the client according to the initial capture encoding sending frequency; when the number of video buffer frames in the video buffer area exceeds the capacity of the video buffer area, the capture encoding is reduced The sending frequency is to send video frame data to the client according to the reduced capture and encoding sending frequency; or, stop capturing the video frame data.

Specifically, when each frame of video data is sent, the video frame data sending module 51 stores the frame of video data in the video buffer, and waits for a reply from the client; receives the reply sent by the client ; Wherein, if the reply is successfully received, then clear the one frame of video data in the video buffer; if the reply of the successful reception is not received, then re-send the one frame of video data to the client.

The client information receiving module 52 is configured to receive the packet loss rate and bandwidth occupation sent by the client at preset time intervals.

Optionally, the client information receiving module 52 is further configured to receive the packet loss rate, bandwidth occupancy, and decoding and rendering capability of the client sent by the client every preset time. The apparatus 50 further includes an encoding frame rate adjustment module 53, configured to compare its current encoding frame rate with the decoding rendering capability, and adjust its encoding frame rate according to the minimum value of the two.

The encoding bit rate adjustment module 54 is configured to predict its own current bandwidth according to the packet loss rate and the bandwidth occupancy, so as to adjust its own encoding bit rate to adjust the video quality presented by the client.

Specifically, the encoding bit rate adjustment module 54 determines the preset bit rate level corresponding to the current bandwidth; searches for a preset adjustment strategy corresponding to the preset bit rate level, and adjusts its own encoding according to the preset adjustment strategy code rate.

The video frame data sending module 51 is further configured to send the video frame data according to the encoding bit rate adjusted by the encoding bit rate adjusting module 54 .

Those skilled in the art should understand that the division of each module in the embodiment of FIG. 5 is only a division of logical functions, and may be fully or partially integrated into one or more physical entities in actual implementation. And these modules can all be implemented in the form of software calling through processing elements, or all of them can be implemented in hardware, and some modules can be implemented in the form of calling software through processing elements, and some modules can be implemented in hardware.

Referring to FIG. 6 , this embodiment provides an electronic device, and the electronic device may be a desktop computer, a portable computer, a smart phone, or the like. In detail, the electronic device at least includes: a memory and a processor connected through a bus, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to execute all or part of the steps in the foregoing method embodiments.

The system bus mentioned above may be a Peripheral Pomponent Interconnect (PCI for short) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA for short) bus or the like. The system bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (eg client, read-write library and read-only library). The memory may include random access memory (Random Access Memory, RAM for short), and may also include non-volatile memory (non-volatile memory), such as at least one disk storage.

The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; may also be a digital signal processor (Digital Signal Processing, referred to as DSP) , Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

To sum up, the remote video transmission method, transmission device, storage medium and electronic equipment of the present invention can adjust and control the encoding, transmission and playback of video streams reasonably and in real time according to changing network conditions, so as to achieve a relatively high level of performance during remote access by the client. Good video definition and video fluency effectively overcome various shortcomings in the prior art and have high industrial application value.

The above-mentioned embodiments merely illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Anyone skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed in the present invention should still be covered by the claims of the present invention.

Claims

A remote video transmission method, characterized in that it is applied to a server; the method comprises:

Send video frame data to the client;

receiving the packet loss rate and bandwidth occupancy sent by the client every preset time;

Predict its own current bandwidth according to the packet loss rate and the bandwidth occupancy, so as to adjust its own encoding bit rate;

The video frame data is sent according to the adjusted coding rate.
The method according to claim 1, wherein the step of sending video frame data to the client comprises:

Send video frame data to the client according to the initial capture code sending frequency; wherein, the step of obtaining the initial capture code sending frequency includes:

Obtain the retransmission timeout time sent by the client;

Calculate the video buffer size according to the retransmission timeout time, and set the video buffer accordingly;

The transmission frequency of the initial capture encoding is determined according to the size of the video buffer and its own encoding capability.
The method according to claim 2, wherein the step of sending video frame data to the client further comprises:

When the number of video buffer frames in the video buffer area exceeds the capacity of the video buffer area, reduce the frequency of capturing and encoding sending, so as to send video frame data to the client according to the reduced sending frequency of capturing and encoding; or, stop Capture video frame data.
The method according to claim 1 or 2, wherein the step of sending video frame data to the client further comprises:

When each frame of video data is sent, store the frame of video data in the video buffer and wait for a reply from the client;

Receive the reply sent by the client; wherein, if the reply is successfully received, clear the one frame of video data in the video buffer; if the reply of the successful reception is not received, the A frame of video data is resent to the client.
The method according to claim 1, wherein the step of adjusting the self-encoding code rate according to the predicted self-current bandwidth comprises:

Determine the preset bit rate gear corresponding to the current bandwidth;

Find the preset adjustment strategy corresponding to the preset bit rate gear, and adjust the encoding bit rate according to the preset adjustment strategy.
The method according to claim 5, wherein the preset adjustment policy rule comprises: increasing the code rate as the packet loss rate decreases, and decreasing the code rate as the packet loss rate increases; wherein, the The higher the preset bit rate scale, the finer the adjustment granularity of the corresponding preset adjustment strategy.
The method of claim 1, further comprising:

receiving the decoding rendering capability sent by the client;

Compare its current encoding frame rate with the decoding rendering capability, and adjust its encoding frame rate according to the minimum value of the two.
A remote video transmission device, comprising:

The video frame data sending module is used to send video frame data to the client;

a client information receiving module, configured to receive the packet loss rate and bandwidth occupancy sent by the client every preset time;

an encoding rate adjustment module, configured to predict its own current bandwidth according to the packet loss rate and the bandwidth occupancy, so as to adjust its own encoding rate;

The video frame data sending module is further configured to send the video frame data according to the encoding bit rate adjusted by the encoding bit rate adjusting module.
A computer-readable storage medium storing a computer program therein, characterized in that, when the computer program is loaded and executed by a processor, the remote video transmission method according to any one of claims 1 to 7 is implemented.
An electronic device, comprising: a processor and a memory; wherein,

the memory is used to store computer programs;

The processor is configured to load and execute the computer program, so that the electronic device executes the remote video transmission method according to any one of claims 1 to 7.