CN112822521B

CN112822521B - Code rate control method, device, equipment and storage medium for audio and video transmission

Info

Publication number: CN112822521B
Application number: CN202011643196.8A
Authority: CN
Inventors: 马杰波; 高华
Original assignee: Bigo Technology Pte Ltd
Current assignee: Bigo Technology Pte Ltd
Priority date: 2020-12-30
Filing date: 2020-12-30
Publication date: 2023-04-25
Anticipated expiration: 2040-12-30
Also published as: CN112822521A

Abstract

The embodiment of the invention discloses a code rate control method, a device, equipment and a storage medium for audio and video transmission. Wherein the method comprises the following steps: and receiving playing condition information fed back by the client, wherein the playing condition information comprises a waiting playing time length corresponding to the audio and video data which are received and not played by the client and a time stamp of a last playing frame, estimating a receiving code rate of the client according to the playing condition information, determining sending condition information of the audio and video data, and determining a code rate control mode of audio and video data transmission according to the playing condition information, the receiving code rate of the client and the sending condition information. According to the technical scheme provided by the embodiment of the invention, comprehensive evaluation is performed by referring to the information of multiple dimensions, so that the code rate of audio and video transmission can be more reasonably controlled, and further the fluency of audio and video playing and the quality of audio and video playing are better considered.

Description

Code rate control method, device, equipment and storage medium for audio and video transmission

Technical Field

The embodiment of the invention relates to the technical field of code rate control of audio and video transmission, in particular to a code rate control method, a device, equipment and a storage medium of audio and video transmission.

Background

With the continuous development of multimedia information technology, audio and video information is largely emerging. Audio and video are used as an integrated medium for expressing information, and become an important information carrier in real life.

In some application scenarios, the audio and video data is usually required to be encoded when being transmitted, and in order to give consideration to the smoothness of audio and video playing and the quality of audio and video playing, the audio and video streams with different gear rates are generally selected according to the actual conditions and are transmitted with corresponding code rates due to the difference of network conditions in the actual scenarios.

Currently, a client Application (APP) on an audio and video data playing side generally provides multiple code rate gears for a user to select, and the higher the code rate, the higher the definition and the higher the requirements on network bandwidth and stability. The user usually performs the operation of manually selecting the code rate after the jamming occurs, for example, the code rate gear is adjusted down, sometimes the bandwidth of the user can support higher code rate and can have higher definition viewing experience, but in order to prevent the jamming generated when the network condition is abrupt, many users conservatively select lower code rate gears, and the mode does not fully utilize the network bandwidth of the user, and also reduces the viewing experience of the user.

In order to further improve the user experience, some APP's already provide a function of adaptive code rate. The self-adaptive code rate is that by adopting a certain algorithm, a proper code rate is automatically selected for the user, and a higher code rate is selected as much as possible while the blocking is reduced, so that better image quality is provided for the user, and the user experience is improved. In the prior art, the decision mode of code rate switching is usually a bandwidth-based method, and the code rate is switched to the corresponding code rate according to the network bandwidth condition detected by a congestion control algorithm adopted when audio and video data are transmitted, however, in a complex and changeable network environment, it is very difficult to detect accurate bandwidth only by the congestion control algorithm, and the inaccurate bandwidth estimation can cause poor effects of the method.

In summary, the current rate control scheme for audio and video transmission is still not perfect enough and needs improvement.

Disclosure of Invention

The embodiment of the invention provides a code rate control method, a device, equipment and a storage medium for audio and video transmission, which can optimize the existing code rate control scheme for audio and video transmission.

In a first aspect, an embodiment of the present invention provides a method for controlling a code rate of audio and video transmission, which is applied to a server in an audio and video transmission system, where the method includes:

Receiving play condition information fed back by a client, wherein the play condition information comprises residual playable duration and a timestamp of a last play frame, and the residual playable duration comprises to-be-played duration corresponding to audio and video data received and not played by the client;

estimating the receiving code rate of the client according to the playing condition information;

determining the sending condition information of the audio and video data;

and determining a code rate control mode of audio and video data transmission according to the play condition information, the receiving code rate of the client and the sending condition information.

In a second aspect, an embodiment of the present invention provides a rate control device for audio and video transmission, configured at a server in an audio and video transmission system, where the device includes:

the system comprises a play condition information receiving module, a play condition information processing module and a play control module, wherein the play condition information is used for receiving play condition information fed back by a client, the play condition information comprises residual playable duration and a time stamp of a last play frame, and the residual playable duration comprises to-be-played duration corresponding to audio and video data which is received and not played by the client;

the receiving code rate estimation module is used for estimating the receiving code rate of the client according to the playing condition information;

The sending condition information determining module is used for determining sending condition information of the audio and video data;

and the code rate control module is used for determining a code rate control mode of audio and video data transmission according to the play condition information, the receiving code rate of the client and the sending condition information.

In a third aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements a rate control method for audio/video transmission as provided in the embodiment of the present invention when the processor executes the computer program.

In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium having stored thereon a computer program, which when executed by a processor, implements a method for controlling a rate of audio/video transmission as provided by the embodiment of the present invention.

The code rate control scheme of the audio and video transmission provided by the embodiment of the invention is applied to a server side in an audio and video transmission system, receives play condition information fed back by a client side, wherein the play condition information comprises residual playable time length and a time stamp of a last play frame, the residual playable time length comprises time length to be played corresponding to audio and video data which is received and not played by the client side, estimates the receiving code rate of the client side according to the play condition information, determines the sending condition information of the audio and video data, and determines the code rate control mode of the audio and video data transmission according to the play condition information, the receiving code rate of the client side and the sending condition information. By adopting the technical scheme, the comprehensive evaluation is performed by referring to the information of multiple dimensions, so that the code rate of audio and video transmission can be more reasonably controlled, and further the fluency of audio and video playing and the quality of audio and video playing are better considered.

Drawings

Fig. 1 is a flow chart of a code rate control method for audio/video transmission according to an embodiment of the present invention;

fig. 2 is a schematic diagram of a process of interaction between a server and a client according to an embodiment of the present invention;

fig. 3 is a flow chart of another method for controlling a code rate of audio/video transmission according to an embodiment of the present invention;

fig. 4 is a schematic view of a video stream status according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of still another video stream status provided in an embodiment of the present invention;

FIG. 6 is a schematic diagram of still another video stream status provided in an embodiment of the present invention;

fig. 7 is a flow chart of another method for controlling the rate of audio/video transmission according to an embodiment of the present invention;

fig. 8 is a flow chart of another method for controlling a code rate of audio/video transmission according to an embodiment of the present invention;

fig. 9 is a block diagram of a code rate control device for audio/video transmission according to an embodiment of the present invention;

fig. 10 is a block diagram of a computer device according to an embodiment of the present invention.

Detailed Description

The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings. Furthermore, embodiments of the invention and features of the embodiments may be combined with each other without conflict.

Fig. 1 is a flow chart of a method for controlling a rate of audio/video transmission according to an embodiment of the present invention, where the method may be performed by a rate control device for audio/video transmission, where the device may be implemented by software and/or hardware, and may generally be integrated in a computer device such as a server. The code rate control method for audio and video transmission provided by the embodiment of the invention can be suitable for various audio and video transmission scenes, including but not limited to various application scenes related to audio and video transmission such as audio and video live broadcasting, audio and video on demand, video call, voice call, video conference, voice conference, remote medical treatment, remote education and the like in the Internet. The system formed by the sending end and the receiving end in the audio and video transmission scene can be called an audio and video transmission system, the technical scheme of the embodiment of the invention is applied to a server end responsible for sending audio and video data in the audio and video transmission system, and the receiving end serving as a playing end of the audio and video data is called a client end.

As shown in fig. 1, the method includes:

step 101, receiving play condition information fed back by a client, wherein the play condition information comprises residual playable duration and a timestamp of a last play frame, and the residual playable duration comprises a duration to be played corresponding to audio and video data received and not played by the client.

For example, the server may maintain a transmission window corresponding to the current client (hereinafter referred to as a client) in the server, calculate a downlink transmission rate by using the transmission window and use a preset congestion control algorithm, and send a data packet, that is, send audio and video data, to the client at the downlink transmission rate. After receiving the data packet sent by the server, the client may return play condition information to the server, where the play condition information may be included in an acknowledgement character (Acknowledge character, ack) data packet (ack packet for short), and the sending timing of the ack packet is not specifically limited, for example, may be a timing sending. The ack packet may be accompanied by a remaining playable duration (jitter len) and a timestamp (lastPlayedPts) of a last playing frame, where jitter len may represent a remaining playing duration of locally received data determined by the client at a time of sending the ack packet, that is, a duration required by the client to play audio and video data that is not played in the received data from a time of sending the ack packet. lastPlayedPts may represent the timestamp of the last frame in the played data frames that the client determines at the time of sending the ack packet.

For example, the play situation information in this step may include play situation information in ack packets that are continuously transmitted multiple times by the client.

And 102, estimating the receiving code rate of the client according to the playing condition information.

The first preset estimation algorithm may be used to calculate the receiving code rate of the client, and may be selected according to the actual situation, considering that the change of the network condition is generally affected by multiple factors in different application scenarios. For example, when the network condition is relatively stable, the reception code rate of the client may be calculated according to the play situation information in the ack packet received last two times; when the requirement on the calculation accuracy is high, the receiving code rate of the client can be calculated according to the playing condition information in the ack packets which are received continuously for many times. Optionally, in the embodiment of the present invention, the receiving code rate of the first preset duration before the last group of pictures (Group of Pictures, GOP) may be calculated according to the playing condition information in the ack packets that are received continuously and repeatedly. The first preset time period may be set according to actual requirements, for example, 200 milliseconds (ms). For example, when calculating the receiving code rate according to the play situation information, for the two ack packet sending moments, the remaining playable duration corresponding to the first moment is denoted as jitter len1, the time stamp of the last play frame is denoted as jitter lens 1, the remaining playable duration corresponding to the second moment is denoted as jitter lens 2, the time stamp of the last play frame is denoted as jitter lens 2, the play duration corresponding to the audio/video data received between the first moment and the second moment, for example, (jitter lens 2-jitter lens 1) +the received audio/video data between the first moment and the second moment, may be calculated, and the receiving code rate of the client may be estimated according to the data amount and the corresponding time difference (the time difference between the first moment and the second moment) according to the play duration and the play rate.

Step 103, determining the sending condition information of the audio and video data.

The transmission condition information may include information related to transmission of audio and video data of the server, for example, may include a transmitted time length of a current frame group, a transmitted data amount size of the current frame group, a transmitted proportion of the current frame group, an untransmitted time length of the current frame group, an untransmitted data amount size of the current frame group, an untransmitted proportion of the current frame group, a current code rate or a current code rate gear of transmitting audio and video data, a time length of waiting for a transmission queue of the server, an audio and video data amount of waiting for a transmission queue of the server, and the like; for application scenes such as live video, the server side also needs to receive the audio and video streams from the anchor side or the content distribution platform, and at this time, the sending condition information may further include remaining transmittable data of the server side, remaining transmittable duration of the server side, duration of unreceived data in the current picture group, and the like.

And 104, determining a code rate control mode of audio and video data transmission according to the play condition information, the receiving code rate of the client and the sending condition information.

The playing condition information can embody the playing progress of the client and the capability of resisting network fluctuation, the receiving code rate can embody the capability of receiving video data by the client, the transmitting condition information can embody the capability of transmitting audio and video data by the server, the code rate control can be performed on the basis of a preset control strategy by integrating the plurality of information, and the specific control strategy can be set according to actual requirements.

According to the bit rate control method for audio and video transmission, which is provided by the embodiment of the invention, the play condition information fed back by the client is received, wherein the play condition information comprises the time to-be-played duration corresponding to the audio and video data which is received and not played by the client and the time stamp of the last play frame, the receiving bit rate of the client is estimated according to the play condition information, the sending condition information of the audio and video data is determined, and the bit rate control mode for the audio and video data transmission is determined according to the play condition information, the receiving bit rate of the client and the sending condition information. According to the technical scheme provided by the embodiment of the invention, comprehensive evaluation and decision are performed by referring to the information of multiple dimensions, so that the accuracy and timeliness of decision can be improved, the code rate of audio and video transmission can be more reasonably controlled, and the fluency of audio and video playing and the quality of audio and video playing can be better considered.

The bit rate refers to the bit rate in unit time and is used for measuring the unit time volume of the audio and video data. The code rate can be divided, each code rate gear can correspond to one code rate value or one code rate value interval, when one code rate value is corresponding, the code rate values corresponding to two adjacent code rate gears can be continuous or jump, and a specific dividing strategy can be set according to actual requirements. For example, the audio/video transmission system can provide several selectable code rate gears for playing of the client, such as standard definition, high definition, super definition, blue light, and the like, and of course, a gear dividing mode with higher precision can be provided, since the audio/video stream conversion rate needs computing resources, an arbitrary code rate value is not generally selected, the audio/video stream can be converted into audio/video data with different code rate gears at a server or a content distribution platform, and the like in advance, and then the audio/video data with appropriate code rate gears is selected according to a specific code rate control strategy and sent to the client. For rate control in the audio/video data transmission process, the method can generally comprise maintaining the current rate gear unchanged, reducing the rate gear, increasing the rate gear and the like.

The control strategy for reducing the code rate gear is further described below.

In some embodiments, the determining a rate control manner of audio and video data transmission according to the play condition information, the receiving rate of the client and the sending condition information includes: estimating estimated residual playable duration corresponding to a first data receiving time of the client according to the play condition information, wherein the first data receiving time comprises a time when currently transmitted data is expected to be received at the client; determining first data to be transmitted according to the transmission condition information; determining playable duration corresponding to the first data to be sent, and determining the first duration corresponding to the first data to be sent according to the receiving code rate of the client; determining a target residual playable duration corresponding to the client after the first duration to be sent is passed according to the estimated residual playable duration and the playable duration; and when the target residual playable duration is smaller than a first preset threshold value, determining to reduce the code rate gear of the audio and video data transmission. The advantage of this arrangement is that the control to reduce the code rate gear can be more reasonably performed.

Fig. 2 is a schematic diagram of an interaction process between a server and a client, where audio and video data are taken as video data, as shown in fig. 2, and at time t1, the server (server) sends video data (video) to the client (client); at time t1', the client receives the video data sent at time t1 of the server, returns an ack packet corresponding to the video data to the server, and is attached with jitter and lastPlayedPts at time t1' of the client; at time t2, the server receives the ack packet and performs up-down gear judgment. The current transmission data refers to data which is expected to be transmitted to the client at the current moment, that is, the current moment is assumed to be the t2 moment, if the video data is transmitted to the client at the t2 moment, the predicted client receives the video data at the t2 'moment, the t2' moment is equal to the first data receiving moment, the predicted residual playable duration corresponding to the first data receiving moment can be understood as the residual playable duration estimated by the server at the client side at the first data receiving moment, and can be noted as jitter len ', because the server starts to judge the cut at the t2 moment, and all data packets transmitted after t2 can reach the client after t2', the specific value of the predicted residual playable duration can be estimated according to the playing condition information, and the specific estimation mode is not limited.

In some embodiments, the estimating, according to the play condition information, an estimated remaining playable duration corresponding to the first data receiving time of the client may include: determining the maximum round trip delay in a first preset historical period according to the play condition information; calculating a first estimated residual playable duration according to the latest residual playable duration, the maximum round trip delay, the latest round trip delay, the receiving code rate of the client and the code rate of the current code rate gear in the play condition information; and/or calculating a second estimated residual playable duration according to the timestamp of the last transmitted frame, the timestamp of the latest last played frame in the play condition information and the maximum round trip delay; and determining the first estimated remaining playable duration, the second estimated remaining playable duration or the minimum value of the first estimated remaining playable duration and the second estimated remaining playable duration as the estimated remaining playable duration corresponding to the first data receiving time of the client. The method has the advantages that the estimated residual playable duration can be estimated more accurately, and further, whether the downshift control is needed or not can be judged accurately.

For example, there may be two ways to estimate the estimated remaining playable time.

First, a first estimated remaining playable duration is calculated according to the latest remaining playable duration, a first difference between a first data receiving time and a sending time of the latest remaining playable duration, a second difference between a current time and a time of a last sending data packet, a receiving code rate of the client, and a code rate of a current code rate gear (which can be understood as an estimated remaining playable duration calculated by adopting a first mode). Specifically, the latest residual playable duration minus the product of the first difference plus the second difference and the first ratio may be calculated to obtain a first estimated residual playable duration, where the first ratio (may be denoted as R0) is a ratio of a receiving code rate (may be denoted as recvCodeRate) of the client to a code rate (may be denoted as curCodeRate) of the current code rate gear. For example, the estimated remaining playable time is calculated as follows:

jitterLen’＝jitterLen–(t2’-t1’)+(t2-t1)*R0

the first difference may be replaced by a maximum Round Trip Time (RTT) within a first preset history period, and the second difference may be replaced by a last Round Trip Time. The maximum round trip time delay can be recorded as maxRtt, the first preset historical time period can be set according to actual conditions, and the current time is generally traced back for a period of time; the last round trip delay may be denoted lastRtt. Correspondingly, t2'-t1' takes maxRtt and t2-t1 takes lastRtt, then the above formula can be expressed as:

jitterLen’＝jitterLen–maxRtt+lastRtt*R0

Second, a second estimated remaining playable duration is calculated according to the timestamp of the last transmitted frame, the timestamp of the latest last played frame in the play condition information, and the first difference value (which may be understood as the estimated remaining playable duration calculated by the second method). Specifically, a difference between the timestamp of the last transmitted frame and a first sum value, which is the sum of the timestamp of the latest last played frame and the first difference value, may be calculated. Wherein the timestamp of the last transmitted frame may be denoted lastSentPts. For example, the estimated remaining playable time is calculated as follows:

jitterLen’＝lastSentPts–(lastPlayedPts+(t2’-t1’))

wherein, t2'-t1' takes maxRtt, the above formula can be expressed as:

jitterLen’＝lastSentPts-lastPlayedPts–maxRtt

alternatively, the two calculation modes may be combined, and the worst case is considered, and the minimum value thereof is taken. Thus, the estimated remaining playable time period may be expressed as:

jitterLen’＝min(jitterLen-maxRtt+lastRtt*R0，lastSentPts-lastPlayedPts-maxRtt)。

the first data to be transmitted is determined according to the transmission condition information, which can be understood as determining the data amount for simulating the next data transceiving process according to the current audio/video data transmission condition of the server. The determining of the playable duration corresponding to the first data to be sent may be understood as estimating the playable duration that may be increased after the client finishes receiving the first data to be sent. The first duration to be sent corresponding to the first data to be sent is determined according to the receiving code rate of the client, which can be understood as the duration consumed by the client for receiving the first data to be sent according to the currently calculated receiving code rate of the server.

After the estimated remaining playable time length, the playable time length and the first waiting time length are obtained, the target remaining playable time length corresponding to the client after the first waiting time length is passed can be estimated. After waiting for the first waiting time period from the first data receiving time, the client is expected to successfully receive the first waiting data, so that the playable time period corresponding to the first waiting data is newly increased on the basis of estimating the residual playable time period, and the residual playable time period is reduced due to the continuing of the playing operation because the client still plays in the process of receiving the first waiting data, and the reduced amount is the first waiting time period. The specific value of the target remaining playable duration may be a sum of the estimated remaining playable duration and the playable duration, minus the first duration to be transmitted.

The remaining playable duration may be understood as a duration that the data cached by the client can be played continuously, and if situations such as poor network quality occur, and the like, the subsequent playing can still be performed according to the cached data when the received video data cannot be successfully received, so that the capability of the client against network fluctuation can be reflected. After the target residual playable duration is obtained, the size of the target residual playable duration and the first preset threshold can be judged, if the target residual playable duration is smaller than the first preset threshold, the current capability of resisting network fluctuation can be poor, and in order to reduce the transmission data quantity with the same playing duration, the code rate gear of audio and video data transmission needs to be reduced, so that the residual playable duration is facilitated to be improved.

In some embodiments, the first preset threshold is determined by: determining the maximum residual playable duration in a second preset history period according to the play condition information; determining a first residual multicast duration threshold according to the product of the maximum residual multicast duration and a preset specific gravity value; and determining the minimum value of the first residual multicast duration threshold and the preset residual multicast duration threshold as a first preset threshold. The advantage of this arrangement is that the first preset threshold value is reasonably determined, and thus whether a downshift is required or not is accurately determined. The preset residual playable duration threshold value can be an experience value set according to actual conditions and can be recorded as jitter; the second preset history period may be set according to actual situations, and generally, a period of time is traced back from the current time, the remaining playable duration with the largest value may be selected from the ack packets received in the second preset history period, which may be denoted as maxJitter, and the product of the maxJitter and a preset specific gravity value (which may be an empirical value set according to actual situations, which may be denoted as maxJitterScale, and may range from 0 to 100%) is determined as the first remaining playable duration threshold. The minimum value of the first remaining playable duration threshold and the preset remaining playable duration threshold is determined as a first preset threshold, that is, the first preset threshold may be expressed as min (maxJitter).

In some embodiments, the transmission case information includes a transmitted time length of the current picture group; the determining the first data to be transmitted according to the transmission condition information comprises the following steps: when the transmitted duration is smaller than or equal to a second preset threshold value, the first data to be transmitted comprises the non-transmitted data corresponding to the current code rate gear in the current picture group; when the transmitted time length is greater than the second preset threshold value, the first data to be transmitted comprises the unsent data corresponding to the current code rate gear in the current picture group and the data corresponding to the current code rate gear in the next picture group. The advantage of this arrangement is that the first data to be transmitted can be determined reasonably, and thus the timing of the downshift determination can be determined accurately.

Illustratively, since the later video frames need to be decoded and played with reference to the previous video frames in the same GOP, switching the entire GOP is required when switching video streams. It is therefore necessary to consider whether to switch from the current GOP or from the next GOP, the current GOP generally having already transmitted a portion of the video duration, and if the current GOP has already been transmitted for a long time (for example, the length of the GOP is 2s and the transmission duration curgolvent is 1.5 s), the switch from the current GOP is not appropriate at this time, since there is 1.5s of data that is repeatedly transmitted, and thus a threshold value gopSentThr (i.e., a second preset threshold value) is set, and when curgolent < = gopSentThr, it is considered to switch from the current GOP to the next GOP when curgolent > gopSentThr. The second preset threshold may be determined according to the length (duration) of one GOP, and may be, for example, a first preset ratio (e.g., 0.5, etc.) of GOP durations.

In some embodiments, when the sent duration is less than or equal to the second preset threshold, after determining to reduce the code rate gear of the audio-video data transmission, the method further includes: determining a second to-be-transmitted duration corresponding to second to-be-transmitted data according to the receiving code rate of the client, wherein the second to-be-transmitted data comprises data corresponding to a candidate code rate in a current picture group, and the candidate code rate is lower than the code rate of the current code rate gear; determining candidate residual playable duration corresponding to the client after the second waiting time duration passes according to the estimated residual playable duration; determining a candidate residual playable duration which is greater than or equal to the first preset threshold value as a first candidate residual playable duration, and determining a corresponding first target code rate gear according to the first candidate residual playable duration; and starting to transmit the audio and video data of the first target code rate gear from the first frame of the current picture group. This has the advantage that for the case of a switch from the current group of pictures, a downshift gear meeting the requirements is selected quickly and accurately.

For example, after the switching from the current picture group is determined, audio/video transmission is required to be performed with the downshifted code rate, that is, all data corresponding to the downshifted code rate of the current picture group is required to be transmitted, and the downshifted code rate is called a candidate code rate, and may be any code rate lower than the current code rate or an unknown code rate required to be obtained. And determining the time required for completing the transmission of all data corresponding to the candidate code rate of the current picture group according to the receiving code rate of the client, namely the second time length to be transmitted. And then, determining the corresponding candidate residual playable duration of the client after the second waiting duration passes, comparing the candidate residual playable duration with a first preset threshold, and if the candidate residual playable duration is larger than or equal to the first preset threshold, indicating that the candidate code rate corresponding to the candidate residual playable duration accords with the expected code rate. It can be understood that there may be a plurality of candidate code rates meeting the expectations, and in order to consider quality indexes such as picture definition of audio and video playing, the highest candidate code rate may be selected. Optionally, the determining, according to the first candidate remaining playable duration, a corresponding first target code rate gear includes: and determining the highest code rate gear in the candidate code rates corresponding to the first candidate residual playable time length as a first target code rate gear, namely determining the highest code rate gear which is smaller than the maximum value of the candidate code rates as the first target code rate gear.

In some embodiments, when the transmitted time period is greater than the second preset threshold, after determining to reduce the code rate gear of the audio-video data transmission, the method further includes: judging whether the residual transmittable data of the server side in the sending condition information contains data corresponding to the current code rate gear in the next picture group or not, wherein the residual transmittable data comprises audio and video data which are received and not sent in the server side; determining a second target code rate gear by adopting a corresponding code rate reduction strategy according to the judging result; and continuing to transmit the audio and video data of the current code rate gear of the current picture group, and starting from the first frame of the next picture group to transmit the audio and video data of the second target code rate gear. The advantage of this arrangement is that, for the case that the current frame group already transmits more data, the data transmission after the downshifting of the next frame group is started after the current frame group is transmitted, so as to avoid transmitting too many repeated audio and video contents and save bandwidth resources.

Because there may be a delay between the audio/video stream of the new gear and the audio/video stream of the old gear when the gear is switched, that is, after the data packet of the current frame group may be sent, the next frame group switched will not start to be received immediately, that is, the data of the next frame group switched will not arrive yet, but only the transmission of the audio/video data can be paused and the arrival of the next frame group is waited. If the remaining transmittable data of the server side does not contain the data corresponding to the current code rate gear in the next picture group, the data corresponding to the current code rate gear of the current picture group needs to be received, and then the transmission of the data corresponding to the downshift code rate in the next picture group can be started at least after the delay. If the remaining transmittable data of the server already contains data corresponding to the current code rate gear in the next picture group, the data of the current picture group do not need to be received, and before the received data of the current picture group are transmitted, the data corresponding to the down-shift code rate in the next picture group can be directly received. Therefore, for the two cases, a second target code rate gear needs to be determined by adopting different code rate reduction strategies.

In some embodiments, the determining the second target code rate gear according to the determination result by using a corresponding code rate reduction policy includes: determining a third to-be-transmitted duration corresponding to third to-be-transmitted data according to a receiving code rate of the client by adopting a corresponding determining mode according to a judging result, wherein the third to-be-transmitted data comprises non-transmitted data corresponding to a current code rate gear in the current picture group and data corresponding to a candidate code rate in a next picture group, and the candidate code rate is lower than the code rate of the current code rate gear; determining candidate residual playable duration corresponding to the client after the third waiting time duration passes according to the estimated residual playable duration; and determining the candidate residual playable duration which is greater than or equal to the first preset threshold value as a second candidate residual playable duration, and determining a corresponding second target code rate gear according to the second candidate residual playable duration. This has the advantage that the desired downshift is selected quickly and accurately for the case of a switch from the next group of pictures. Optionally, the determining, according to the second candidate remaining playable duration, a corresponding second target code rate gear includes: and determining the highest code rate gear in the candidate code rates corresponding to the second candidate residual playable time length as a second target code rate gear, namely determining the highest code rate gear which is smaller than the maximum value of the candidate code rates as the second target code rate gear.

In some embodiments, when the remaining transmittable data of the server does not include data corresponding to the current code rate gear in the next frame group, determining, according to the determination result, a third duration to be transmitted corresponding to the third data to be transmitted according to the receiving code rate of the client by adopting a corresponding determination method, where the determining includes: calculating a fourth time length to be transmitted of the unsent data corresponding to the current code rate gear in the current picture group according to the receiving code rate of the client; determining the sum of a first receiving duration and a first estimated switching delay duration of unreceived data corresponding to a current code rate gear in the current picture group as a fifth duration to be transmitted; calculating a sixth time length to be transmitted according to the candidate code rate, the receiving code rate of the client and the time length of the next picture group; and determining the sum of the maximum value of the fourth to-be-sent time length and the fifth to-be-sent time length and the sixth to-be-sent time length as a third to-be-sent time length corresponding to third to-be-sent data. The method has the advantages that the third waiting sending duration can be calculated in a targeted manner according to the situation that the residual transmittable data of the server does not contain the data corresponding to the current code rate gear in the next picture group, and the second target code rate gear can be calculated accurately.

The fourth duration to be sent may be understood as, for example, the time required for the server to complete the sending of the remaining unsent data in the current frame group. The fifth time length to be sent may be understood as a total time length of receiving the unreceived data of the current frame group and waiting for the switching delay, where the first estimated switching delay time length may be estimated by a historical audio/video stream switching process, for example, may be an average value of multiple switching delay time lengths, and a difference between any one of the code rate gear before switching and the code rate gear after switching may make the corresponding first estimated switching delay time length different. The sixth duration to be sent may be understood as a time required for sending data corresponding to the candidate code rate of the next frame group, and may specifically be a value obtained by dividing a product of the code rate of the candidate code rate and the duration of the next frame group by the receiving code rate of the client. Since the transmission of the data of the next group of pictures needs to be performed after the transmission of the current group of pictures is completed and after the reception of the data of the next group of pictures has been started, the total waiting-to-be-transmitted time period (third waiting-to-be-transmitted time period) is the sum of the sixth waiting-to-be-transmitted time period and the larger value of the fourth waiting-to-be-transmitted time period and the fifth waiting-to-be-transmitted time period.

In some embodiments, when the remaining transmittable data of the server already includes data corresponding to a current code rate gear in a next frame group, determining, according to a determination result, a third duration to be transmitted corresponding to the third data to be transmitted according to a receiving code rate of the client by adopting a corresponding determination method, where the determining includes: calculating a fourth time length to be transmitted of the unsent data corresponding to the current code rate gear in the current picture group according to the receiving code rate of the client; determining a difference between a first estimated switching delay time and a first time length as a seventh time length to be transmitted, wherein the first time length is a difference between a transmission time length corresponding to the residual transmittable data of the server and the fourth time length to be transmitted; calculating a sixth time length to be transmitted according to the candidate code rate, the receiving code rate of the client and the time length of the next picture group; and determining the sum of the maximum value of the fourth to-be-sent time length and the seventh to-be-sent time length and the sixth to-be-sent time length as a third to-be-sent time length corresponding to third to-be-sent data. The method has the advantages that the third waiting sending duration can be calculated in a targeted manner according to the condition that the residual transmittable data of the server contains the data corresponding to the current code rate gear in the next picture group, and the second target code rate gear can be calculated accurately.

For example, the calculation manner may be the same when the fourth to-be-transmitted duration and the sixth to-be-transmitted duration are the same as those when the remaining transmittable data of the server does not include the data corresponding to the current code rate gear in the next frame group. The seventh waiting time period may be understood as the time required to wait for the end of the switching delay.

Fig. 3 is a flow chart of another method for controlling a code rate of audio/video transmission according to an embodiment of the present invention, as shown in fig. 3, the method may include:

step 301, receiving the play condition information fed back by the client, determining the sending condition information of the audio and video data, and estimating the receiving code rate of the client and the estimated residual playable duration corresponding to the first data receiving time of the client according to the play condition information.

The play condition information includes a remaining playable duration (jitter len) and a timestamp (lastPlayedPts) of a last play frame, where the remaining playable duration includes a duration to be played corresponding to audio and video data received and not played by the client. The sending condition information of the audio and video data comprises the sent time length (curGopSent) of the current picture group and the time length (waitLen) corresponding to the residual transmittable data of the server side. For example, the receiving code rate of the client is recvCodeRate, the estimated remaining playable duration is jitterLen', and the specific calculation method can refer to the above related content.

Step 302, judging whether the sent duration of the current picture group is less than or equal to a second preset threshold value, if yes, executing step 303; otherwise, step 305 is performed.

The sent duration of the current frame group may be denoted as curgolpsent, and the second preset threshold may be denoted as gopsetthr. Switching from the current GOP is considered when curgolvent < = gopSentThr, and switching from the next GOP is considered when curgolvent > gopSentThr.

Fig. 4 is a schematic view of a video stream state provided by the embodiment of the present invention, as shown in fig. 4, a current stream may be understood as an audio/video stream corresponding to a current code rate gear, a switching stream may be understood as an audio/video stream corresponding to a code rate gear after switching, a position of a last frame sent in a current frame group may be denoted as lastend, waitLen represents a duration (unit ms) of a waiting for a sending queue of a server, that is, a duration corresponding to remaining transmittable data of the server, and curgolncome represents a duration (unit: ms) of a current frame group not reaching.

Step 303, determining the unsent data corresponding to the current code rate gear in the current picture group as first data to be sent, determining a playable duration corresponding to the first data to be sent and the first duration to be sent, and determining a target remaining playable duration corresponding to the client after the first duration to be sent passes from the first data receiving moment according to the estimated remaining playable duration and the playable duration corresponding to the first data to be sent.

Illustratively, the remaining playable duration of the current GOP (playable duration corresponding to the first data to be transmitted): t0=curgolpfleft=waitlen+curgolpincomem.

The predicted remaining transmission duration (first to-be-transmitted duration) of the current GOP is: t1=curgolepleftsize 8/recvCodeRate.

After the T1 time, the playable duration of the client (the target remaining playable duration) may be estimated: jitterlent1=jitterlen' +t0-T1.

And 304, when the target residual playable duration is smaller than a first preset threshold value, determining a corresponding first target code rate gear, and transmitting audio and video data of the first target code rate gear from a first frame of the current picture group.

The first predetermined threshold may be expressed as min (maxjittr, maxjittterscale), and the specific calculation method may refer to the above related content. That is, a downshift is required when the jitter lent1< min (jitter thr, maxJitter), and then it is determined to which code rate level to specifically downshift, and after the downshift, the jitter lent1 cannot be lower than the first preset threshold when the current GOP is transmitted.

Switching from the current GOP to send the whole GOP, that is, the second to-be-sent data includes data corresponding to the candidate code rate in the current group of pictures, the predicted remaining sending duration (second to-be-sent duration) of the current GOP after the downshift is: t2=targetcoderate =goplen/recvCodeRate, where targetCodeRate is the target rate (i.e. candidate rate), and gopLen represents the duration (units: ms) of the GOP.

After the T2 time, the playable duration of the client (candidate remaining playable duration) may be estimated:

jitterLenT2＝jitterLen’+T0–T2

＝jitterLen’+T0-targetCodeRate*gopLen/recvCodeRate。

let jitterlent2=min (jitterr, maxjittterscale) calculate the critical value (i.e. the maximum value) of targetCodeRate:

targetCodeRate＝(min(jitterThr,maxJitter*maxJitterScale)-jitterLen’-T0)*recvCodeRate/gopLen。

and selecting a gear with the highest code rate from the gears with the code rate lower than the targetCodeRate during the downshift, namely determining the gear with the highest code rate from the gears with the code rate lower than the targetCodeRate as the first target code rate gear.

Alternatively, when the target remaining playable duration is greater than or equal to the first preset threshold, it may be considered that no downshift is required, step 301 may be executed back, or an upshift determination may be made.

Step 305, determining the unsent data corresponding to the current code rate gear in the current picture group and the data corresponding to the current code rate gear in the next picture group as first data to be transmitted, determining a playable duration corresponding to the first data to be transmitted and a first duration to be transmitted, and determining a target remaining playable duration corresponding to the client after the first duration to be transmitted passes from the first data receiving moment according to the estimated remaining playable duration and the playable duration corresponding to the first data to be transmitted.

Illustratively, the playable duration (playable duration corresponding to the first data to be transmitted) until the next GOP is: t0=curgolpfleft+hoplen.

The predicted transmission period (first to-be-transmitted period) until the next GOP is: t1=curgolefsize 8/recvcoderate+hoplen/R0.

And 306, when the target residual playable duration is smaller than a first preset threshold value, determining a corresponding second target code rate gear, continuing to transmit the audio and video data of the current code rate gear of the current picture group, and starting to transmit the audio and video data of the second target code rate gear from the first frame of the next picture group.

A downshift is required when jitterLenT1< min (jitterThr, maxJitterScale), and then it is determined to which code rate level to specifically downshift, and after the downshift, jitterLenT1 cannot be lower than the first preset threshold when the current GOP is transmitted. Specifically, the second target code rate gear is determined according to whether the remaining transmittable data of the server side contains data corresponding to the current code rate gear in the next picture group, that is, according to whether the duration corresponding to the remaining transmittable data of the server side contains duration in the next picture group, in two cases.

In the first case, the remaining transmittable data of the server does not include data corresponding to the current code rate gear in the next frame group. Fig. 5 is a schematic view of still another video stream state provided in an embodiment of the present invention, where now represents the current time, and the data corresponding to the current waitLen has no packet of the next group of pictures.

After the shift, the predicted transmission duration for the next GOP is:

T2＝max(curGopLeftSize*8/recvCodeRate,curGopIncom+delay)+targetCodeRate*gopLen/recvCodeRate

here, curgolftsize 8/recvCodeRate is the time length when the current GOP is transmitted (fourth to-be-transmitted time length), curgolncom+delay is the time length when the next GOP is waiting for the shift position to be transmitted (fifth to-be-transmitted time length), and since the next GOP for switching the shift position is not yet reached when the video packet of the current GOP is possibly transmitted, only no packet is available, and so on, GOP, targetCodeRate ×golen/recvCodeRate of the shift position is transmitted after the time length of max (curgolftsize 8/recvCodeRate, curGopIncom +delay), and the sum is the predicted transmission time length T2 (third to-be-transmitted time length) of the next GOP.

jitterLenT2＝jitterLen’+T0–T2

＝jitterLen’+T0–max(curGopLeftSize*8/recvCodeRate,curGopIncom+delay)+targetCodeRate*gopLen/recvCodeRate

let jitterlent2=min (jitterr, maxjittterscale) calculate the threshold value of targetCodeRate:

targetCodeRate＝((jitterLen’+T0-min(jitterThr,maxJitter*maxJitterScale))-max(curGopLeftSize*8/recvCodeRate,curGopIncome+delay))*recvCodeRate/gopLen

and selecting a gear with the highest code rate from the gears with the code rate lower than the targetCodeRate during the downshift.

In the second case, the remaining transmittable data of the server already includes data corresponding to the current code rate gear in the next frame group. Fig. 6 is a schematic view of still another video stream state according to an embodiment of the present invention, as shown in fig. 6, now represents the current time, and the data corresponding to the current waitLen already includes the packet of the next frame group.

After the shift, the transmission duration is expected until the next GOP:

T2＝max(curGopLeftSize*8/recvCodeRate,delay-(waitLen-curGopLeft))+targetCodeRate*gopLen/recvCodeRate。

after the time T2, the playable duration of the client may be estimated:

jitterLenT2＝jitterLen’+T0-T2

＝jitterLen’+T0-max(curGopLeftSize*8/recvCodeRate,delay-(waitLen-curGopLeft))+targetCodeRate*gopLen/recvCodeRate。

targetCodeRate＝((jitterLen+T0-min(jitterThr,maxJitter*maxJitterScale))-max(curGopLeftSize/recvCodeRate,delay-(waitLen-curGopLeft))*recvCodeRate/gopLen。

In some embodiments, an upshift decision for rate control may also be performed according to the play condition information, the receiving rate of the client, and the sending condition information.

The control strategy for raising the code rate gear is further described below.

The method for determining the rate control mode of audio and video data transmission according to the play condition information, the receiving rate of the client and the sending condition information includes: when the transmission condition information meets a preset upshift detection condition, starting to transmit a detection redundant packet and gradually increasing the duty ratio of the detection redundant packet in transmission data; and in the process of sending the detection redundant packet, if the play condition information and the receiving code rate of the client side are determined to meet the preset upshift condition, stopping sending the detection redundant packet, and improving the code rate gear of audio and video data transmission. The advantage of setting up like this is that before carrying out the upshift, can carry out the simulation upshift through the mode that sends the detection redundant package, and then confirm whether current network reaches the upshift condition in order to reach the purpose that prevents misjudgement, prevents that blind upshift from appearing and leads to the network congestion to appear the problem of blocking. The preset upshift detection condition may be set according to an actual requirement, and is generally a condition for primarily determining that an upshift requirement exists, which is not particularly limited. The detection redundancy packet may be freely set, for example, may be a retransmission packet in a retransmission queue, or may be a newly received data packet, etc. In the process of sending the detection redundant packet, the duty ratio of the detection redundant packet is continuously increased, namely the redundancy rate is improved, meanwhile, the receiving code rate of the client estimated by the server is also continuously increased, and when the preset upshift condition is met, upshift can be truly started. The preset upshift condition can be set according to actual requirements, and is generally a condition for determining that the current network bandwidth supports upshift. In the upshift, the upshift range is not particularly limited, and may be, for example, a code rate shift range.

In some embodiments, the transmitting the condition information satisfies a preset upshift detection condition, including: and the residual transmittable data of the server side contained in the transmission condition information is smaller than or equal to a third preset threshold value, and the minimum bandwidth calculated by the transmission window is larger than the product of a preset coefficient and the maximum transmission rate. The advantage of this arrangement is that by reasonably setting the preset upshift detection conditions, the timing of upshift detection can be accurately determined. The minimum bandwidth may be recorded as minBandwidth, and may be calculated by using a preset congestion control algorithm. The maximum transmission rate may be denoted as maxshendrate and may be determined based on the actual transmission rate of the transmission window. The preset coefficient may be noted as n and may be an empirical value between 1 and 2. For example, the maximum transmission rate may be a larger value of the actual transmission rate and the current code rate (1+lossrate), where lossRate represents the current packet loss rate, and the specific calculation manner is not limited.

In some embodiments, the determining that the play condition information and the receiving code rate of the client meet a preset upshift condition includes: the difference between the latest residual playable time length and the first estimated switching delay time length in the playing condition information is larger than or equal to a first preset threshold value, and the receiving code rate of the client is larger than the alternative code rate, wherein the gear of the alternative code rate is higher than the gear of the current code rate. The advantage of this is that by setting the preset upshift conditions reasonably, the timing of the upshift can be determined accurately, ensuring that a true upshift is only initiated if the network bandwidth supports the upshift.

The method includes that when the difference between the latest residual playable time length and the first estimated switching delay time length is greater than or equal to a first preset threshold value, the residual playable time length of the client can reach the minimum requirement of the capability of resisting network fluctuation even after the first estimated switching delay time length corresponding to the code rate gear switching, so that a certain upshift basis is provided. Therefore, when the above two conditions are satisfied, an upshift can be truly started.

In some embodiments, after the starting to transmit the probe redundancy packet and gradually increasing the duty ratio of the probe redundancy packet in the transmission data, the method further includes: and stopping sending the detection redundant packet when the residual transmittable data of the server side contained in the sending condition information is larger than a third preset threshold value. The advantage of this is that when the remaining transmittable data of the server is greater than a certain value, it indicates that there is backlog of some unsent data, at this time, the receiving code rate of the client may not support upshift, and the sending of the detection redundancy packet may be stopped, that is, the upshift detection is ended. The third preset threshold may be set according to practical situations, and may be 0, or other values greater than 0.

In some embodiments, the starting to transmit the probe redundancy packet and gradually increasing the duty ratio of the probe redundancy packet in transmitting data includes: acquiring a retransmission packet from a retransmission queue as a detection redundant packet to be sent; and after the second time length, acquiring a newly received video packet as a detection redundant packet to be sent, and gradually increasing the duty ratio of the detection redundant packet in the sending data in the sending process until the duty ratio reaches the upper limit of the redundancy rate. This has the advantage that the retransmission packet is preferentially transmitted as a detection redundancy packet, and the probability of successful reception of the retransmission packet can be improved while up-shift detection is achieved. In addition, the upper limit of the redundancy rate is set to control the duty ratio of the detection redundancy packet, so that the transmission of excessive detection redundancy packets is avoided, and excessive network bandwidth resources are occupied. The upper limit of the redundancy rate may be set according to practical situations, and may be 100%, 200%, or the like, for example.

In some embodiments, the redundancy rate upper limit is determined from a code rate difference between a current code rate gear and an adjacent gear higher than the current code rate gear. This has the advantage that the upper redundancy limit can be set reasonably. For example, the code rate between two gears is less than doubled, and can be set to 100%.

In the embodiment of the present invention, the play condition information, the receiving code rate of the client, and the transmission condition information are referred to when the downshift judgment and the upshift judgment are performed, but there is a certain difference in specific judgment strategies, the downshift judgment may be performed first and then the upshift judgment may be performed, or the upshift judgment may be performed first and then the downshift judgment may be performed, or of course, the downshift judgment and the upshift judgment may be performed simultaneously and concurrently, and the present invention is not limited in detail.

Fig. 7 is a flowchart of another method for controlling a bit rate of audio/video transmission according to an embodiment of the present invention, where the method mainly introduces steps related to upshift determination, as shown in fig. 7, and the method may include:

step 701, receiving play condition information fed back by a client, wherein the play condition information comprises a residual playable duration and a timestamp of a last play frame, and the residual playable duration comprises a duration to be played corresponding to audio and video data received and not played by the client.

Step 702, estimating a receiving code rate of the client according to the playing condition information.

Step 703, determining transmission condition information of the audio/video data.

And step 704, when the transmission condition information meets a preset upshift detection condition, starting to transmit the detection redundancy packet and gradually increasing the duty ratio of the detection redundancy packet in the transmission data.

For example, the preset upshift detection condition may be expressed as: waitlen=0 and minBandwidth > n x maxsendstate.

Step 705, in the process of sending the detection redundancy packet, if it is determined that the play condition information and the receiving code rate of the client meet the preset upshift condition, stopping sending the detection redundancy packet, and improving the code rate gear of the audio/video data transmission.

Wherein the preset upshift condition may be expressed as: jitter len-delay > =jitter thr, and recvCodeRate > nextCodeRate. That is, in adding the redundant packet, if recvCodeRate > nextCodeRate, if jitter len-delay > =jitter thr, transmission of the redundant packet is stopped and upshift is started. Wherein, nextCodeRate represents the next code rate gear higher than the current code rate gear.

According to the code rate control method for audio and video transmission, provided by the embodiment of the invention, comprehensive evaluation is performed by referring to information of multiple dimensions, the code rate upshift of audio and video transmission can be controlled more reasonably, and before upshift, the mode of sending a detection redundancy packet is used for simulating upshift, so that whether the current network reaches the upshift condition is confirmed, the purpose of preventing misjudgment is achieved, and the problem of blocking caused by network congestion due to blind upshift is prevented.

Fig. 8 is a flow chart of another method for controlling a code rate of audio/video transmission according to an embodiment of the present invention, as shown in fig. 8, the method may include:

step 801, receiving the play condition information fed back by the client, determining the transmission condition information of the audio and video data, and estimating the receiving code rate of the client and the estimated residual playable duration corresponding to the first data receiving time of the client according to the play condition information.

Step 802, judging whether the sent duration of the current picture group is less than or equal to a second preset threshold value, if yes, executing step 803; otherwise, step 806 is performed.

Step 803, determining the unsent data corresponding to the current code rate gear in the current picture group as first data to be sent, and determining a target residual playable duration corresponding to the client after the first duration to be sent passes from the first data receiving moment.

Step 804, judging whether the target remaining playable duration is smaller than a first preset threshold, if yes, executing step 805; otherwise, step 809 is performed.

Step 805, determining a corresponding first target code rate gear, and starting from a first frame of the current frame group to transmit audio/video data of the first target code rate gear.

The first target code rate gear is lower than the current code rate gear.

Step 806, determining the unsent data corresponding to the current code rate gear in the current picture group and the data corresponding to the current code rate gear in the next picture group as first data to be sent, and determining a target remaining playable duration corresponding to the client after the first duration to be sent passes from the first data receiving moment.

Step 807, judging whether the target remaining playable duration is smaller than a first preset threshold, if yes, executing step 808; otherwise, step 809 is performed.

Step 808, determining a corresponding second target code rate gear, continuing to transmit the audio/video data of the current code rate gear of the current picture group, and starting from the first frame of the next picture group to transmit the audio/video data of the second target code rate gear.

Step 809, judging whether the transmission condition information meets the preset upshift detection condition, if yes, executing step 810; otherwise, step 801 is performed.

Step 810, starting to transmit the probe redundancy packet and gradually increasing the duty ratio of the probe redundancy packet in the transmission data.

Step 811, judging whether the play condition information and the receiving code rate of the client meet the preset upshift condition, if yes, executing step 812; otherwise, step 801 is performed.

Step 812, stopping sending the detection redundant packet, and improving the code rate gear of the audio and video data transmission, and executing step 801.

For example, when the code rate gear is lifted, the switching can be performed from the next picture group, because for the upshift condition, no stuck condition occurs, and in order to avoid occupation of network bandwidth resources, the switching can be performed after the data transmission of the current picture group is completed. Optionally, the step of improving the code rate of the audio and video data transmission may be that the audio and video data of the code rate step after the step is improved from the first frame of the next frame group.

In summary, according to the audio/video transmission rate control method provided by the embodiment of the invention, the actual bandwidth of the current network can be estimated more accurately through comprehensive evaluation of a plurality of network indexes such as the buffer time of the client player, the sending condition of the current GOP, the current receiving rate and the like, and meanwhile, whether the current network reaches an upshift condition is further confirmed by using redundant packet detection so as to achieve the purpose of preventing erroneous judgment.

Fig. 9 is a block diagram of a code rate control device for audio/video transmission according to an embodiment of the present invention, where the device may be implemented by software and/or hardware, and may be generally integrated in a computer device, and may perform code rate control by executing a code rate control method for audio/video transmission. As shown in fig. 9, the apparatus includes:

the play situation information receiving module 901 is configured to receive play situation information fed back by a client, where the play situation information includes a remaining playable duration and a timestamp of a last play frame, and the remaining playable duration includes a duration to be played corresponding to audio and video data that has been received and not played by the client;

a receiving code rate estimating module 902, configured to estimate a receiving code rate of the client according to the play condition information;

a transmission condition information determining module 903, configured to determine transmission condition information of the audio and video data;

and the code rate control module 904 is configured to determine a code rate control manner of audio and video data transmission according to the play condition information, the receiving code rate of the client and the sending condition information.

The code rate control device for audio and video transmission provided by the embodiment of the invention is applied to a server side in an audio and video transmission system, receives play condition information fed back by a client side, wherein the play condition information comprises residual playable time length and a time stamp of a last play frame, the residual playable time length comprises time length to be played corresponding to audio and video data which is received and not played by the client side, estimates the receiving code rate of the client side according to the play condition information, determines the sending condition information of the audio and video data, and determines a code rate control mode for the audio and video data transmission according to the play condition information, the receiving code rate of the client side and the sending condition information. By adopting the technical scheme, the comprehensive evaluation is performed by referring to the information of multiple dimensions, so that the code rate of audio and video transmission can be more reasonably controlled, and further the fluency of audio and video playing and the quality of audio and video playing are better considered.

The embodiment of the invention provides a computer device, and the code rate control device for audio and video transmission provided by the embodiment of the invention can be integrated in the computer device. Fig. 10 is a block diagram of a computer device according to an embodiment of the present invention. The computer device 1000 comprises a memory 1001, a processor 1002 and a computer program stored in the memory 1001 and capable of running on the processor 1002, wherein the processor 1002 implements the method for controlling the rate of audio/video transmission provided by the embodiment of the invention when executing the computer program.

The embodiment of the invention also provides a storage medium containing computer executable instructions which are used for executing the code rate control method of the audio/video transmission provided by the embodiment of the invention when being executed by a computer processor.

The device, the equipment and the storage medium for controlling the code rate of the audio and video transmission provided by the embodiment can execute the method for controlling the code rate of the audio and video transmission provided by any embodiment of the invention, and have the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the above embodiments may be referred to the rate control method for audio/video transmission provided in any embodiment of the present invention.

Note that the above is only a preferred embodiment of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.

Claims

1. The code rate control method for audio and video transmission is characterized by being applied to a server side in an audio and video transmission system, and comprises the following steps:

determining the sending condition information of the audio and video data;

Determining a code rate control mode of audio and video data transmission according to the play condition information, the receiving code rate of the client and the sending condition information;

the method for determining the rate control mode of audio and video data transmission according to the play condition information, the receiving rate of the client and the sending condition information comprises the following steps:

estimating estimated residual playable duration corresponding to a first data receiving time of the client according to the play condition information, wherein the first data receiving time comprises a time when currently transmitted data is expected to be received at the client;

determining first data to be transmitted according to the transmission condition information;

determining playable duration corresponding to the first data to be sent, and determining the first duration corresponding to the first data to be sent according to the receiving code rate of the client;

determining a target residual playable duration corresponding to the client after the first duration to be sent is passed according to the estimated residual playable duration and the playable duration;

and when the target residual playable duration is smaller than a first preset threshold value, determining to reduce the code rate gear of the audio and video data transmission.

2. The method of claim 1, wherein the estimating, according to the play condition information, an estimated remaining playable duration corresponding to the first data receiving time of the client includes:

Determining the maximum round trip delay in a first preset historical period according to the play condition information;

calculating a first estimated residual playable duration according to the latest residual playable duration, the maximum round trip delay, the latest round trip delay, the receiving code rate of the client and the code rate of the current code rate gear in the play condition information; and/or calculating a second estimated residual playable duration according to the timestamp of the last transmitted frame, the timestamp of the latest last played frame in the play condition information and the maximum round trip delay;

and determining the first estimated remaining playable duration, the second estimated remaining playable duration or the minimum value of the first estimated remaining playable duration and the second estimated remaining playable duration as the estimated remaining playable duration corresponding to the first data receiving time of the client.

3. The method of claim 1, wherein the transmission case information includes a transmitted time length of a current picture group;

the determining the first data to be transmitted according to the transmission condition information comprises the following steps:

when the transmitted duration is smaller than or equal to a second preset threshold value, the first data to be transmitted comprises the non-transmitted data corresponding to the current code rate gear in the current picture group; when the transmitted time length is greater than the second preset threshold value, the first data to be transmitted comprises the unsent data corresponding to the current code rate gear in the current picture group and the data corresponding to the current code rate gear in the next picture group.

4. The method of claim 1, wherein the first preset threshold is determined by:

determining the maximum residual playable duration in a second preset history period according to the play condition information;

determining a first residual multicast duration threshold according to the product of the maximum residual multicast duration and a preset specific gravity value;

and determining the minimum value of the first residual multicast duration threshold and the preset residual multicast duration threshold as a first preset threshold.

5. The method of claim 3, wherein when the transmitted duration is less than or equal to the second preset threshold, after determining to reduce the code rate gear of the audio-video data transmission, further comprising:

determining a second to-be-transmitted duration corresponding to second to-be-transmitted data according to the receiving code rate of the client, wherein the second to-be-transmitted data comprises data corresponding to a candidate code rate in a current picture group, and the candidate code rate is lower than the code rate of the current code rate gear;

determining candidate residual playable duration corresponding to the client after the second waiting time duration passes according to the estimated residual playable duration;

determining a candidate residual playable duration which is greater than or equal to the first preset threshold value as a first candidate residual playable duration, and determining a corresponding first target code rate gear according to the first candidate residual playable duration;

And starting to transmit the audio and video data of the first target code rate gear from the first frame of the current picture group.

6. The method of claim 5, wherein the determining a corresponding first target code rate gear based on the first candidate remaining playable time length comprises:

and determining the highest code rate gear in the candidate code rates corresponding to the first candidate residual playable duration as a first target code rate gear.

7. The method of claim 3, further comprising, after said determining to reduce the rate shift of the transmission of the audiovisual data when said transmitted time period is greater than said second preset threshold value:

judging whether the residual transmittable data of the server side in the sending condition information contains data corresponding to the current code rate gear in the next picture group or not, wherein the residual transmittable data comprises audio and video data which are received and not sent in the server side;

determining a second target code rate gear by adopting a corresponding code rate reduction strategy according to the judging result;

and continuing to transmit the audio and video data of the current code rate gear of the current picture group, and starting from the first frame of the next picture group to transmit the audio and video data of the second target code rate gear.

8. The method of claim 7, wherein the determining the second target code rate gear using the corresponding code rate reduction strategy according to the determination result comprises:

determining a third to-be-transmitted duration corresponding to third to-be-transmitted data according to a receiving code rate of the client by adopting a corresponding determining mode according to a judging result, wherein the third to-be-transmitted data comprises non-transmitted data corresponding to a current code rate gear in the current picture group and data corresponding to a candidate code rate in a next picture group, and the candidate code rate is lower than the code rate of the current code rate gear;

determining candidate residual playable duration corresponding to the client after the third waiting time duration passes according to the estimated residual playable duration;

and determining the candidate residual playable duration which is greater than or equal to the first preset threshold value as a second candidate residual playable duration, and determining a corresponding second target code rate gear according to the second candidate residual playable duration.

9. The method of claim 8, wherein when the remaining transmittable data of the server does not include data corresponding to the current code rate gear in the next frame group, determining, according to the determination result, a third duration to be transmitted corresponding to the third data to be transmitted according to the receiving code rate of the client by using a corresponding determination method, includes:

Calculating a fourth time length to be transmitted of the unsent data corresponding to the current code rate gear in the current picture group according to the receiving code rate of the client;

determining the sum of a first receiving duration and a first estimated switching delay duration of unreceived data corresponding to a current code rate gear in the current picture group as a fifth duration to be transmitted;

calculating a sixth time length to be transmitted according to the candidate code rate, the receiving code rate of the client and the time length of the next picture group;

and determining the sum of the maximum value of the fourth to-be-sent time length and the fifth to-be-sent time length and the sixth to-be-sent time length as a third to-be-sent time length corresponding to third to-be-sent data.

10. The method of claim 8, wherein when the remaining transmittable data of the server already includes data corresponding to a current code rate gear in a next frame group, determining, according to the determination result, a third duration to be transmitted corresponding to the third data to be transmitted according to the receiving code rate of the client by adopting a corresponding determination method, includes:

Determining a difference between a first estimated switching delay time and a first time length as a seventh time length to be transmitted, wherein the first time length is a difference between a transmission time length corresponding to the residual transmittable data of the server and the fourth time length to be transmitted;

and determining the sum of the maximum value of the fourth to-be-sent time length and the seventh to-be-sent time length and the sixth to-be-sent time length as a third to-be-sent time length corresponding to third to-be-sent data.

11. The method according to any one of claims 1-10, wherein determining a rate control manner of audio/video data transmission according to the play condition information, the receiving rate of the client, and the sending condition information includes:

when the transmission condition information meets a preset upshift detection condition, starting to transmit a detection redundant packet and gradually increasing the duty ratio of the detection redundant packet in transmission data;

and in the process of sending the detection redundant packet, if the play condition information and the receiving code rate of the client side are determined to meet the preset upshift condition, stopping sending the detection redundant packet, and improving the code rate gear of audio and video data transmission.

12. The method according to claim 11, wherein the transmission case information satisfies a preset upshift detection condition, comprising:

and the residual transmittable data of the server side contained in the transmission condition information is smaller than or equal to a third preset threshold value, and the minimum bandwidth calculated by the transmission window is larger than the product of a preset coefficient and the maximum transmission rate.

13. The method of claim 11, wherein the determining that the play condition information and the receiving code rate of the client meet a preset upshift condition comprises:

the difference between the latest residual playable time length and the first estimated switching delay time length in the playing condition information is larger than or equal to a first preset threshold value, and the receiving code rate of the client is larger than the alternative code rate, wherein the gear of the alternative code rate is higher than the gear of the current code rate.

14. The method of claim 12, further comprising, after said starting to transmit the probe redundancy packet and gradually increasing the duty cycle of the probe redundancy packet in transmitting data:

and stopping sending the detection redundant packet when the residual transmittable data of the server side contained in the sending condition information is larger than a third preset threshold value.

15. The method of claim 11, wherein starting transmission of the probe redundancy packet and gradually increasing the duty cycle of the probe redundancy packet in the transmission data comprises:

acquiring a retransmission packet from a retransmission queue as a detection redundant packet to be sent;

and after the second time length, acquiring a newly received video packet as a detection redundant packet to be sent, and gradually increasing the duty ratio of the detection redundant packet in the sending data in the sending process until the duty ratio reaches the upper limit of the redundancy rate.

16. The code rate control method for audio and video transmission is characterized by being applied to a server side in an audio and video transmission system, and comprises the following steps:

determining the sending condition information of the audio and video data;

17. The utility model provides a code rate controlling means of audio-video transmission which characterized in that disposes the server in audio-video transmission system, said device includes:

the code rate control module is used for determining a code rate control mode of audio and video data transmission according to the play condition information, the receiving code rate of the client and the sending condition information;

18. The utility model provides a code rate controlling means of audio-video transmission which characterized in that disposes the server in audio-video transmission system, said device includes:

19. A rate control device for audio-video transmission, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any one of claims 1-15 or 16 when executing the computer program.

20. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method of any one of claims 1-15 or claim 16.