CN113271316B

CN113271316B - Multimedia data transmission control method and device, storage medium and electronic equipment

Info

Publication number: CN113271316B
Application number: CN202110645772.0A
Authority: CN
Inventors: 罗杰; 王迅
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-06-09
Filing date: 2021-06-09
Publication date: 2022-09-13
Anticipated expiration: 2041-06-09
Also published as: CN113271316A

Abstract

The invention discloses a multimedia data transmission control method and device, a storage medium and electronic equipment. Wherein, the method comprises the following steps: the method comprises the steps of transmitting a first data packet set to a proxy server of a second transmission protocol through the second transmission protocol according to a current packet transmission interval, determining a network congestion state between a target application and the proxy server according to a target transmission parameter when the target application transmits the first data packet set to the proxy server, and adjusting the current packet transmission interval to be the target packet transmission interval according to the network congestion state. The invention solves the technical problem of unstable multimedia data transmission in the poor network environment in the related technology.

Description

Multimedia data transmission control method and device, storage medium and electronic equipment

Technical Field

The present invention relates to the field of computers, and in particular, to a method and an apparatus for controlling transmission of multimedia data, a storage medium, and an electronic device.

Background

With the development of mobile networks, the application of live webcasting has been developed rapidly in recent years, such as live game broadcasting, live entertainment broadcasting, live sporting events, online education, online meeting and the like. However, due to a complex network environment, especially a weak network environment (which may also be called a poor network environment), such as scenes where bandwidth is insufficient, delay is high, jitter is frequent, and the like, problems such as high delay and serious frame drop may occur in a push stream in a live broadcast process (where the push stream refers to a process of transmitting multimedia data (e.g., audio and video data) to a server), which further affects viewing experience of viewers, such as frame skipping and pause in a live broadcast picture.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

The embodiment of the invention provides a multimedia data transmission control method and device, a storage medium and electronic equipment, which are used for at least solving the technical problem of unstable multimedia data transmission in a relatively poor network environment in the related art.

According to an aspect of an embodiment of the present invention, there is provided a method for controlling transmission of multimedia data, including: coding first multimedia information acquired by target application according to the current code rate to obtain a first multimedia data set; encapsulating the multimedia data in the first multimedia data set into a data packet of a first transmission protocol to obtain a first data packet set; according to the current packet sending interval, transmitting the first data packet set to a proxy server of a second transmission protocol through the second transmission protocol, wherein the proxy server is used for transmitting the first data packet set to a target server of the first transmission protocol, the second transmission protocol is a data transmission protocol based on a User Datagram Protocol (UDP), and the value of the packet sending interval parameter is allowed to be dynamically adjusted; determining a network congestion state between the target application and the proxy server according to a target transmission parameter when the target application transmits the first data packet set to the proxy server; and adjusting the current packet sending interval to be a target packet sending interval according to the network congestion state, wherein the target application is used for sending the data packet of the first transmission protocol to the proxy server according to the target packet sending interval.

According to another aspect of the embodiments of the present invention, there is also provided a transmission control apparatus for multimedia data, including: the encoding module is used for encoding the first multimedia information acquired by the target application according to the current code rate to obtain a first multimedia data set; the encapsulation module is used for encapsulating the multimedia data in the first multimedia data set into a data packet of a first transmission protocol to obtain a first data packet set; a transmission module, configured to transmit the first data packet set to a proxy server of a second transmission protocol through the second transmission protocol according to a current packet sending interval, where the proxy server is configured to transmit the first data packet set to a target server of the first transmission protocol, the second transmission protocol is a data transmission protocol based on a user datagram protocol UDP, and a value of a packet sending interval parameter is allowed to be dynamically adjusted; a determining module, configured to determine a network congestion state between the target application and the proxy server according to a target transmission parameter when the target application transmits the first data packet set to the proxy server; and an adjusting module, configured to adjust the current packet sending interval to a target packet sending interval according to the network congestion state, where the target application is configured to send the data packet of the first transport protocol to the proxy server according to the target packet sending interval.

Optionally, the determining module includes: a first obtaining unit, configured to obtain M groups of transmission parameters, where each group of transmission parameters includes transmission parameters counted N times within a preset time duration, and each counted transmission parameter includes a round-trip delay and a sending bandwidth, where the round-trip delay is a delay between a sending time and a receiving time, the sending time is a time when the target application sends the data packet to the proxy server, the receiving time is a time when the target application receives acknowledgement information of the data packet sent by the proxy server, the sending bandwidth is a bandwidth when the target application sends the data packet to the proxy server, where M is 1 or M is greater than or equal to 2, N is 1 or N is greater than or equal to 2; a second obtaining unit, configured to obtain a minimum value of the round trip delay and a maximum value of the sending bandwidth in each group of transmission parameters, and obtain a minimum value of the round trip delay and a maximum value of the sending bandwidth for M pairs; a determining unit, configured to determine, according to the minimum value of the round trip delay and the maximum value of the sending bandwidth, a network congestion state between the target application and the proxy server.

Optionally, the apparatus is configured to determine a network congestion state between the target application and the proxy server according to the minimum value of the round trip delay and the maximum value of the sending bandwidth by M: obtaining the product between the minimum value of each pair of round-trip delay and the maximum value of the sending bandwidth to obtain M bandwidth delay products; obtaining unconfirmed data volume within a preset time length corresponding to each group of transmission parameters, and obtaining M unconfirmed data volumes in total, wherein each unconfirmed data volume represents the data volume corresponding to the data packet which is not confirmed by the proxy server and is sent to the proxy server by the target application within the corresponding preset time length; determining a difference between the M bandwidth-delay products and the M unacknowledged data volumes; and determining the network congestion state between the target application and the proxy server according to the difference value.

Optionally, the apparatus is configured to determine a network congestion state between the target application and the proxy server according to the difference value by: when the difference value is larger than a first preset threshold value, determining the network congestion state as a congestion reduction state; and when the difference value is smaller than a second preset threshold value, determining the network congestion state as a congestion increasing state.

Optionally, the apparatus is configured to adjust the current packet sending interval to a target packet sending interval according to the network congestion status by: when the network congestion state is a congestion reduction state, reducing the current packet sending interval to the target packet sending interval; and when the network congestion state is a congestion increasing state, increasing the current packet sending interval to the target packet sending interval.

Optionally, the apparatus is configured to reduce the current packet transmission interval to the target packet transmission interval by: increasing a value of a preset gain coefficient to a first value, wherein the gain coefficient is used for determining a maximum value of a transmission bandwidth, and the transmission bandwidth is a bandwidth for the target application to transmit a data packet to the proxy server; determining the maximum value of the sending bandwidth according to the value of the gain coefficient and a predetermined receiving rate, wherein the value of the gain coefficient is positively correlated with the maximum value of the sending bandwidth, and the receiving rate is the rate of the proxy server receiving the data packet sent by the target application; and determining the value of a packet sending interval parameter as the target packet sending interval according to the maximum value of the sending bandwidth, wherein the value of the packet sending interval parameter is in negative correlation with the maximum value of the sending bandwidth, and the value of the packet sending interval parameter indicates the packet sending interval adopted when the target application transmits the data packet to the proxy server.

Optionally, the apparatus is configured to increase the current packet sending interval to the target packet sending interval by: reducing a value of a preset gain coefficient to a second value, wherein the gain coefficient is used for determining a maximum value of a transmission bandwidth, and the transmission bandwidth is a bandwidth for the target application to transmit a data packet to the proxy server; determining the maximum value of the sending bandwidth according to the value of the gain coefficient and a predetermined receiving rate, wherein the value of the gain coefficient is positively correlated with the maximum value of the sending bandwidth, and the receiving rate is the rate of the proxy server receiving the data packet sent by the target application; and determining the value of a packet sending interval parameter as the target packet sending interval according to the maximum value of the sending bandwidth, wherein the value of the packet sending interval parameter is in negative correlation with the maximum value of the sending bandwidth, and the value of the packet sending interval parameter indicates the packet sending interval adopted when the target application transmits the data packet to the proxy server.

Optionally, the apparatus is further configured to: after the current packet sending interval is adjusted to a target packet sending interval according to the network congestion state, coding second multimedia information acquired by the target application according to the current code rate to obtain a second multimedia data set; encapsulating the multimedia data in the second multimedia data set into a data packet of the first transmission protocol to obtain a second data packet set; and transmitting the second data packet set to the proxy server of the second transmission protocol through the second transmission protocol according to the target packet sending interval.

Optionally, the apparatus is further configured to: and adjusting the current code rate to a target code rate according to the network congestion state, wherein the target application is used for coding the multimedia information acquired by the target application according to the target code rate.

Optionally, the apparatus is configured to adjust the current bitrate to a target bitrate according to the network congestion status by: when the network congestion state is a congestion reduction state, increasing the current code rate to the target code rate; and when the network congestion state is a congestion increasing state, reducing the current code rate to the target code rate.

Optionally, the apparatus is further configured to: after the current code rate is adjusted to a target code rate according to the network congestion state, encoding third multimedia information acquired by the target application according to the target code rate to obtain a third multimedia data set; encapsulating the multimedia data in the third multimedia data set into a data packet of the first transmission protocol to obtain a third data packet set; and transmitting the third data packet set to the proxy server of the second transmission protocol through the second transmission protocol according to the target packet sending interval.

According to still another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the above-mentioned transmission control method of multimedia data when running.

According to still another aspect of the embodiments of the present invention, there is also provided an electronic device including a memory in which a computer program is stored and a processor configured to execute the above-mentioned transmission control method of multimedia data by the computer program.

In the embodiment of the invention, a bottom layer transmission mode is changed, namely, multimedia data is transmitted through a TCP protocol in the related technology and is adjusted to be transmitted through a UDP-based data transmission protocol, so that the problem of unstable push stream in poor network environment caused by the fact that the multimedia data is transmitted through the TCP protocol in the related technology is solved. Furthermore, in the embodiment of the present invention, a network congestion control algorithm is also changed, and a correlation coefficient is dynamically adjusted according to a network congestion degree, where the correlation coefficient is used to affect a packet sending interval of a data packet, so that when a network environment is poor (for example, bandwidth is insufficient), the packet sending interval can be reduced and a situation of network congestion can be alleviated by adjusting the correlation coefficient, thereby further solving a problem of unstable flow pushing when the network environment is poor. Furthermore, in the embodiment of the present invention, the bit rate of the multimedia data may be dynamically adjusted according to the network congestion degree, so that when the network environment is poor (for example, bandwidth is insufficient), the situation of network congestion is alleviated by reducing the bit rate of the multimedia data, and thus the problem of unstable push flow when the network environment is poor can be further solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:

fig. 1 is a schematic diagram of an application environment of an alternative multimedia data transmission control method according to an embodiment of the present invention;

fig. 2 is a flow chart illustrating an alternative method for controlling transmission of multimedia data according to an embodiment of the present invention;

fig. 3 is a diagram illustrating an alternative method for controlling transmission of multimedia data according to an embodiment of the present invention;

fig. 4 is a diagram illustrating still another alternative multimedia data transmission control method according to an embodiment of the present invention;

fig. 5 is a diagram illustrating still another alternative transmission control method of multimedia data according to an embodiment of the present invention;

fig. 6 is a diagram illustrating still another alternative multimedia data transmission control method according to an embodiment of the present invention;

fig. 7 is a diagram illustrating still another alternative multimedia data transmission control method according to an embodiment of the present invention;

fig. 8 is a diagram illustrating still another alternative transmission control method of multimedia data according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an alternative multimedia data transmission control device according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of an alternative electronic device according to an embodiment of the invention.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

First, partial nouns or terms appearing in the description of the embodiments of the present application are applicable to the following explanations:

and (4) live broadcast: the method refers to a technology that broadcasting data is collected through certain equipment, and the broadcasting data is compressed into a viewable and transmittable video stream through a series of processing such as video coding and is output to a watching user side.

Pushing flow: the method refers to a process of transmitting collected audio and video data to a server.

RTMP (Real-Time Messaging Protocol) Real-Time message transmission Protocol, which is based on TCP transmission and designed for Real-Time data communication, is mainly used for streaming audio and video data communication.

SRT (secure removable transport) is an open source Internet transport protocol based on a UDT protocol, the bottom layer is based on UDP transmission, and the transmission of network audio and video is mainly carried out with low delay.

Automatic Repeat request (arq), which is an automatic error correction mechanism in the network protocol transmission process.

Round Trip time (rtt) is a network performance indicator, which indicates the total time delay from the time when the transmitting end transmits data to the time when the transmitting end receives an acknowledgement from the receiving end (the receiving end immediately transmits an acknowledgement after receiving the data).

The bandwidth Delay product, which is a network performance indicator, represents the product of the capability of a data link (bits per second) and the round-trip communication Delay (seconds).

The invention is illustrated below with reference to examples:

according to an aspect of the embodiment of the present invention, a method for controlling transmission of multimedia data is provided, and optionally, in this embodiment, the method for controlling transmission of multimedia data may be applied to a hardware environment formed by a server 101 and a user terminal 103 as shown in fig. 1. As shown in fig. 1, a server 101 is connected to a terminal 103 through a network 105, and may be used to provide services for a user terminal or a client installed on the user terminal, where the client may be a video client, an instant messaging client, a browser client, an education client, a game client, and the like. The network 105 may include, but is not limited to: a wired network, a wireless network, wherein the wired network comprises: the internet, the wireless network comprising: bluetooth, WIFI and other networks implementing wireless communication, the user terminal 103 may be a terminal configured with a live application, and the live application may also be other applications that need to stably transmit multimedia information in real time, for example, a monitoring application, and the user terminal 103 may include but is not limited to at least one of the following: computer Devices such as Mobile phones (e.g., Android phone, iOS phone, etc.), notebook computers, tablet computers, palm computers, MID (Mobile Internet Devices), PAD, desktop computers, smart televisions, etc., the server 101 may be a single server, or a server cluster composed of a plurality of servers, or a cloud server, specifically, a proxy server and a target server, which may be deployed on the same machine or established as a local deployment, it will be appreciated that the above-described process of data transfer between the proxy server and the target server is stable, low-latency, and the user terminal 103 establishes an internet connection with the server 101 through the network 105, and the target application 107 using the above-described transmission control method of multimedia data is displayed through the user terminal 103.

As shown in fig. 1, the method for controlling transmission of multimedia data can be implemented in the user terminal 103 by the following steps:

s1, starting the target application 107 in the user terminal 103, and encoding the first multimedia information acquired by the target application 107 according to the current code rate to obtain a first multimedia data set;

s2, encapsulating the multimedia data in the first multimedia data set into a data packet of the first transmission protocol in the target application 107 of the user terminal 103, so as to obtain a first data packet set;

s3, in the target application 107 of the user terminal 103, according to the current packet sending interval, transmitting the first data packet set to a proxy server of a second transmission protocol (corresponding to the server 101) through the second transmission protocol, where the proxy server is configured to transmit the first data packet set to the target server of the first transmission protocol, the second transmission protocol is a data transmission protocol based on the UDP, and a value of the packet sending interval parameter is allowed to be dynamically adjusted;

s4, determining, in the target application 107 of the user terminal 103, a network congestion state between the target application and the proxy server according to the target transmission parameter when the target application 107 transmits the first data packet set to the proxy server;

s5, in the target application 107 of the user terminal 103, according to the network congestion state, adjusting the current packet sending interval to a target packet sending interval, where the target application is configured to send the data packet of the first transmission protocol to the proxy server according to the target packet sending interval.

Optionally, in this embodiment, the transmission control method of the multimedia data may also be used by a client configured in a server, but not limited to.

The above is only an example, and the present embodiment is not particularly limited.

Optionally, as an optional implementation manner, as shown in fig. 2, the method for controlling transmission of multimedia data includes:

s202, encoding first multimedia information acquired by target application according to the current code rate to obtain a first multimedia data set;

optionally, in this embodiment, the code rate may include, but is not limited to, a bit rate, which refers to a number of bits (bits) transmitted per second. The unit is bps (bit per second) and b/s, and the higher the bit rate, the larger the amount of data (number of bits) transferred per unit time. Information in a computer is represented by binary 0 and 1, where each 0 or 1 is called a bit, and is represented by a lower case b, i.e., bit. The capital B indicates byte, i.e., byte, and 1 byte-8 bits, i.e., 1B-8B. The size unit of a file is generally expressed by using Kilobytes (KB).

For example, a video file in RMVB contains video information and audio information, and both audio and video have different sampling modes and bit rates, that is, the audio and video bit rates of the same video file are not the same. What we speak of a video file codestream size generally refers to the sum of the codestreams of audio and video information in a video file.

Taking an RMVB video file as an example, VB in RMVB refers to VBR, which is an abbreviation of Variable BitRate, and chinese means Variable bit rate, which means that RMVB adopts a dynamic coding mode, and a higher sampling rate is used for complex dynamic pictures (singing and dancing, flying, war, action, and the like), and a lower sampling rate is used for static pictures, so that resources are reasonably utilized, and the effect that picture quality and volume can be achieved at the same time is achieved.

Optionally, in this embodiment, the first multimedia information may include, but is not limited to, a combination of multiple media, generally including multiple media forms such as text, sound, and image, and may include, but is not limited to, one or a combination of multiple media forms.

It should be noted that, in the case of a man-machine interactive information exchange and transmission medium including two or more media, the media used include characters, pictures, photos, sounds, animations and movies, and other interactive functions provided by programs.

Optionally, in this embodiment, the encoding operation may include, but is not limited to, a manner of converting a file in a certain video format into a file in another video format by using a specific compression technique.

Specifically, the codec standards that are most important in video streaming include, but are not limited to, h.261 and h.263 of the international telecommunication union, M-JPEG of the moving picture experts group and MPEG series standards of the moving picture experts group of the international organization for standardization, and Real video of Real-Networks, WMV of microsoft corporation, QuickTime of Apple corporation, and the like, which are widely used on the internet.

MPEG is an abbreviation of Moving Picture experts Group (Moving Picture extensions Group), and has been an organization that has now established the standard of "Moving Picture and audio coding" in response to different application requirements, without the restriction of "for digital storage media". The various standards established by the MPEG organization have different goals and applications, and the MPEG-1, MPEG-2, MPEG-4, MPEG-7, and MPEG-21 standards have been proposed.

In this embodiment, the encoding is generally to encode the multimedia data collected by the target application for subsequent multimedia data processing, for example, multimedia data transmission.

S204, encapsulating the multimedia data in the first multimedia data set into a data packet of a first transmission protocol to obtain a first data packet set;

optionally, in this embodiment, the first transmission Protocol may include, but is not limited to, a Real-Time Messaging Protocol (RTMP), a Real-Time Messaging Protocol (rtp), which is designed for Real-Time data communication based on TCP transmission and is mainly used for streaming audio/video data communication, and a specific encapsulation process is not described herein again.

S206, transmitting the first data packet set to a proxy server of a second transmission protocol through the second transmission protocol according to the current packet transmission interval, wherein the proxy server is used for transmitting the first data packet set to a target server of the first transmission protocol, the second transmission protocol is a data transmission protocol based on a User Datagram Protocol (UDP), and the value of a packet transmission interval parameter is allowed to be dynamically adjusted;

optionally, in this embodiment, the current packet sending interval may include, but is not limited to, being flexibly preconfigured by a worker or a user according to an actual service scenario, and may also include, but is not limited to, performing matching according to a current network state, for example, dividing a round trip delay of the network into a plurality of intervals, and selecting a corresponding current packet sending interval by referring to the plurality of intervals by detecting the round trip delay of the current network state.

Optionally, in this embodiment, the second transmission protocol includes, but is not limited to, a data transmission protocol based on a user datagram protocol UDP, for example, srt (secure reusable transport) is an open source internet transmission protocol based on a UDT protocol, the underlying layer is based on UDP transmission, and mainly low-latency network audio and video transmission.

Optionally, in this embodiment, the proxy server may include, but is not limited to, proxy a network user to obtain network information. The network information transfer station is a transfer station of network information, is an intermediate agency between a personal network and an Internet service provider, and is responsible for transferring legal network information and controlling and registering the transfer.

The proxy server, as a bridge connecting the Internet and the Intranet, can be used for multiple purposes in practical applications, and the most basic functions are connection, and in addition, the functions include security, caching, content filtering, access control management and the like. More importantly, the proxy server is an important security function provided by the Internet link level gateway, which works mainly at the session layer of the Open Systems Interconnection (OSI) model.

The proxy server is an intermediate program of both the server and the client, and is mainly used for forwarding the network access request of the client system. However, rather than simply forwarding the request to the actual internet server, the proxy server may also control the user's behavior, make decisions on the received client requests, and filter the user requests according to filtering rules.

In this embodiment, the proxy server is configured to transmit the first set of data packets to a target server of a first transport protocol, where the target server is a server that needs to transmit multimedia data for the target application, for example, a distribution server of multimedia information, and may specifically include, but is not limited to, an RTMP server and the like.

Optionally, in this embodiment, the value of the packet sending interval parameter allows dynamic adjustment, which may include but is not limited to dynamic adjustment according to parameters such as a current network state and a multimedia data amount to be transmitted.

S208, determining a network congestion state between the target application and the proxy server according to the target transmission parameters when the target application transmits the first data packet set to the proxy server;

optionally, in this embodiment, the target transmission parameters may include, but are not limited to, parameters such as a load average value, a receiving rate of the proxy server, a maximum value of a transmission bandwidth of the target application, a maximum value of a detection bandwidth, and RTT round trip delay.

Optionally, in this embodiment, the network congestion status may include, but is not limited to, a network status when the target transmission parameter is greater than or less than a predetermined threshold, for example, a duration of transmission of the first data packet to the proxy server by the target application is greater than a predetermined duration, or a receiving rate of the proxy server is less than a predetermined threshold.

The above is merely an example, and the present embodiment is not limited in any way.

S210, according to the network congestion state, adjusting the current packet sending interval to be a target packet sending interval, wherein the target application is used for sending the data packet of the first transmission protocol to the proxy server according to the target packet sending interval.

Optionally, in this embodiment, the target packet sending interval is a packet sending interval obtained by performing dynamic adjustment on the basis of the current packet sending interval according to the network congestion state. For example, when the current network state is in the network congestion state, the current packet transmission interval is decreased to the target packet transmission interval, and when the current network state is in the network clear state, the current packet transmission interval is increased to the target packet transmission interval.

Optionally, in this embodiment, the application scenario of the transmission control of the multimedia data may include, but is not limited to, a plurality of application scenarios such as medical, financial, credit, banking, government, energy, education, security, building, game, traffic, internet of things, industry, and the target application may include, but is not limited to, an application program applied in the application scenario.

Optionally, in this embodiment, taking a virtual live broadcast scene as an example, the method may specifically include, but is not limited to, the following steps:

s1, collecting audio and video information by the live broadcast application;

s2, the live broadcast application adopts code rate (capable of being dynamically adjusted) to encode the audio and video information to obtain audio and video data;

s3, encapsulating the audio and video data to obtain an RTMP data packet (wherein RTMP is only one example);

s4, sending the RTMP data packets to the proxy server of the SRT via the UDP based SRT protocol (wherein the SRT protocol is only an example and may be another UDP based protocol, e.g., QUIC) at packet sending intervals (which may be dynamically adjusted);

and S5, the SRT proxy server transmits the RTMP data packet to the RTMP server through a TCP protocol.

The SRT proxy server and the RTMP server are usually deployed on the same machine, so that a TCP protocol is adopted between the SRT proxy server and the RTMP server to transmit data packets, and the condition of network instability cannot occur.

For example, taking a virtual live broadcast scene as an example, fig. 3 is a schematic diagram of an optional multimedia data transmission control method according to an embodiment of the present invention, as shown in fig. 3, including: the system comprises a user terminal A and a server B, wherein the user terminal A is provided with a live broadcast application, the server B comprises an SRT server (corresponding to the proxy server) and an RTMP server (corresponding to the target server), the SRT protocol corresponds to the first transmission protocol, and the TCP protocol corresponds to the second transmission protocol.

S1, the terminal user A collects the audio and video information through the live broadcast application installed on the user terminal A;

s2, in the live broadcast application, the audio and video information is coded according to the (dynamic) code rate to obtain audio and video data;

s3, packaging the audio and video data into an RTMP data packet in the live broadcast application;

s4, based on SRT protocol, sending RTMO data packet to SRT server according to (dynamic) packet sending interval;

and S5, sending the RTMP data packet to the RTMP server on the SRT server according to the TCP protocol.

It should be noted that, when the network status of the user terminal a and the server B is good, the data packet may also be directly transmitted to the RTMP server through the TCP protocol.

For another example, taking a security monitoring scenario as an example, fig. 4 is a schematic diagram of another optional multimedia data transmission control method according to an embodiment of the present invention, and as shown in fig. 4, the method includes: the system comprises a user terminal A and a server B, wherein the user terminal A is provided with a monitoring application, the server B comprises a QUIC server (corresponding to the proxy server) and an RTMP server (corresponding to the target server), a QUIC protocol corresponds to the first transmission protocol, and a TCP protocol corresponds to the second transmission protocol.

s2, encoding the audio and video information according to the (dynamic) code rate in the live broadcast application to obtain audio and video data;

s4, sending RTMO data packets to the QUIC server according to the (dynamic) packet sending interval based on the QUIC protocol;

and S5, sending the RTMP data packet to the RTMP server on the QUIC server according to the TCP protocol.

Optionally, in this embodiment, fig. 5 is a schematic diagram of another optional multimedia data transmission control method according to an embodiment of the present invention, and as shown in fig. 5, in the prior art, transmission control of multimedia data is generally implemented through a conventional RTMP stream pushing process.

Wherein, the plug flow process before improvement is as follows:

the application layer (Client) encapsulates the collected audio and video data (corresponding to the aforementioned multimedia data) into an RTMP data packet format, and then transmits the data to an RTMP server (corresponding to the aforementioned target server) through a TCP protocol (corresponding to the aforementioned first transmission protocol).

In this embodiment, the transmission control of the multimedia data is realized by changing the underlying transmission mode, as shown in fig. 5, so that the improved stream pushing process is implemented.

Wherein, the improved plug flow process is as follows:

the method comprises the steps that an application layer (Client) encapsulates collected audio and video data (corresponding to the multimedia data) into an RTMP data packet format, then the data are transmitted to an SRT proxy server through an SRT protocol (corresponding to the second transmission protocol), the proxy server receives the data and then transmits the RTMP data packet to the RTMP server through a TCP (transmission control protocol), wherein the SRT is used for reliable UDP transmission to replace the original TCP.

Fig. 6 is a schematic diagram of another alternative multimedia data transmission control method according to an embodiment of the present invention, and as shown in fig. 6, the multimedia data transmission control method may include, but is not limited to, the following steps:

s1-1, collecting sound in a target application;

s1-2, collecting pictures in the target application;

s2-1, applying the coded audio at the target;

s2-2, applying the coded video at the target;

s3, packaging the coded audio and video into audio and video information in the target application;

s4, packaging the audio and video information into an RTMP data packet in the target application;

s5, transmitting the RTMP data packet to the SRT server through the SRT network in the target application;

s6, transferring the RTMP data packet to the RTMP server in the SRT server;

and S7, receiving and distributing the RTMP data packet at the RTMP server.

With this embodiment, the first set of data packets is transmitted to the proxy server of the second transmission protocol via the second transmission protocol according to the current packet transmission interval, determining a network congestion status between the target application and the proxy server based on target transmission parameters when the target application transmits the first set of data packets to the proxy server, controlling the multimedia data to be transmitted at the proxy server by a UDP-based data transmission protocol in a manner of adjusting the current packet transmission interval to a target packet transmission interval according to the congestion state of the network, so as to realize flexible adjustment of packet transmission interval according to the network congestion state between the target application and the proxy server, achieve the purpose of stably transmitting multimedia data, thereby realizing the technical effects of improving the transmission stability of the multimedia data and optimizing the transmission efficiency of the multimedia data, and the technical problem of unstable multimedia data transmission in a relatively poor network environment in the related technology is solved.

As an optional solution, determining a network congestion state between the target application and the proxy server according to a target transmission parameter when the target application transmits the first set of data packets to the proxy server, includes:

acquiring M groups of transmission parameters, wherein each group of transmission parameters comprises transmission parameters which are counted for N times within a preset time length, the transmission parameters counted each time comprise round-trip delay and sending bandwidth, the round-trip delay is the delay between the sending time and the receiving time, the sending time is the time when the target application sends a data packet to the proxy server, the receiving time is the time when the target application receives confirmation information of the data packet sent by the proxy server, the sending bandwidth is the bandwidth when the target application sends the data packet to the proxy server, M is 1 or M is more than or equal to 2, and N is 1 or N is more than or equal to 2;

obtaining the minimum value of the round-trip delay and the maximum value of the sending bandwidth in each group of transmission parameters, and obtaining the minimum value of M pairs of round-trip delay and the maximum value of the sending bandwidth;

and determining the network congestion state between the target application and the proxy server according to the minimum value of the M pairs of round trip delay and the maximum value of the sending bandwidth.

Optionally, in this embodiment, the network congestion state between the target application and the proxy server may include, but is not limited to, determining by dynamically calculating a current packet sending interval, where a specific formula is as follows:

sendPeriod＝avgPayloadSize/sendMaxBW；

wherein, the avgPayloadSize is an average value of the load of the sending packet (the amount of data received within a certain time period counted by the receiving end), and is calculated in an iir (infinite Impulse response) weighting manner, and the formula is as follows:

curAvg＝avgPayloadSize＝(lastAvg*(N-1)+curPktSize)/N；

wherein, N represents the number of times of statistics, PktSize represents the size of a data packet, Avg represents the average value in a preset time length, and cur represents a function pointer.

For example, with N as the number of windows, avg1 of the 1 st packet is pkt1, avg2 of the 2 nd packet is (avg1 (N-1) + pkt2)/N, and so on, where N is a fixed value, it can be understood that the last average weight is (N-1)/N, and the weight of the current packet is 1/N.

Optionally, in this embodiment, the method may include, but is not limited to, obtaining a minimum value of the round trip delay and a maximum value of the transmission bandwidth in each set of transmission parameters, and obtaining a minimum value of M pairs of round trip delays and a maximum value of the transmission bandwidth in the following manners:

s1, recording minimum value (minRTT) (corresponding to the minimum value of the aforementioned round-trip delay) of total delay experienced by N times of RTT (round-trip delay, which indicates the total delay from the time when the transmitting end transmits data to the time when the transmitting end receives an acknowledgement from the receiving end, wherein the receiving end immediately transmits an acknowledgement after receiving the data) in each round of M recent rounds;

s2, M recent rounds, each round records the maximum value (maxBW) sendMaxBW (corresponding to the maximum value of the transmission bandwidth) among N transmission bandwidths.

The maximum value sendMaxBW of the transmission bandwidth may include, but is not limited to, being implemented by the following formula:

the receiving rate at the receiving end is multiplied by a fixed gain coefficient, and the formula is as follows:

sendMaxBW＝recvBW*(100+overhead)/100

wherein, recvBW is the receiving rate of the receiving end (proxy server), the unit includes but is not limited to bps, and is calculated by sampling the total received data (recdata) with a predetermined duration (duration), and the formula is as follows:

recvBW＝recvData/duration；

for example, when Duration is 1s, N is 10, and the unit time length for each statistic is 100 ms.

The overhead is a gain factor used to provide extra bandwidth for the retransmitted packet and ensure that the retransmitted packet is sent out as soon as possible, and the default configuration may include, but is not limited to, 25.

The default congestion control algorithm can lead the sending rate to be higher due to the action of the gain coefficient under the condition of insufficient bandwidth, so the override value can be dynamically adjusted according to the congestion degree of the network, the override is increased under the condition of sufficient bandwidth, and the override is reduced under the condition of insufficient bandwidth.

As an alternative, determining the network congestion state between the target application and the proxy server according to the minimum value of M pairs of round trip delays and the maximum value of the transmission bandwidth includes:

obtaining the product between the minimum value of each pair of round-trip delay and the maximum value of the sending bandwidth to obtain M bandwidth delay products;

obtaining unconfirmed data volume in a preset time length corresponding to each group of transmission parameters, and obtaining M unconfirmed data volumes in total, wherein each unconfirmed data volume represents the data volume corresponding to the data packet which is not confirmed by the proxy server in the data packets which are sent to the proxy server by the target application in the corresponding preset time length;

determining a difference between the M bandwidth delay products and the M unacknowledged data amounts;

and determining the network congestion state between the target application and the proxy server according to the difference value.

Optionally, in this embodiment, after determining the minimum value of M pairs of round trip delays and the maximum value of the transmission bandwidth, the method may further include, but is not limited to, determining M bandwidth-delay products according to the obtained latest M rounds, recording N times of RTTs per round, and the maximum value (maxBW) in the transmission bandwidth.

The BDP is a bandwidth-delay product, which is a network performance index and represents the product of the capability of a data link (bits per second) and the round-trip communication delay (bits per second), and the formula is as follows:

BDP＝minRTT*sendMaxBW。

optionally, in this embodiment, the determining, without limitation, a difference between the M bandwidth delay products and the M unacknowledged data volumes may include, for example, recording a difference (delta) N times per round for the latest M rounds, where the offset represents a data volume corresponding to a data packet that is not acknowledged by the proxy server in the data packets sent to the proxy server by the target application within the corresponding preset time period.

As an alternative, determining the network congestion state between the target application and the proxy server according to the difference includes:

when the difference value is larger than a first preset threshold value, determining the network congestion state as a congestion reduction state;

and when the difference value is smaller than a second preset threshold value, determining the network congestion state as a congestion increasing state.

Optionally, in this embodiment, the values of the first preset threshold and the second preset threshold may be the same or different, and may include but are not limited to determining the network congestion state between the target application and the proxy server according to the difference value in the following manner:

s1, counting the delta sum (sumDelta) for M times;

s2, when sumDelta is greater than the lifting threshold, detecting the trend as INCR (corresponding to the congestion reduction status);

s3, if sumDelta is less than the decrease threshold, a trend is detected as DECR (corresponding to the congestion increase status described above).

As an optional solution, adjusting the current packet sending interval to the target packet sending interval according to the network congestion state includes:

when the network congestion state is a congestion reduction state, reducing the current packet sending interval to a target packet sending interval;

and when the network congestion state is a congestion increasing state, increasing the current packet sending interval to the target packet sending interval.

Optionally, in this embodiment, when the network congestion state is a congestion reduction state, the current packet sending interval is decreased to the target packet sending interval, that is, the packet sending frequency is increased, and when the network congestion state is a congestion increase state, the current packet sending interval is increased to the target packet sending interval, that is, the packet sending frequency is decreased.

As an alternative, reducing the current packet sending interval to the target packet sending interval includes:

increasing the value of a preset gain coefficient to a first value, wherein the gain coefficient is used for determining the maximum value of a transmission bandwidth, and the transmission bandwidth is the bandwidth of a data packet transmitted to a proxy server by a target application;

determining the maximum value of the sending bandwidth according to the value of the gain coefficient and a predetermined receiving rate, wherein the value of the gain coefficient is in positive correlation with the maximum value of the sending bandwidth, and the receiving rate is the rate of the proxy server for receiving the data packet sent by the target application;

and determining the value of a packet sending interval parameter as a target packet sending interval according to the maximum value of the sending bandwidth, wherein the value of the packet sending interval parameter is in negative correlation with the maximum value of the sending bandwidth, and the value of the packet sending interval parameter indicates the packet sending interval adopted when the target application transmits the data packet to the proxy server.

Optionally, in this embodiment, the increasing the value of the preset gain coefficient to the first value may include, but is not limited to, increasing the value of the gain coefficient to adaptively adjust and increase the maximum value of the transmission bandwidth when it is detected that the bandwidth of the current network state is sufficient, and further, decreasing the value of the packet transmission interval parameter to dynamically adjust the current packet transmission interval to be the target packet transmission interval.

Optionally, in this embodiment, the gain factor can be used to provide extra bandwidth for the retransmission packet, so as to ensure that the retransmission packet is sent out as soon as possible.

Optionally, in this embodiment, the positive correlation between the value of the gain coefficient and the maximum value of the transmission bandwidth may include, but is not limited to, that the value of the gain coefficient is directly proportional to the maximum value of the transmission bandwidth or in other correlation manners, as long as the maximum value of the transmission bandwidth increases with the increase of the gain coefficient, and the negative correlation between the value of the packet interval parameter and the maximum value of the transmission bandwidth may include, but is not limited to, that the value of the packet interval parameter is inversely proportional to the maximum value of the transmission bandwidth or in other correlation manners, as long as the value of the packet interval parameter decreases with the increase of the maximum value of the transmission bandwidth.

Specifically, the following formula shows that:

sendPeriod＝avgPayloadSize/sendMaxBW；

sendMaxBW＝recvBW*(100+overhead)/100。

wherein sendPeriod represents the packet sending interval parameter, avgPayloadSize represents the size of a data packet to be sent, sendMaxBW represents the maximum value of a sending bandwidth, recvBW represents the rate of the proxy server receiving the data packet sent by the target application, and overhead represents the gain coefficient.

Therefore, when the value of the gain coefficient is increased, the maximum value of the sending bandwidth is determined according to the value of the gain coefficient and the predetermined receiving rate, the value of the packet sending interval parameter is determined as the target packet sending interval according to the maximum value of the sending bandwidth, and then the first data packet set is transmitted to the proxy server of the second transmission protocol through the second transmission protocol according to the updated target packet sending interval, so that the stability of data packet transmission is ensured.

As an alternative, adding the current packetization interval to the target packetization interval includes:

reducing the value of a preset gain coefficient to a second value, wherein the gain coefficient is used for determining the maximum value of a transmission bandwidth, and the transmission bandwidth is the bandwidth of a target application for transmitting a data packet to a proxy server;

Optionally, in this embodiment, the reducing the value of the preset gain coefficient to the second value may include, but is not limited to, being implemented when it is detected that the bandwidth of the current network state is insufficient, reducing the value of the gain coefficient to adaptively adjust and reduce the maximum value of the transmission bandwidth, and further increasing the value of the packet transmission interval parameter to dynamically adjust the current packet transmission interval to be the target packet transmission interval.

Alternatively, in this embodiment, the positive correlation between the value of the gain coefficient and the maximum value of the transmission bandwidth may include, but is not limited to, that the value of the gain coefficient is directly proportional to the maximum value of the transmission bandwidth or in other correlation manners, as long as the maximum value of the transmission bandwidth increases with the increase of the gain coefficient, and the negative correlation between the value of the packet interval parameter and the maximum value of the transmission bandwidth may include, but is not limited to, that the value of the packet interval parameter is inversely proportional to the maximum value of the transmission bandwidth or in other correlation manners, as long as the value of the packet interval parameter decreases with the increase of the maximum value of the transmission bandwidth.

As an optional solution, after adjusting the current packet sending interval to the target packet sending interval according to the network congestion state, the method further includes:

coding second multimedia information acquired by the target application according to the current code rate to obtain a second multimedia data set;

encapsulating the multimedia data in the second multimedia data set into a data packet of a first transmission protocol to obtain a second data packet set;

and transmitting the second data packet set to a proxy server of a second transmission protocol through the second transmission protocol according to the target packet sending interval.

Optionally, in this embodiment, the code rate is used to encode the audio/video information by using the code rate (which is adjustable) when the client (or application) acquires the audio/video information, so as to obtain the audio/video data, and then encapsulate the audio/video data into the RTMP data packet.

And after the RTMP data packets are packaged, transmitting the second data packet set to a proxy server of a second transmission protocol through the second transmission protocol according to the updated target packet sending interval.

For example, fig. 7 is a schematic diagram of another optional multimedia data transmission control method according to an embodiment of the present invention, and as shown in fig. 7, taking a virtual live broadcast scene as an example, the method includes: the system comprises a user terminal A and a server B, wherein the user terminal A is provided with a live broadcast application, the server B comprises an SRT server (corresponding to the proxy server) and an RTMP server (corresponding to the target server), an SRT protocol corresponds to the first transmission protocol, and a TCP protocol corresponds to the second transmission protocol.

s4, dynamically determining a packet sending interval according to the network transmission parameters through a dynamic adjustment module;

s5, based on SRT protocol, sending RTMO data packet to SRT server according to (dynamic) packet sending interval;

and S6, sending the RTMP data packet to the RTMP server on the SRT server according to the TCP protocol.

As an optional solution, the method further includes:

and adjusting the current code rate to a target code rate according to the network congestion state, wherein the target application is used for coding the multimedia information acquired by the target application according to the target code rate.

Optionally, in this embodiment, how to adjust the current bitrate to the target bitrate may be determined according to the current network congestion state, for example, when the network congestion state is a congestion reduction state, the current bitrate is increased to the target bitrate, and when the network congestion state is a congestion increase state, the current bitrate is reduced to the target bitrate.

It should be noted that, in the process of adjusting the current code rate to the target code rate, the range of the target code rate may include, but is not limited to, a corresponding range configured in advance according to the service scenario and the terminal performance, for example, the range is not greater than the preset maximum code rate in the process of increasing the code rate, and is not less than the preset minimum code rate in the process of reducing the code rate.

As an alternative, adjusting the current code rate to the target code rate according to the network congestion state includes:

when the network congestion state is a congestion reduction state, increasing the current code rate to a target code rate;

and when the network congestion state is a congestion increasing state, reducing the current code rate to the target code rate.

Optionally, in this embodiment, the adjusting the current bitrate to the target bitrate according to the network congestion state may include, but is not limited to, the following steps:

s1, acquiring the current detection trend state (corresponding to the network congestion state) of the transport layer;

s2, if the state is INCR, increasing the bitrate, and adjusting the current bitrate to the target bitrate, in which α >1, α is a preset value, and bitrate is the current bitrate);

s3, if the state is DECR, the bitrate is reduced, and the specific implementation manner of adjusting the current bitrate to the target bitrate is target bitrate β, where β < 1, β is a preset value, and bitrate is the current bitrate).

As an optional scheme, after adjusting the current code rate to the target code rate according to the network congestion state, the method further includes:

coding third multimedia information acquired by target application according to the target code rate to obtain a third multimedia data set;

encapsulating the multimedia data in the third multimedia data set into a data packet of a first transmission protocol to obtain a third data packet set;

and transmitting the third data packet set to a proxy server of a second transmission protocol through the second transmission protocol according to the target packet sending interval.

Optionally, in this embodiment, the target code rate is used to encode the audio/video information by using the target code rate under the condition that the client (or the application) acquires the audio/video information, so as to obtain the audio/video data, and then encapsulate the audio/video data into the RTMP data packet.

For example, fig. 8 is a schematic diagram of another optional multimedia data transmission control method according to an embodiment of the present invention, and as shown in fig. 8, taking a virtual live broadcast scene as an example, the method includes: the system comprises a user terminal A and a server B, wherein the user terminal A is provided with a live broadcast application, the server B comprises an SRT server (corresponding to the proxy server) and an RTMP server (corresponding to the target server), the SRT protocol corresponds to the first transmission protocol, and the TCP protocol corresponds to the second transmission protocol.

The dynamic adjustment module adjusts the packet sending interval in the following way, so that the user terminal A sends the RTMP data packet to the RTMP server according to the following conditions:

s1, the dynamic adjustment module acquires RTT and SendMaxBW;

s2, the dynamic adjustment module determines the current network congestion state according to the RTT and the SendMaxBW;

s3-1, when the network congestion state is a congestion reduction state, the dynamic adjustment module increases the gain coefficient to increase SendMaxBW, and further reduces the current packet sending interval to a target packet sending interval;

and S3-2, when the network congestion state is a congestion increasing state, reducing the gain coefficient to reduce SendMaxBW, and further increasing the current packet sending interval to the target packet sending interval.

And S4, based on the dynamically determined packet sending interval, sending RMTP data packets from the user terminal A to the SRT server according to the SRT protocol.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.

According to another aspect of the embodiments of the present invention, there is also provided a transmission control apparatus for multimedia data for implementing the transmission control method for multimedia data. As shown in fig. 9, the apparatus includes:

the encoding module 902 is configured to encode first multimedia information acquired by a target application according to a current code rate to obtain a first multimedia data set;

an encapsulating module 904, configured to encapsulate multimedia data in the first multimedia data set into a data packet of a first transmission protocol, so as to obtain a first data packet set;

a transmission module 906, configured to transmit the first data packet set to a proxy server of a second transmission protocol through the second transmission protocol according to the current packet sending interval, where the proxy server is configured to transmit the first data packet set to a target server of the first transmission protocol, the second transmission protocol is a data transmission protocol based on a user datagram protocol UDP, and a value of a packet sending interval parameter allows for dynamic adjustment;

a determining module 908, configured to determine a network congestion state between the target application and the proxy server according to a target transmission parameter when the target application transmits the first data packet set to the proxy server;

an adjusting module 910, configured to adjust a current packet sending interval to a target packet sending interval according to a network congestion state, where the target application is configured to send a data packet of the first transport protocol to the proxy server according to the target packet sending interval.

As an alternative, the determining module 908 includes:

the first obtaining unit is used for obtaining M groups of transmission parameters, wherein each group of transmission parameters comprises N times of statistical transmission parameters within a preset time length, each statistical transmission parameter comprises round-trip delay and sending bandwidth, the round-trip delay is the delay between the sending time and the receiving time, the sending time is the time when the target application sends a data packet to the proxy server, the receiving time is the time when the target application receives the confirmation information of the data packet sent by the proxy server, the sending bandwidth is the bandwidth when the target application sends the data packet to the proxy server, M is 1 or M is not less than 2, and N is 1 or N is not less than 2;

a second obtaining unit, configured to obtain a minimum value of round-trip delay and a maximum value of transmission bandwidth in each group of transmission parameters, and obtain a minimum value of M pairs of round-trip delay and a maximum value of transmission bandwidth;

and the determining unit is used for determining the network congestion state between the target application and the proxy server according to the minimum value of the M pairs of round trip delay and the maximum value of the sending bandwidth.

As an alternative, the apparatus is configured to determine the network congestion state between the target application and the proxy server according to the minimum value of M pairs of round trip delays and the maximum value of the transmission bandwidth by:

As an alternative, the apparatus is configured to determine the network congestion state between the target application and the proxy server according to the difference by:

As an alternative, the apparatus is configured to adjust the current packet sending interval to the target packet sending interval according to the network congestion state by:

As an alternative, the apparatus is configured to reduce the current packetization interval to the target packetization interval by:

As an alternative, the apparatus is configured to increase the current packetization interval to the target packetization interval by:

As an optional solution, the apparatus is further configured to:

after the current packet sending interval is adjusted to the target packet sending interval according to the network congestion state, coding second multimedia information acquired by target application according to the current code rate to obtain a second multimedia data set;

As an optional solution, the apparatus is further configured to:

As an alternative, the apparatus is configured to adjust the current bitrate to the target bitrate according to the network congestion status by:

As an optional solution, the apparatus is further configured to:

after the current code rate is adjusted to the target code rate according to the network congestion state, encoding third multimedia information acquired by target application according to the target code rate to obtain a third multimedia data set;

According to another aspect of the embodiment of the present invention, there is also provided an electronic device for implementing the transmission control method of multimedia data, where the electronic device may be a terminal device or a server shown in fig. 1. The present embodiment takes the electronic device as an example for explanation. As shown in fig. 10, the electronic device comprises a memory 1002 and a processor 1004, the memory 1002 having stored therein a computer program, the processor 1004 being arranged to execute the steps of any of the method embodiments described above by means of the computer program.

Optionally, in this embodiment, the electronic device may be located in at least one network device of a plurality of network devices of a computer network.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

s1, encoding the first multimedia information acquired by the target application according to the current code rate to obtain a first multimedia data set;

s2, encapsulating the multimedia data in the first multimedia data set into a data packet of a first transmission protocol to obtain a first data packet set;

s3, according to the current packet sending interval, the first data packet set is transmitted to a proxy server of a second transmission protocol through the second transmission protocol, wherein the proxy server is used for transmitting the first data packet set to a target server of the first transmission protocol, the second transmission protocol is a data transmission protocol based on a User Datagram Protocol (UDP), and the value of the packet sending interval parameter allows dynamic adjustment;

s4, determining the network congestion state between the target application and the proxy server according to the target transmission parameters when the target application transmits the first data packet set to the proxy server;

and S5, adjusting the current packet sending interval to a target packet sending interval according to the network congestion state, wherein the target application is used for sending the data packets of the first transmission protocol to the proxy server according to the target packet sending interval.

Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 10 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palmtop computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 10 is a diagram illustrating a structure of the electronic apparatus. For example, the electronics may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 10, or have a different configuration than shown in FIG. 10.

The memory 1002 may be used to store software programs and modules, such as program instructions/modules corresponding to the multimedia data transmission control method and apparatus in the embodiments of the present invention, and the processor 1004 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002, that is, implements the above-described multimedia data transmission control method. The memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1002 may further include memory located remotely from the processor 1004, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 1002 may be, but not limited to, specifically configured to store information such as multimedia data. As an example, as shown in fig. 10, the memory 1002 may include, but is not limited to, an encoding module 902, an encapsulating module 904, a transmitting module 906, a determining module 908, and an adjusting module 910 in the transmission control apparatus for multimedia data. In addition, the multimedia data transmission control device may further include, but is not limited to, other module units in the multimedia data transmission control device, which is not described in detail in this example.

Optionally, the above-mentioned transmission device 1006 is used for receiving or sending data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 1006 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices so as to communicate with the internet or a local area Network. In one example, the transmission device 1006 is a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.

In addition, the electronic device further includes: a display 1008 for displaying the multimedia data; and a connection bus 1010 for connecting the respective module parts in the above-described electronic apparatus.

In other embodiments, the terminal device or the server may be a node in a distributed system, where the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting a plurality of nodes through a network communication. Nodes can form a Peer-To-Peer (P2P, Peer To Peer) network, and any type of computing device, such as a server, a terminal, and other electronic devices, can become a node in the blockchain system by joining the Peer-To-Peer network.

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various alternative implementations of the aspect of transmission control of multimedia data described above. Wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.

Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:

Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention.

In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. A method for controlling transmission of multimedia data, comprising:

coding first multimedia information acquired by target application according to the current code rate to obtain a first multimedia data set;

encapsulating the multimedia data in the first multimedia data set into a data packet of a first transmission protocol to obtain a first data packet set;

according to the current packet sending interval, transmitting the first data packet set to a proxy server of a second transmission protocol through the second transmission protocol, wherein the proxy server is used for transmitting the first data packet set to a target server of the first transmission protocol, the second transmission protocol is a data transmission protocol based on a User Datagram Protocol (UDP), and the value of the packet sending interval parameter is allowed to be dynamically adjusted;

determining a network congestion state between the target application and the proxy server according to a target transmission parameter when the target application transmits the first data packet set to the proxy server;

according to the network congestion state, adjusting the current packet sending interval to a target packet sending interval, wherein the target application is used for sending the data packet of the first transmission protocol to the proxy server according to the target packet sending interval;

the determining a network congestion state between the target application and the proxy server according to the target transmission parameter when the target application transmits the first data packet set to the proxy server includes:

acquiring M groups of transmission parameters, wherein each group of transmission parameters comprises transmission parameters counted for N times in a preset time length, the transmission parameters counted each time comprise round-trip delay and sending bandwidth, the round-trip delay is the delay between the sending time and the receiving time, the sending time is the time when the target application sends the data packet to the proxy server, the receiving time is the time when the target application receives confirmation information of the data packet sent by the proxy server, the sending bandwidth is the bandwidth when the target application sends the data packet to the proxy server, M is 1 or M is more than or equal to 2, and N is 1 or N is more than or equal to 2;

obtaining the minimum value of the round trip delay and the maximum value of the sending bandwidth in each group of transmission parameters, and obtaining the minimum value of M pairs of the round trip delay and the maximum value of the sending bandwidth together, wherein the maximum value of the sending bandwidth is the product of the receiving rate of the proxy server and a preset gain coefficient;

determining a network congestion state between the target application and the proxy server according to the minimum value of the round trip delay and the maximum value of the sending bandwidth;

the determining a network congestion state between the target application and the proxy server according to the minimum value of the round trip delay and the maximum value of the sending bandwidth by the M pairs comprises:

obtaining unconfirmed data volume within a preset time length corresponding to each group of transmission parameters, and obtaining M unconfirmed data volumes in total, wherein each unconfirmed data volume represents the data volume corresponding to the data packet which is not confirmed by the proxy server and is sent to the proxy server by the target application within the corresponding preset time length;

determining a difference between the M bandwidth-delay products and the M unacknowledged data volumes; and determining the network congestion state between the target application and the proxy server according to the difference value.

2. The method of claim 1, wherein determining the network congestion status between the target application and the proxy server according to the difference comprises:

3. The method of claim 1, wherein the adjusting the current packet sending interval to a target packet sending interval according to the network congestion state comprises:

when the network congestion state is a congestion reduction state, reducing the current packet sending interval to the target packet sending interval;

4. The method of claim 3, wherein the reducing the current packet interval to the target packet interval comprises:

increasing a value of a preset gain coefficient to a first value, wherein the gain coefficient is used for determining a maximum value of a transmission bandwidth, and the transmission bandwidth is a bandwidth for the target application to transmit a data packet to the proxy server;

determining the maximum value of the sending bandwidth according to the value of the gain coefficient and a predetermined receiving rate, wherein the value of the gain coefficient is positively correlated with the maximum value of the sending bandwidth, and the receiving rate is the rate of the proxy server receiving the data packet sent by the target application;

and determining the value of a packet sending interval parameter as the target packet sending interval according to the maximum value of the sending bandwidth, wherein the value of the packet sending interval parameter is in negative correlation with the maximum value of the sending bandwidth, and the value of the packet sending interval parameter indicates the packet sending interval adopted when the target application transmits the data packet to the proxy server.

5. The method of claim 3, wherein said increasing the current packet interval to the target packet interval comprises:

reducing a value of a preset gain coefficient to a second value, wherein the gain coefficient is used for determining a maximum value of a transmission bandwidth, and the transmission bandwidth is a bandwidth for the target application to transmit a data packet to the proxy server;

6. The method according to any one of claims 1 to 5, wherein after adjusting the current packet transmission interval to a target packet transmission interval according to the network congestion state, the method further comprises:

encapsulating the multimedia data in the second multimedia data set into a data packet of the first transmission protocol to obtain a second data packet set;

and transmitting the second data packet set to the proxy server of the second transmission protocol through the second transmission protocol according to the target packet sending interval.

7. The method according to any one of claims 1 to 5, further comprising:

8. The method of claim 7, wherein adjusting the current code rate to a target code rate according to the network congestion status comprises:

when the network congestion state is a congestion reduction state, increasing the current code rate to the target code rate;

9. The method of claim 7, wherein after the adjusting the current code rate to a target code rate according to the network congestion status, the method further comprises:

coding third multimedia information acquired by the target application according to the target code rate to obtain a third multimedia data set;

encapsulating the multimedia data in the third multimedia data set into a data packet of the first transmission protocol to obtain a third data packet set;

and transmitting the third data packet set to the proxy server of the second transmission protocol through the second transmission protocol according to the target packet sending interval.

10. A transmission control apparatus for multimedia data, comprising:

the encoding module is used for encoding the first multimedia information acquired by the target application according to the current code rate to obtain a first multimedia data set;

the encapsulation module is used for encapsulating the multimedia data in the first multimedia data set into a data packet of a first transmission protocol to obtain a first data packet set;

a transmission module, configured to transmit the first data packet set to a proxy server of a second transmission protocol through the second transmission protocol according to a current packet sending interval, where the proxy server is configured to transmit the first data packet set to a target server of the first transmission protocol, the second transmission protocol is a data transmission protocol based on a user datagram protocol UDP, and a value of a packet sending interval parameter is allowed to be dynamically adjusted;

a determining module, configured to determine a network congestion state between the target application and the proxy server according to a target transmission parameter when the target application transmits the first data packet set to the proxy server;

an adjusting module, configured to adjust the current packet sending interval to a target packet sending interval according to the network congestion state, where the target application is configured to send a data packet of the first transport protocol to the proxy server according to the target packet sending interval;

the determining module includes:

a first obtaining unit, configured to obtain M groups of transmission parameters, where each group of transmission parameters includes transmission parameters counted N times within a preset time duration, and each counted transmission parameter includes a round-trip delay and a sending bandwidth, where the round-trip delay is a delay between a sending time and a receiving time, the sending time is a time when the target application sends the data packet to the proxy server, the receiving time is a time when the target application receives acknowledgement information of the data packet sent by the proxy server, the sending bandwidth is a bandwidth when the target application sends the data packet to the proxy server, where M is 1 or M is greater than or equal to 2, N is 1 or N is greater than or equal to 2;

a second obtaining unit, configured to obtain a minimum value of the round trip delay and a maximum value of the transmission bandwidth in each group of transmission parameters, and obtain a minimum value of M pairs of the round trip delay and a maximum value of the transmission bandwidth together, where the maximum value of the transmission bandwidth is a product of a reception rate of the proxy server and a preset gain coefficient;

a determining unit, configured to determine a network congestion state between the target application and the proxy server according to a minimum value of the round trip delay and a maximum value of the sending bandwidth;

the determining unit is configured to determine a network congestion state between the target application and the proxy server according to the minimum value of the round trip delay and the maximum value of the transmission bandwidth by M:

obtaining unconfirmed data volume in a preset time length corresponding to each group of transmission parameters, and obtaining M unconfirmed data volumes in total, wherein each unconfirmed data volume represents the data volume corresponding to a data packet which is not confirmed by the proxy server and is sent to the proxy server by the target application in the corresponding preset time length;

determining a difference between the M bandwidth-delay products and the M unacknowledged data volumes;

11. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein the program is executable by a terminal device or a computer to perform the method of any one of claims 1 to 9.

12. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 9 by means of the computer program.