WO2024056032A1

WO2024056032A1 - Decoding method and apparatus, data transmission method and apparatus, terminal, and server

Info

Publication number: WO2024056032A1
Application number: PCT/CN2023/118839
Authority: WO
Inventors: 邵胜均
Original assignee: 维沃移动通信有限公司
Priority date: 2022-09-16
Filing date: 2023-09-14
Publication date: 2024-03-21
Also published as: CN115484240A

Abstract

The present application relates to the technical field of video transmission, and discloses a decoding method and apparatus, a data transmission method and apparatus, a terminal, and a server. The data transmission method comprises: receiving a plurality of RTP data packets of a target multimedia file sent by a server, protocol headers of the RTP data packets carrying shared codes corresponding to the RTP data packets; when it is determined that the RTP data packets do not comprise a first RTP data packet of the target multimedia file sent by the server, sending a shared code of the first RTP data packet to a second terminal; receiving the first RTP data packet, and inserting the first RTP data packet into a cache for decoding of the target multimedia file, the first RTP data packet being sent by the second terminal according to the shared code of the first RTP data packet, and a first terminal and the second terminal jointly receiving the RTP data packets of the target multimedia file sent by the server.

Description

Decoding, data transmission method, device, terminal and server

Cross-references to related applications

This application claims priority to Chinese Patent Application No. 202211127820.8 filed in China on September 16, 2022, the entire content of which is incorporated herein by reference.

Technical field

This application belongs to the field of video transmission technology, and particularly relates to a decoding and data transmission method, device, terminal and server.

Background technique

In a video conferencing scenario, in order to ensure the real-time nature of audio and video, Real Time Protocol (RTP) audio and video data packets and Real-time Transport Control Protocol (RTCP) control packets pass through the underlying user datagram protocol ( User Datagram Protocol, UDP) is transmitted. UDP transmission is unreliable, and the data sent out is not guaranteed to reach the destination. Especially under network jitter or weak network, data will be seriously lost. In order to ensure the continuity of sound and the smoothness of video, the client and server solve the problem of data loss through packet loss and retransmission.

The server delivers the pronunciation video to the terminal, and the terminal feedbacks the service quality of the pronunciation video to the server through the RTCP protocol, as shown in Figure 1.

However, when the network quality is poor, if the terminal cannot successfully receive the data packet, it needs to request the server to retransmit the data packet. If multiple terminals request the server to retransmit the data packet at the same time, it will cause the server to be overloaded and also As a result, the terminal cannot obtain the lost data packets in time and cannot ensure smooth video playback.

Contents of the invention

Embodiments of the present application provide a decoding and data transmission method, device, terminal and server, which can solve the problem that if multiple terminals request retransmission of data packets from the server, the server will be overloaded and the terminal will be unable to obtain the lost data in time. The package cannot guarantee the smooth playback of the video.

In a first aspect, embodiments of the present application provide a decoding method, executed by a first terminal, including:

Receive multiple real-time transmission protocol RTP data packets of the target multimedia file sent by the server. The protocol header of the RTP data packet carries the shared code corresponding to the RTP data packet. The shared code is used to indicate the RTP data packet. The arrangement number in the target multimedia file and the shared code corresponding to the RTP data packet are determined by the server based on the obtained initial value of the shared code corresponding to the target multimedia file;

When it is determined that the RTP data packet does not include the first RTP data packet of the target multimedia file sent by the server, send the shared code of the first RTP data packet to the second terminal;

Receive the first RTP data packet and insert the first RTP data packet into the cache to decode the target multimedia file. The first RTP data packet is the second terminal's data according to the first RTP data packet. Sent by shared code;

Wherein, the first terminal and the second terminal jointly receive the RTP data packet of the target multimedia file sent by the server.

In a second aspect, embodiments of the present application provide a decoding device, executed by a first terminal, including:

The first receiving module is used to receive multiple real-time transmission protocol RTP data packets of the target multimedia file sent by the server. The protocol header of the RTP data packet carries the shared code corresponding to the RTP data packet. The shared code is In order to indicate the sequence number of the RTP data packet in the target multimedia file, the shared code corresponding to the RTP data packet is determined by the server based on the obtained initial value of the shared code corresponding to the target multimedia file;

A first sending module configured to send the shared code of the first RTP data packet to the second terminal when it is determined that the RTP data packet does not include the first RTP data packet of the target multimedia file sent by the server;

a decoding module, configured to receive the first RTP data packet and insert the first RTP data packet into the cache to decode the target multimedia file, where the first RTP data packet is generated by the second terminal according to the first An RTP packet is sent in the shared encoding;

In the third aspect, embodiments of the present application provide a data transmission method, executed by a server, including:

Get the initial value of the shared encoding corresponding to the target multimedia file;

According to the initial value of the shared encoding, determine the shared encoding corresponding to each of the multiple real-time transmission protocol RTP data packets of the target multimedia file;

Send multiple RTP data packets of the target multimedia file to the first terminal and the second terminal;

Wherein, the protocol header of the RTP data packet carries a shared code corresponding to the RTP data packet, and the shared code is used to indicate the sequence number of the RTP data packet in the target multimedia file.

In the fourth aspect, embodiments of the present application provide a data transmission device, executed by a server, including:

The acquisition module is used to obtain the initial value of the shared encoding corresponding to the target multimedia file;

A determination module, configured to determine the shared encoding corresponding to each of the multiple real-time transmission protocol RTP data packets of the target multimedia file according to the initial value of the shared encoding;

a second sending module, configured to send multiple RTP data packets of the target multimedia file to the first terminal and the second terminal;

In a fifth aspect, embodiments of the present application provide an electronic device. The electronic device is a first terminal and includes a processor and a memory. The memory stores programs or instructions that can be run on the processor. The program Or when the instruction is executed by the processor, the steps of the decoding method as described in the first aspect are implemented.

In a sixth aspect, embodiments of the present application provide an electronic device. The electronic device is a server and includes a processor and a memory. The memory stores programs or instructions that can be run on the processor. The program or instructions When executed by the processor, the steps of the data transmission method described in the third aspect are implemented.

In a seventh aspect, embodiments of the present application provide a readable storage medium. Programs or instructions are stored on the readable storage medium. When the programs or instructions are executed by a processor, the implementation is as described in the first aspect or the third aspect. steps of the method.

In an eighth aspect, embodiments of the present application provide a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the first aspect. or the method described in the third aspect.

In a ninth aspect, embodiments of the present application provide a computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the method described in the first aspect or the third aspect.

In this embodiment of the present application, when it is determined that the multiple RTP data packets of the target multimedia file sent by the receiving server do not include the first RTP data packet of the target multimedia file sent by the server, sending the first RTP data packet of the target multimedia file sent by the server to the second terminal. A shared code of an RTP data packet, inserting the received first RTP data packet sent by the second terminal according to the shared code of the first RTP data packet into the cache to decode the target multimedia file; thus, it is possible to avoid The server requests to retransmit the data packet, causing the server to be overloaded and the terminal cannot obtain the lost data packet in time. By obtaining the lost data packet from other terminals, it is ensured that the terminal can obtain the complete RTP data packet of the target multimedia file in time, ensuring The target multimedia file can be decoded to obtain the complete target multimedia file to ensure smooth playback of the target multimedia file.

Description of drawings

Figure 1 is a schematic diagram of the interaction process between the server and the terminal;

Figure 2 is one of the flow diagrams of the decoding method according to the embodiment of the present application;

Figure 3 is a schematic diagram of the communication process between the first terminal, the second terminal and the server;

Figure 4 is a schematic structural diagram of an RTP data packet;

Figure 5 is a schematic diagram of the location of the shared coding calculation module;

Figure 6 is a schematic flow chart of the data transmission method according to the embodiment of the present application;

Figure 7 is a schematic module diagram of a decoding device according to an embodiment of the present application;

Figure 8 is a schematic structural diagram of a terminal according to an embodiment of the present application;

Figure 9 is a structural block diagram of a terminal according to an embodiment of the present application;

Figure 10 is a module schematic diagram of a data transmission device according to an embodiment of the present application;

Figure 11 is a structural block diagram of a server according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the facts in this application Examples, and all other examples obtained by those of ordinary skill in the art, fall within the scope of protection of this application.

The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It is to be understood that the figures so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in orders other than those illustrated or described herein, and that "first," "second," etc. are distinguished Objects are usually of one type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the related objects are in an "or" relationship.

The decoding and data transmission methods, devices, terminals and servers provided by the embodiments of the present application will be described in detail through specific embodiments and application scenarios in conjunction with the accompanying drawings.

As shown in Figure 2, the decoding method in this embodiment of the present application includes:

Step 201: Receive multiple real-time transmission protocol RTP data packets of the target multimedia file sent by the server, and the protocol header of the RTP data packet carries the shared code corresponding to the RTP data packet;

It should be noted that the shared code is used to indicate the sequence number of the RTP data packet in the target multimedia file, and the shared code corresponding to the RTP data packet is the corresponding code of the target multimedia file obtained by the server. The initial value of the shared encoding is determined.

Step 202: If it is determined that the RTP data packet does not include the first RTP data packet of the target multimedia file sent by the server, send the shared code of the first RTP data packet to the second terminal;

It should be noted that the first terminal and the second terminal jointly receive the RTP data packet of the target multimedia file sent by the server.

Step 203: Receive the first RTP data packet, and insert the first RTP data packet into the cache to decode the target multimedia file. The first RTP data packet is the second terminal's data according to the first RTP The shared encoding of the data packet is sent;

It should be noted that, when it is determined that the multiple RTP data packets of the target multimedia file sent by the receiving server do not include the first RTP data packet of the target multimedia file sent by the server, sending the first RTP to the second terminal. The shared coding of the data packet inserts the received first RTP data packet sent by the second terminal according to the shared coding of the first RTP data packet into the cache to decode the target multimedia file; in this way, the data packet can be lost In this case, it can also ensure that the terminal can obtain the complete RTP data packet of the target multimedia file to ensure the smooth playback of the multimedia file. At the same time, this method can also avoid the server load caused by multiple terminals requesting the retransmission of data packets from the server. Overweight occurs.

It should be noted that the target multimedia file mentioned in the embodiment of this application refers to the multimedia service performed by the first terminal together with other terminals including the second terminal. The multimedia service may be audio and/or video services. For example, when multiple terminals make a video call together, the video can be understood as a video shared by multiple terminals, and can also be called a shared video.

In the embodiment of this application, the shared code of the RTP data packet not received by the first terminal is sent to the second terminal, because the same video stream uses a set of shared codes, that is to say, the second terminal determines the third code using the shared codes. a terminal The lost RTP data packet is consistent with the RTP data packet lost by the first terminal determined by the server using the packet sequence of the RTP data packet. This can ensure that the first terminal can reacquire the lost RTP data packet from channels other than the server. Ensure that the terminal can decode and obtain the complete video stream.

Optionally, in order to ensure that the first terminal can communicate with the second terminal, in at least one embodiment of the present application, before the first terminal sends the shared code of the first RTP data packet to the second terminal, the method further includes :

Step 200: When the packet loss rate of the first terminal meets the first condition, establish a connection with the second terminal;

Wherein, the first condition includes: the packet loss rate is greater than or equal to the first preset threshold;

Optionally, when the packet loss rate of the first terminal meets the first condition, establishing a connection with the second terminal may refer to when the first terminal determines that its packet loss rate is greater than or equal to the first preset threshold. , immediately establishes a connection with the second terminal; of course, in order to avoid misjudgment by the first terminal, the first terminal can also communicate with the second terminal after the packet loss rate is greater than or equal to the first preset threshold for a duration greater than or equal to the first duration. The terminal establishes a connection. In this case, it can be understood that the first condition includes that the packet loss rate is greater than or equal to the first preset threshold and the duration for which the packet loss rate is greater than or equal to the first preset threshold is greater than or equal to the first duration. .

Optionally, the first preset threshold and the first duration may be configured, preconfigured, or agreed upon by a protocol.

Optionally, in at least one embodiment of this application, the implementation of step 200 includes:

Step 2001: Receive information about at least one terminal sent by the server;

It should be noted that the packet loss rate of each terminal in the at least one terminal is less than or equal to the second preset threshold;

Optionally, the second preset threshold may be configured, preconfigured, or agreed upon by a protocol.

It should be noted that under normal circumstances, all terminals accessing the target multimedia file need to obtain the packet loss rate of the terminal based on multiple received RTP data packets and unreceived RTP data packets, that is, the packet loss rate = The number of RTP data packets that have not been received/(the number of RTP data packets that have been received + the number of RTP data packets that have not been received); then the packet loss rate is sent to the server, and the server chooses to lose it among all terminals. The terminal whose packet rate is less than or equal to the second preset threshold then sends the terminal information to the first terminal. The terminal information sent by the server to the first terminal may be the identification of the terminal (for example, terminal index) or the terminal information. IP port address.

Step 2002: When the packet loss rate of the first terminal meets the first condition, select a second terminal from the at least one terminal;

Step 2003: Establish a connection with the second terminal.

The establishment of a connection with the second terminal can be understood as performing P2P hole drilling. If the hole drilling between the first terminal and the second terminal is successful, the first terminal and the second terminal are successfully connected.

Optionally, in the process of selecting the second terminal, the first terminal can sort multiple other terminals based on the packet loss rate, and first select the terminal with the lowest packet loss rate in order from low to high packet loss rate to try to connect with it. , after the connection is successful, the first terminal will no longer try to connect with other terminals. If the first terminal does not successfully connect to the terminal with the lowest packet loss rate, then the terminal with the next lowest packet loss rate will be selected to try to connect to it, and so on. until a connection is successfully established with a terminal. This method can ensure that the first terminal can connect to the second terminal with better network quality, thereby increasing the probability that the first terminal obtains the complete video stream.

Optionally, in at least one embodiment of this application, the implementation of step 202 includes:

Step 2021: Determine whether the first RTP data packet has not been received based on the received multiple RTP data packets of the target multimedia file sent by the server;

Step 2022: If the first RTP data packet is not received, send the shared code of the first RTP data packet to the second terminal.

Optionally, step 2021 can be further implemented by: based on the shared codes of the received multiple RTP data packets, in the case where the shared codes of the multiple RTP data packets are discontinuous, determine that there is unreceived first RTP data. Bag.

It should be noted that because RTP data packets are sent continuously in order, when the first terminal receives some RTP data packets, it can infer the unsuccessfully received RTP data packets based on the received data packets. This method can ensure Obtain shared encoding accuracy for lost RTP packets.

Optionally, the server sends multiple RTP data packets to the first terminal and the second terminal, the protocol header of the RTP data packet carries the shared code corresponding to the RTP data packet, and the first terminal performs the processing based on the obtained RTP Whether some RTP data packets are lost can be determined by whether the shared codes of the data packets are continuous. For example, if the shared codes of the RTP data packets received by the first terminal are 101, 103, 104, 105, and 106, then the first terminal can It is determined that the RTP packet with shared encoding 102 was not received.

It should also be noted that before the server sends multiple RTP data packets of the target multimedia file to the first terminal and the second terminal, it needs to obtain the initial value of the shared encoding corresponding to the target multimedia file; according to the shared encoding The initial value determines the shared encoding corresponding to each of the multiple RTP data packets of the target multimedia file, and then sends the multiple RTP data packets.

Optionally, the server obtains the initial value of the shared encoding corresponding to the target multimedia file by: based on the initial value of the shared encoding corresponding to the first multimedia file, the size of the data block of the target multimedia file, and the size of the RTP data packet. The size and the first coefficient determine the initial value of the shared encoding corresponding to the target multimedia file;

Wherein, the first coefficient is determined by the size of the data block of the target multimedia file and the size of the RTP data packet. The first multimedia file is a multimedia file transmitted before the target multimedia file.

Optionally, the server determines the initial value of the shared encoding corresponding to the target multimedia file based on the formula v'=v+m/p+r, where v' is the initial value of the shared encoding corresponding to the target multimedia file, and v is the first multimedia file. The initial value of the shared encoding corresponding to the body file, m is the size of the data block of the target multimedia file, and p is the size of the RTP data packet. When m/p is evenly divisible, r=0; when m/p is not evenly divisible, r=1.

It should be noted that by adding a shared encoding indication in the extension field of the protocol header of the RTP data packet, a unified identifier is assigned to the RTP data packet to ensure that different terminals have a consistent understanding of the RTP data packet.

Optionally, the protocol header of each RTP data packet sent by the server to the first terminal includes a packet sequence number. The packet sequence number is used to indicate the sequence number of the RTP data packet sent by the server to the first terminal. Because different terminals access The server time may be different, so the same data packet sent by the server to different terminals may have different packet sequence numbers. In another embodiment of the present application, the first terminal sends the shared code of the first RTP data packet to the second terminal. At the same time, the packet sequence number of the first RTP data packet can also be sent to the server, and then the first terminal waits for the feedback of the first RTP data packet. If the terminal first receives the first RTP data packet sent by the second terminal, the first terminal inserts the received first RTP data packet sent by the second terminal into the cache to decode the target multimedia file; if the terminal first receives The first RTP data packet retransmitted by the server, then the first terminal inserts the received retransmitted first RTP data packet into the cache to decode the target multimedia file; by obtaining the data packet from the second terminal and the server at the same time, Added a way to obtain data packets to avoid the situation where the server is overloaded due to a simple server request to retransmit data packets and the terminal cannot obtain the lost data packets in time. It can ensure that the first terminal obtains the lost data packets in time and ensures the playback of the video. Smooth.

Taking multiple terminals making a shared video call as an example, the specific applications of the embodiments of the present application are described in detail below.

The specific implementation process includes:

Step 1: The quality of the network is judged by the packet loss rate (packet loss rate = lost data packets/total data packets). Each terminal will report its own packet loss situation in real time, and the system will report it every interval (such as 10s ), screen out several terminals with the smallest packet loss rate, and the packet loss rate must be less than a certain threshold (such as below 5%), and then deliver the IP port information of the qualified terminals to all terminals (including qualified terminals) Own).

Step 2: When the packet loss rate of a terminal (such as terminal 1 in Figure 3) continues to be higher than a certain threshold (such as 20%), the terminal with the best network will be selected in turn to perform P2P hole punching (you can understand the attempt Connect other terminals), if the hole drilling is successful (that is, the two terminals can send data to each other), stop p2p hole drilling with subsequent terminals.

Step 3: When terminal 1 loses a packet, it sends the shared code of the lost data packet to terminal 2 and obtains the lost data packet from terminal 2.

What needs to be explained here is that if terminal 1 sends the packet sequence number of the lost data packet to terminal 2, there is a high probability that it cannot get the data from terminal 2, or the data it gets is wrong. The reasons are as follows: For example, when terminal 2 joins the conference first , when terminal 1 joins the conference, the RTP packet sequence number of terminal 2 has rotated from 0 to 65535 many times, and terminal 1 has just entered the conference. Although the shared video seen is the same as that of terminal 2, the packet sent to terminal 1 The sequence number starts counting, so the packet sequence numbers of terminal 2 and terminal 1 are completely different at the same time. If the packet sequence numbers are very close to each other, due to buffering, the RTP data packet with the same sequence number (the packet sequence number of the lost packet from terminal 1) can be found from terminal 2, but the RTP video data is also different. So how can we obtain the correct data from the terminal 2? This problem is solved in four steps in the embodiment of this application.

The first step is to extend a field in the RTP header. The X in Figure 4 is used for extension. If The packet sequence number is distinguished. The sequence number of the extension field is later called the shared encoding. If it is other video streams (such as cameras), this field defaults to 0.

The second step is to share the encoded I-frame or p-frame (hereinafter referred to as data block) of the video. Before sending the RTP data packet to generate, a new module will be added, which can be called the shared encoding calculation module, as shown in Figure 5. , the shared encoding calculation module will not increase with the increase of terminals. It is a shared module like the encoding module. There is a variable v in the shared coding calculation module (0 when the system starts), which records the first initial value of the previous data block. When the data block is sent to the shared coding calculation module, the current value is calculated based on the size of the data block sent. The second initial value v' of the secondary data block (variable v' The calculation is as follows: v'=v+m/p+r, where m is the size of this data block and p is the size of an RTP video data. When m/p is divisible, r=0; when m/p is not divisible, r=1. When v' is greater than 65535, v'=v'-65536), then v' is assigned to v, the initial value v' and the data block are sent to the RTP data of all data streams that need to share the video (terminals watching the shared video) After the packet generation module, the data block will be split to generate RTP data packets. The shared encoding of the first RTP data packet generated is the initial value v'. After that, 1 is added to each RTP data packet generated. If the shared encoding number reaches 65535, starting from 0 again. This can ensure that no matter when and where you enter the meeting, or you enter the meeting and perform other actions (such as exiting the background, or watching the camera video stream, etc.), as long as you watch the data stream of the shared video, the shared RTP received will be Data packets, whose shared coding and video data are the same as those of other terminals.

In the third step, after the terminal receives the RTP data packet, it finds that the RTP data packet is a data stream of the shared video (can be judged by the SSRC field in Figure 4). The received data will be put into the cache in turn, and try to frame it. , and at the same time, put the received shared RTP data packets into a special shared cache pool (used by other terminals to obtain lost data). The shared cache pool has a priority order (sorted according to the shared encoding), and only caches 2 seconds of shared data. .

Step 4: In the previous steps, we have already introduced how terminal 1 found terminal 2 and successfully made a hole with terminal 2. If terminal m finds that the RTP data packet with packet sequence number 123 is lost (the RTP data packets that have been received have packet sequence numbers 121 and 124), by reading the extension fields of packets 121 and 124, it knows that the shared codes are 4671 and 124 respectively. 4673, so the shared code of the missing 123 packages is 4672. Terminal 1 notifies Terminal 2 through the P2P channel that Terminal 1 is missing the RTP packet with shared code 4672, as shown in Figure 3. Terminal 1 receives the RTP data packet from Terminal 2 and inserts it into the ordered buffer for decoding according to the packet sequence number.

In summary, it can be seen that the implementation of this application can achieve the following effects:

The embodiments of this application are mainly used in video conference sharing scenarios to increase the solution of obtaining lost RTP data packets from other terminals, thereby increasing the probability of terminals obtaining lost RTP data packets, and also speeding up the speed of terminals obtaining lost RTP data packets, thus Ensure the real-time and smoothness of shared videos.

Corresponding to the implementation on the first terminal side, as shown in Figure 6, the data transmission method in the embodiment of the present application is executed by the server and includes:

Step 601: Obtain the initial value of the shared encoding corresponding to the target multimedia file;

Step 602: Determine the shared encoding corresponding to each of the multiple real-time transmission protocol RTP data packets of the target multimedia file according to the initial value of the shared encoding;

Step 603: Send multiple RTP data packets of the target multimedia file to the first terminal and the second terminal;

Optionally, the obtaining the initial value of the shared encoding corresponding to the target multimedia file includes:

Determine the initial value of the shared encoding corresponding to the target multimedia file according to the initial value of the shared encoding corresponding to the first multimedia file, the size of the data block of the target multimedia file, the size of the RTP data packet, and the first coefficient;

Optionally, the method also includes:

Receive packet loss rates sent by multiple terminals;

Information about at least one terminal that meets a second condition is sent to the first terminal, and the second condition includes: the packet loss rate is less than or equal to a second preset threshold.

It should be noted that all descriptions about the server side in the above embodiments are applicable to the embodiment of the data transmission method applied to the server side, and the same technical effect can be achieved, which will not be described again here.

It should be noted that, for the method provided by the embodiments of the present application, the execution subject may be a device, or a control module in the device for executing the method. In the embodiments of this application, the device execution method is taken as an example to describe the device provided by the embodiments of this application.

As shown in Figure 7, this embodiment of the present application also provides a decoding device 700, which is applied to the first terminal and includes:

The first receiving module 701 is used to receive multiple real-time transmission protocol RTP data packets of the target multimedia file sent by the server. The protocol header of the RTP data packet carries the shared code corresponding to the RTP data packet. The shared code Used to indicate the sequence number of the RTP data packet in the target multimedia file, and the shared code corresponding to the RTP data packet is determined by the server based on the obtained initial value of the shared code corresponding to the target multimedia file;

The first sending module 702 is configured to send the shared code of the first RTP data packet to the second terminal when it is determined that the RTP data packet does not include the first RTP data packet of the target multimedia file sent by the server;

Decoding module 703, configured to receive the first RTP data packet and insert the first RTP data packet into the cache to decode the target multimedia file. The first RTP data packet is generated by the second terminal according to The shared encoding of the first RTP packet is sent;

Optionally, the first sending module 702 includes:

A determining unit configured to determine whether the first RTP data packet has not been received based on the received multiple RTP data packets of the target multimedia file sent by the server;

A sending unit, configured to send the shared code of the first RTP data packet to the second terminal when the first RTP data packet is not received.

Optionally, the determining unit is used for:

Based on the shared codes of the received multiple RTP data packets, if the shared codes of the multiple RTP data packets are discontinuous, it is determined that there is an unreceived first RTP data packet.

Optionally, the device also includes:

A connection module, configured to establish a connection with the second terminal when the packet loss rate of the first terminal meets a first condition, where the first condition includes: the packet loss rate is greater than or equal to a first preset threshold.

Optionally, the connection module includes:

A receiving unit, configured to receive information about at least one terminal sent by the server, each of the at least one terminal The packet loss rate of each terminal is less than or equal to the second preset threshold;

A selection unit configured to select a second terminal from the at least one terminal when the packet loss rate of the first terminal satisfies the first condition;

An establishing unit is configured to establish a connection with the second terminal.

It should be noted that this device embodiment is a device that corresponds one-to-one to the above-mentioned decoding method embodiment applied to the first terminal side. All implementation methods in the above-mentioned method embodiments are applicable to the embodiment of the device, and can also achieve Same technical effect.

The decoding device in the embodiment of the present application may be a device, or may be a component, integrated circuit, or chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (personal digital assistant). assistant, PDA), etc., non-mobile electronic devices can be servers, network attached storage (Network Attached Storage, NAS), personal computers (personal computers, PC), televisions (television, TV), teller machines or self-service machines, etc., this application The examples are not specifically limited.

The decoding device in the embodiment of the present application may be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of this application.

The decoding device provided by the embodiment of the present application can implement each process implemented by the method embodiment in Figure 2.

Optionally, as shown in Figure 8, this embodiment of the present application also provides a terminal 800. The terminal is a first terminal and includes a processor 801 and a memory 802. A program or instruction running on the processor 801 implements each process of the above decoding method embodiment when executed by the processor 801, and can achieve the same technical effect. To avoid duplication, it will not be described again here.

It should be noted that the terminals in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

Figure 9 is a schematic diagram of the hardware structure of a terminal that implements an embodiment of the present application.

The terminal 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, a processor 910 and other components. .

Those skilled in the art can understand that the terminal 900 may also include a power supply (such as a battery) that supplies power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby managing charging, discharging, and power consumption through the power management system. Management and other functions. The structure of the electronic device shown in Figure 9 does not constitute a limitation on the electronic device. The electronic device may include more or less components than shown in the figure, or combine certain components, or arrange different components, which will not be described again here. .

Specifically, when the terminal 900 is the first terminal, the radio frequency unit 901 is used for:

Receive multiple real-time transmission protocol RTP data packets of the target multimedia file sent by the server. The protocol header of the RTP data packet carries the shared code corresponding to the RTP data packet. The shared code is used to indicate the RTP The sequence number of the data packet in the target multimedia file, and the shared code corresponding to the RTP data packet is determined by the server based on the obtained initial value of the shared code corresponding to the target multimedia file;

Processor 910, configured to receive the first RTP data packet and insert the first RTP data packet into the cache to decode the target multimedia file. The first RTP data packet is generated by the second terminal according to The shared encoding of the first RTP packet is sent;

Optionally, the processor 910 is used to:

Determine whether the first RTP data packet has not been received based on the received multiple RTP data packets of the target multimedia file sent by the server;

The radio frequency unit 901 is used for:

If the first RTP data packet is not received, the shared code of the first RTP data packet is sent to the second terminal.

Optionally, the processor 910 is configured to determine, based on the shared codes of the received multiple RTP data packets, that there is unreceived first RTP data when the shared codes of the multiple RTP data packets are discontinuous. Bag.

Optionally, the radio frequency unit 901 is configured to establish a connection with the second terminal when the packet loss rate of the first terminal satisfies a first condition. The first condition includes: the packet loss rate is greater than or equal to the first preset threshold.

Optionally, the radio frequency unit 901 is configured to: receive information about at least one terminal sent by the server, and the packet loss rate of each terminal in the at least one terminal is less than or equal to the second preset threshold;

The processor 910 is configured to select a second terminal from the at least one terminal when the packet loss rate of the first terminal meets the first condition;

The radio frequency unit 901 is used to establish a connection with the second terminal.

In the embodiment of the present application, when it is determined that the multiple RTP data packets of the target multimedia file sent by the receiving server do not include the first RTP data packet of the target multimedia file sent by the server, the first RTP data is sent to the second terminal. According to the shared coding of the packet, the received first RTP data packet sent by the second terminal according to the shared coding of the first RTP data packet is inserted into the cache to decode the target multimedia file; in this way, repeated requests to the server can be avoided. Transmitting data packets causes the server to be overloaded and the terminal cannot obtain the lost data packets in time. By obtaining the lost data packets from other terminals, it is ensured that the terminal can obtain the complete RTP data packet of the target multimedia file in time, ensuring that the target multimedia file is It can decode and obtain the complete target multimedia file to ensure smooth playback of the target multimedia file.

It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The graphics processor 9041 is responsible for the image capture device (GPU) in the video capture mode or the image capture mode. Process the image data of still pictures or videos obtained by cameras (such as cameras). Display unit 906 A display panel 9061 may be included, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 907 includes a touch panel 9071 and other input devices 9072. Touch panel 9071, also known as touch screen. The touch panel 971 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include but are not limited to physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be described again here. Memory 909 may be used to store software programs as well as various data, including but not limited to application programs and operating systems. The processor 910 can integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above modem processor may not be integrated into the processor 910.

Embodiments of the present application also provide a readable storage medium, which stores a program or instructions. When the program or instructions are executed by a processor, each of the above-mentioned decoding method embodiments applied to the first terminal side is implemented. The process can achieve the same technical effect. To avoid repetition, it will not be described again here.

Wherein, the processor is the processor in the first terminal described in the above embodiment. The readable storage media includes computer-readable storage media, such as computer read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disks or optical disks, etc.

As shown in Figure 10, this embodiment of the present application also provides a data transmission device 1000, which is applied to a server and includes:

The acquisition module 1001 is used to obtain the initial value of the shared encoding corresponding to the target multimedia file;

Determining module 1002, configured to determine the shared encoding corresponding to each of the multiple real-time transmission protocol RTP data packets of the target multimedia file according to the initial value of the shared encoding;

The second sending module 1003 is used to send multiple RTP data packets of the target multimedia file to the first terminal and the second terminal;

Optionally, the acquisition module 1001 is used for:

Optionally, the device also includes:

The second receiving module is used to receive packet loss rates sent by multiple terminals;

The third sending module is configured to send the information of at least one terminal that meets the second condition to the first terminal. The second condition includes: the packet loss rate is less than or equal to the second preset threshold.

It should be noted that the device embodiment is a device that corresponds one-to-one to the above-mentioned data transmission method embodiment applied to the server side. All implementation methods in the above-mentioned method embodiment are applicable to the device embodiment and can achieve the same technical effect.

The data transmission device in the embodiment of the present application may be a device with an operating system. The operating system can be Android The (Android) operating system can be an ios operating system or other possible operating systems, which are not specifically limited in the embodiments of this application.

The data transmission device provided by the embodiment of the present application can implement each process implemented by the method embodiment in Figure 6.

Optionally, as shown in Figure 11, this embodiment of the present application also provides a server 1100, including a processor 1101, a memory 1102, and programs or instructions stored on the memory 1102 and executable on the processor 1101. When the program or instruction is executed by the processor 1101, each process of the above-mentioned data transmission method embodiment applied to the server side is implemented, and the same technical effect can be achieved. To avoid duplication, the details will not be described here.

An embodiment of the present application further provides a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the above data transmission method embodiment. Each process can achieve the same technical effect. To avoid duplication, it will not be described again here.

The embodiment of the present application further provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement each process of the above data transmission method embodiment, and can achieve the same technology. The effect will not be described here to avoid repetition.

It should be understood that the chips mentioned in the embodiments of this application may also be called system-on-chip, system-on-a-chip, system-on-a-chip or system-on-chip, etc.

It should be noted that, in this document, the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device that includes a series of elements not only includes those elements, It also includes other elements not expressly listed or inherent in the process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions may be performed, for example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a computer software product that is essentially or contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM, disk , optical disk), including several instructions to cause a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Inspired by this application, many forms can be made without departing from the purpose of this application and the scope protected by the claims, all of which fall within the protection of this application.

Claims

A decoding method, executed by a first terminal, includes:

Receive multiple real-time transmission protocol RTP data packets of the target multimedia file sent by the server. The protocol header of the RTP data packet carries a shared code corresponding to the RTP data packet. The shared code is used to indicate the RTP data packet. The sequence number in the target multimedia file and the shared code corresponding to the RTP data packet are determined by the server based on the obtained initial value of the shared code corresponding to the target multimedia file;

When it is determined that the RTP data packet does not include the first RTP data packet of the target multimedia file sent by the server, send the shared code of the first RTP data packet to the second terminal;

Receive the first RTP data packet and insert the first RTP data packet into the cache to decode the target multimedia file. The first RTP data packet is the second terminal's data according to the first RTP data packet. Sent by shared code;

Wherein, the first terminal and the second terminal jointly receive the RTP data packet of the target multimedia file sent by the server.
The method according to claim 1, wherein the first RTP data is sent to the second terminal when it is determined that the RTP data packet does not include the first RTP data packet of the target multimedia file sent by the server. The shared encoding of the package, including:

Determine whether the first RTP data packet has not been received based on the received multiple RTP data packets of the target multimedia file sent by the server;

If the first RTP data packet is not received, the shared code of the first RTP data packet is sent to the second terminal.
The method according to claim 2, wherein the determining whether the first RTP data packet is not received includes:

Based on the shared codes of the received multiple RTP data packets, if the shared codes of the multiple RTP data packets are discontinuous, it is determined that there is an unreceived first RTP data packet.
The method of claim 1, further comprising:

When the packet loss rate of the first terminal meets a first condition, a connection is established with the second terminal. The first condition includes: the packet loss rate is greater than or equal to a first preset threshold.
The method according to claim 4, wherein, when the packet loss rate of the first terminal meets the first condition, establishing a connection with the second terminal includes:

Receive information from at least one terminal sent by the server, and the packet loss rate of each terminal in the at least one terminal is less than or equal to a second preset threshold;

When the packet loss rate of the first terminal meets the first condition, select a second terminal from the at least one terminal;

Establish a connection with the second terminal.
A data transmission method, executed by a server, the method includes:

Get the initial value of the shared encoding corresponding to the target multimedia file;

According to the initial value of the shared encoding, determine the shared encoding corresponding to each of the multiple real-time transmission protocol RTP data packets of the target multimedia file;

Send multiple RTP data packets of the target multimedia file to the first terminal and the second terminal;

Wherein, the protocol header of the RTP data packet carries a shared code corresponding to the RTP data packet, and the shared code is used to indicate the sequence number of the RTP data packet in the target multimedia file.
The method according to claim 6, wherein said obtaining the initial value of the shared encoding corresponding to the target multimedia file includes:

Determine the initial value of the shared encoding corresponding to the target multimedia file according to the initial value of the shared encoding corresponding to the first multimedia file, the size of the data block of the target multimedia file, the size of the RTP data packet, and the first coefficient;

Wherein, the first coefficient is determined by the size of the data block of the target multimedia file and the size of the RTP data packet. The first multimedia file is a multimedia file transmitted before the target multimedia file.
The method of claim 6, further comprising:

Receive packet loss rates sent by multiple terminals;

Information about at least one terminal that meets a second condition is sent to the first terminal, and the second condition includes: the packet loss rate is less than or equal to a second preset threshold.
A decoding device, executed by a first terminal, the decoding device includes:

The first receiving module is used to receive multiple real-time transmission protocol RTP data packets of the target multimedia file sent by the server. The protocol header of the RTP data packet carries the shared code corresponding to the RTP data packet. The shared code is In order to indicate the sequence number of the RTP data packet in the target multimedia file, the shared code corresponding to the RTP data packet is determined by the server based on the obtained initial value of the shared code corresponding to the target multimedia file;

A first sending module configured to send the shared code of the first RTP data packet to the second terminal when it is determined that the RTP data packet does not include the first RTP data packet of the target multimedia file sent by the server;

a decoding module, configured to receive the first RTP data packet and insert the first RTP data packet into the cache to decode the target multimedia file, where the first RTP data packet is generated by the second terminal according to the first An RTP packet is sent in the shared encoding;

Wherein, the first terminal and the second terminal jointly receive the RTP data packet of the target multimedia file sent by the server.
The device according to claim 9, wherein the first sending module includes:

A determining unit configured to determine whether the first RTP data packet has not been received based on the received multiple RTP data packets of the target multimedia file sent by the server;

A sending unit, configured to send the shared code of the first RTP data packet to the second terminal when the first RTP data packet is not received.
The device according to claim 10, wherein the determining unit is used for:

Based on the shared codes of the received multiple RTP data packets, if the shared codes of the multiple RTP data packets are discontinuous, it is determined that there is an unreceived first RTP data packet.
The device according to claim 9, further comprising:

A connection module, configured to establish a connection with the second terminal when the packet loss rate of the first terminal meets a first condition, where the first condition includes: the packet loss rate is greater than or equal to a first preset threshold.
The device according to claim 12, wherein the connection module includes:

A receiving unit configured to receive information about at least one terminal sent by the server, where the packet loss rate of each terminal in the at least one terminal is less than or equal to the second preset threshold;

A selection unit configured to select a second terminal from the at least one terminal when the packet loss rate of the first terminal satisfies the first condition;

An establishing unit is configured to establish a connection with the second terminal.
A data transmission device, executed by a server, the data transmission device includes:

The acquisition module is used to obtain the initial value of the shared encoding corresponding to the target multimedia file;

A determination module, configured to determine the shared encoding corresponding to each of the multiple real-time transmission protocol RTP data packets of the target multimedia file according to the initial value of the shared encoding;

a second sending module, configured to send multiple RTP data packets of the target multimedia file to the first terminal and the second terminal;

Wherein, the protocol header of the RTP data packet carries a shared code corresponding to the RTP data packet, and the shared code is used to indicate the sequence number of the RTP data packet in the target multimedia file.
The device according to claim 14, wherein the acquisition module is used for:

Determine the initial value of the shared encoding corresponding to the target multimedia file according to the initial value of the shared encoding corresponding to the first multimedia file, the size of the data block of the target multimedia file, the size of the RTP data packet, and the first coefficient;

Wherein, the first coefficient is determined by the size of the data block of the target multimedia file and the size of the RTP data packet. The first multimedia file is a multimedia file transmitted before the target multimedia file.
The device of claim 14, further comprising:

The second receiving module is used to receive packet loss rates sent by multiple terminals;

The third sending module is configured to send the information of at least one terminal that meets the second condition to the first terminal. The second condition includes: the packet loss rate is less than or equal to the second preset threshold.
An electronic device, including a processor, a memory and a program or instructions stored on the memory and executable on the processor, wherein when the program or instructions are executed by the processor, the implementation as claimed in claim 1 The steps of the decoding method described in any one of -5 or the steps of implementing the data transmission method described in any one of claims 6-8.
A readable storage medium that stores programs or instructions, wherein when the program or instructions are executed by a processor, the steps or implementation of the decoding method according to any one of claims 1-5 are implemented. The steps of the data transmission method according to any one of claims 6-8.
A chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the decoding method according to any one of claims 1-5. steps or practices Now the steps of the data transmission method as described in any one of claims 6-8.
A computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the decoding method as claimed in any one of claims 1-5 or to implement the steps of claim 6- The steps of the data transmission method described in any one of 8.