CN110087140B

CN110087140B - Method, device, medium and equipment for transmitting stream media data

Info

Publication number: CN110087140B
Application number: CN201810078872.8A
Authority: CN
Inventors: 冯松佳
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2018-01-26
Filing date: 2018-01-26
Publication date: 2022-07-05
Anticipated expiration: 2038-01-26
Also published as: CN110087140A

Abstract

An embodiment of the present application provides a method, an apparatus, a medium, and a device for transmitting stream media data, where the method applied to the transmission stream media data at a transmitting end includes: determining the coding redundancy of a current second packet group to be sent according to the number of streaming media data packets and the number of redundant data packets of an adjacent last sent first packet group fed back by a receiving end; according to the coding redundancy of the second packet group, carrying out Forward Error Correction (FEC) redundancy coding on the streaming media data packet of the second packet group to obtain a redundancy data packet of the second packet group; and transmitting the streaming media data packet and the redundant data packet of the second packet group. The implementation method provided by the embodiment of the application can provide coding redundancy enough for recovering the lost streaming media data packet under the condition of occupying less network bandwidth as much as possible, and can avoid the video blockage problem caused by the loss of the streaming media data packet.

Description

Method, device, medium and equipment for transmitting stream media data

Technical Field

The present application relates to the field of multimedia transmission technologies, and in particular, to a method, an apparatus, a medium, and a device for transmitting streaming media data.

Background

This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

Streaming media refers to a media format played on the internet by adopting a streaming transmission mode, and is also called streaming media. The merchant uses a video code to be played by a video transmission server to be packaged into a data packet to be transmitted to the Internet, and a user can perform video decoding on the corresponding data packet in the Internet through a client and play the video obtained by decoding.

Currently, Real-time Transport Protocol (RTP) is mostly adopted to transmit streaming media data, so as to ensure Real-time transmission. RTP generally uses UDP (User data Protocol) to realize data transmission on an IP (Internet Protocol), and since there is no retransmission mechanism in the transmission process of UDP, loss of transmission data is easily caused, and loss of streaming media data packets in the transmission process of streaming media causes video blocking.

Disclosure of Invention

The application provides a method, a device, a medium and equipment for transmitting streaming media data, which are used for solving the problem of video blockage caused by loss of streaming media data packets in the transmission process of streaming media in the prior art.

In a first aspect, an embodiment of the present application provides a method for transmitting streaming media data, including:

determining the coding redundancy of a current second packet group to be sent according to the number of streaming media data packets and the number of redundant data packets of an adjacent last sent first packet group fed back by a receiving end;

according to the coding redundancy of the second packet group, carrying out Forward Error Correction (FEC) redundancy coding on the streaming media data packet of the second packet group to obtain a redundancy data packet of the second packet group;

and transmitting the streaming media data packet and the redundant data packet of the second packet group.

Optionally, the determining the coding redundancy of the second packet group to be currently sent specifically includes:

taking the sum of the number of the streaming media data packets and the number of the redundant data packets of the first packet group fed back by the receiving end as a first sum; and the number of the first and second groups,

taking the sum of the number of the stored streaming media data packets of the first packet group and the number of the stored redundant data packets as a second sum;

determining the packet loss rate of the first packet group according to the first sum and the second sum;

determining the product of the number of the streaming media data packets of the second packet group and the packet loss rate of the first packet group;

and taking the result of rounding up the product as the coding redundancy of the second packet group.

Optionally, the determining the packet loss rate of the first packet group specifically includes:

determining a difference between the first sum and the second sum;

and dividing the absolute value of the difference by the second sum to obtain a result, wherein the result is used as the packet loss rate of the first packet group.

Optionally, before FEC redundancy coding is performed on the streaming media data packets of the second packet group, the method further includes:

and determining that the number of the stored streaming media data packets of the second packet group is equal to a preset value.

and performing 0 complementing operation on the length of the streaming media data packet with the length smaller than the preset length in the second packet group to enable the length to be equal to the preset length.

Optionally, the streaming media data packet includes an extension packet header, where the extension packet header includes a sequence number of a packet group to which the streaming media data packet belongs, an effective data length of the streaming media data packet, a sequence number of the streaming media data packet in the packet group to which the streaming media data packet belongs, and a distinguishing identifier of the streaming media data packet; the redundant data packet comprises an extension packet header, and the extension packet header comprises a serial number of a packet group to which the redundant data packet belongs, an effective data length of the redundant data packet, a serial number of the redundant data packet in the packet group to which the redundant data packet belongs, and a distinguishing identifier of the redundant data packet.

In a second aspect, an embodiment of the present application provides an apparatus for transmitting streaming media data, including:

the first determining module is used for determining the coding redundancy of the current second packet group to be sent according to the streaming media data packet number and the redundancy data packet number of the adjacent last sent first packet group fed back by the receiving end;

the coding module is used for carrying out Forward Error Correction (FEC) redundant coding on the streaming media data packets of the second packet group according to the coding redundancy of the second packet group to obtain redundant data packets of the second packet group;

and the sending module is used for sending the streaming media data packet and the redundant data packet of the second packet group.

In a third aspect, an embodiment of the present application provides a non-volatile computer storage medium, where the computer storage medium stores an executable program, and the executable program is executed by a processor to implement any of the steps of the method for transmitting streaming media data provided in the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored on the memory, and when the processor executes the program, the processor implements the steps of any method for transmitting streaming media data provided in the first aspect.

In a fifth aspect, an embodiment of the present application provides a method for transmitting streaming media data, including:

receiving a streaming media data packet and a redundant data packet of a packet group sent by a sending end, wherein the redundant data packet is obtained after Forward Error Correction (FEC) redundant coding;

determining the number of received streaming media data packets and the number of received redundant data packets of the packet group;

and feeding back the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group to the sending end.

Optionally, the feeding back the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group to the sending end specifically includes:

adding a first extension field, a second extension field and a third extension field in a receiver report RR packet of a real-time control protocol RTCP;

writing the sequence number of the packet group into the first extension field, writing the number of the streaming media data packets of the packet group into the second extension field, and writing the number of the redundant data packets of the packet group into the third extension field to obtain a write-in RR packet;

and feeding back the written RR packet to the transmitting end.

Optionally, the streaming media data packet includes an extension packet header, where the extension packet header includes a sequence number of a packet group to which the streaming media data packet belongs and a distinguishing identifier of the streaming media data packet, and the redundant data packet includes an extension packet header, where the extension packet header includes a sequence number of a packet group to which the redundant data packet belongs and a distinguishing identifier of the redundant data packet, and then the receiving end receives the streaming media data packet and the redundant data packet of the packet group sent by the sending end, which specifically includes:

responding to the data packet arrival notice, and determining whether the data packet is a streaming media data packet or a redundant data packet according to the distinguishing identification in the extension packet head of the data packet;

if the data packet is a stream media data packet, storing the stream media data packet to a first linked list buffer area;

and if the data packet is a redundant data packet, storing the redundant data packet to a second linked list cache region.

Optionally, the determining the number of received streaming media data packets and the number of redundant data packets of the packet group specifically includes:

determining the number of streaming media data packets of which the extended packet headers contain the sequence numbers of the packet groups in the first linked list buffer area, and,

and determining the number of redundant data packets of which the extended packet headers contain the sequence numbers of the packet groups in the buffer area of the second linked list.

and if the sum of the determined number of the streaming media data packets and the number of the redundant data packets of the packet group is not less than a preset value, or if the first linked list buffer area has no available space, feeding back the determined number of the streaming media data packets and the number of the redundant data packets of the packet group to the transmitting end.

Optionally, the extended header of the streaming media data packet further includes an effective data length of the streaming media data packet and a sequence number of the streaming media data packet in the packet group to which the streaming media data packet belongs, and the extended header of the redundant data packet further includes an effective data length of the redundant data packet and a sequence number of the redundant data packet in the packet group to which the redundant data packet belongs, then the method further includes:

storing the streaming media data packets of the packet group stored in the first linked list cache region and the redundant data packets of the packet group stored in the second linked list cache region into a third linked list cache region;

removing the extended packet headers of the streaming media data packets and the redundant data packets in the third linked list buffer area;

and performing FEC decoding processing on the packet group according to the extension packet headers of all the streaming media data packets and the extension packet headers of the redundant data packets in the third linked list buffer area, and the streaming media data packets and the redundant data packets after the extension packet headers are removed.

Optionally, the method for transmitting streaming media data provided in the embodiment of the present application further includes:

deleting the streaming media data packets which are processed by the FEC decoding in the first linked list buffer, and,

deleting the redundant data packets which are subjected to the FEC decoding processing in the second linked list buffer area; and the number of the first and second groups,

and clearing the buffer area of the third linked list.

In a sixth aspect, an embodiment of the present application provides an apparatus for transmitting streaming media data, including:

the receiving module is used for receiving a streaming media data packet and a redundant data packet of a packet group sent by a sending end, wherein the redundant data packet is obtained after Forward Error Correction (FEC) redundant coding;

a determining module, configured to determine the number of received streaming media data packets and the number of redundant data packets of the packet group;

and the feedback module is used for feeding back the determined streaming media data packet number and the determined redundant data packet number of the packet group to the sending end.

In a seventh aspect, an embodiment of the present application provides a non-volatile computer storage medium, where the computer storage medium stores an executable program, and the executable program is executed by a processor to implement any of the steps of the method for transmitting streaming media data provided in the fifth aspect.

In an eighth aspect, an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored on the memory, and when the processor executes the computer program, the processor implements the steps of any method for transmitting streaming media data provided in the fifth aspect.

In a ninth aspect, an embodiment of the present application provides a system for transmitting streaming media data, including:

the receiving end is used for receiving a streaming media data packet and a redundant data packet of a first packet group sent by the sending end, wherein the redundant data packet is obtained after Forward Error Correction (FEC) redundant coding; determining the number of received streaming media data packets and the number of received redundant data packets of the first packet group; feeding back the determined number of the streaming media data packets and the determined number of the redundant data packets of the first packet group to the transmitting end;

the transmitting end is used for determining the coding redundancy of a second packet group to be transmitted currently according to the streaming media data packet number and the redundancy data packet number of the first packet group fed back by the receiving end; according to the coding redundancy of the second packet group, carrying out FEC redundancy coding on the stream media data packet of the second packet group to obtain a redundancy data packet of the second packet group; and transmitting the streaming media data packet and the redundant data packet of the second packet group, wherein the first packet group is a last transmitted packet group adjacent to the second packet group.

The method, the device, the medium and the equipment for transmitting the streaming media data have the following beneficial effects: the coding redundancy of the current to-be-sent second packet group is determined according to the number of streaming media data packets and the number of redundant data packets of the adjacent last sent first packet group fed back by the receiving end, and after the second packet group is subjected to FEC redundant coding, the streaming media data packets and the redundant data packets of the second packet group are sent, so that the coding redundancy of the current to-be-sent packet group is dynamically adjusted, the coding redundancy enough for recovering lost streaming media data packets can be provided under the condition of occupying less network bandwidth as much as possible, and the video blocking problem caused by the loss of the streaming media data packets can be avoided.

Drawings

The above and other objects, features and advantages of exemplary embodiments of the present application will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present application are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:

fig. 1 is a schematic view of an application scenario according to an embodiment of the present application;

fig. 2 is a schematic flowchart of a method for transmitting streaming media data to a transmitting end according to an embodiment of the present application;

fig. 3 is a flowchart illustrating a method for determining coding redundancy of a second packet group according to an embodiment of the present application;

fig. 4 is a schematic diagram of a format of a data packet according to an embodiment of the present application;

fig. 5 is a schematic diagram of an architecture of a transmitting end according to an embodiment of the present application;

fig. 6 is a schematic diagram of a format of a video encoding data packet according to an embodiment of the present application;

fig. 7 is a flowchart illustrating a method for transmitting streaming media data to a receiving end according to an embodiment of the present disclosure;

fig. 8 is a flowchart illustrating a method for feeding back a number of streaming media packets and a number of redundant packets to a transmitting end according to an embodiment of the present application;

fig. 9 is a flowchart illustrating a method for receiving a streaming media data packet and a redundant data packet sent by a sending end according to an embodiment of the present application;

fig. 10 is a schematic flowchart of a method for FEC redundancy decoding processing performed by a receiving end according to an embodiment of the present application;

fig. 11 is a schematic diagram of an apparatus for transmitting streaming media data applied to a transmitting end according to an embodiment of the present application;

fig. 12 is a schematic diagram of an apparatus for transmitting streaming media data at a receiving end according to an embodiment of the present disclosure;

fig. 13 is a hardware structural diagram of a computing device for implementing a method for transmitting streaming media data at a transmitting end according to an embodiment of the present application;

fig. 14 is a hardware structural diagram of a computing device for implementing a method for transmitting streaming media data at a receiving end according to an embodiment of the present application.

Detailed Description

In the prior art, in order to alleviate the video deadlock problem caused by the loss of the streaming media data packet, an FEC (Forward error Correction) redundant data packet may be added when the streaming media is transmitted, so that the streaming media receiving end recovers the streaming media data packet lost in the streaming media transmission process through the FEC redundant data packet and the received streaming media data packet. However, the inventor finds that, in the existing FEC redundant coding technology, the coding redundancy of FEC is a fixed value, and under the condition of a better network environment, all lost streaming media data packets may be recovered by using a coding redundancy smaller than the fixed value, and at this time, if a fixed value is used, a problem of occupying a higher network bandwidth may be caused; under the condition of poor network environment, the number of lost streaming media data packets is large, and all the lost streaming media data packets can be recovered only by using coding redundancy which is larger than a fixed value, and at this time, if the fixed value is used, the streaming media data packets are still lost.

To this end, an embodiment of the present application provides a method for transmitting streaming media data, where the method may include: determining the coding redundancy of a current second packet group to be sent according to the number of streaming media data packets and the number of redundant data packets of an adjacent last sent first packet group fed back by a receiving end; according to the coding redundancy of the second packet group, carrying out Forward Error Correction (FEC) redundancy coding on the streaming media data packet of the second packet group to obtain a redundancy data packet of the second packet group; and transmitting the streaming media data packet and the redundant data packet of the second packet group.

The method for transmitting streaming media data provided by the embodiment of the application can determine the coding redundancy of the current second packet group to be transmitted according to the number of streaming media data packets and the number of redundant data packets of the adjacent last transmitted first packet group fed back by the receiving end, and after FEC redundant coding is performed on the second packet group, the streaming media data packets and the redundant data packets of the second packet group are transmitted, so that the coding redundancy of the current packet group to be transmitted is dynamically adjusted, the coding redundancy enough for recovering lost streaming media data packets can be provided under the condition of occupying less network bandwidth as much as possible, and the video blocking problem caused by the loss of the streaming media data packets can be avoided.

An application scenario of the scheme for transmitting streaming media data provided in the embodiment of the present application is described below with reference to fig. 1, as shown in fig. 1, including a receiving end 101 and a sending end 102, where the receiving end 101 receives a streaming media data packet and a redundant data packet of a packet group sent by the sending end 102; determining the number of the received streaming media data packets and the number of the received redundant data packets; and feeding back the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group to the transmitting end 102. The sending end 102 determines the coding redundancy of the current second packet group to be sent according to the number of streaming media data packets and the number of redundant data packets of the adjacent last sent first packet group fed back by the receiving end 101; according to the coding redundancy of the second packet group, carrying out FEC redundancy coding on the streaming media data packet of the second packet group to obtain a redundancy data packet of the second packet group; and sending the streaming media data packet and the redundant data packet of the second packet group to the receiving end 101. The receiving end 101 and the sending end 102 can communicate through a communication network such as a local area network, a wide area network or a mobile internet; the receiving end and the transmitting end can be portable devices (such as mobile phones, tablet computers, notebook computers, etc.), and can also be Personal Computers (PCs). In some cases, the transmitting end may have the function of the receiving end, and the receiving end may have the function of the transmitting end, that is, the receiving end and the transmitting end may be interchanged. The receiving end is a client, and the sending end is a server. Fig. 1 shows an application scenario including only one receiving end, and in practical applications, multiple receiving ends may also be included, that is, the multiple receiving ends respectively perform transmission of streaming media data with the same transmitting end.

In order to make the objects, technical solutions and advantages of the present application more clear, possible embodiments of the present application will be further described below. It should be noted that the above application scenarios are only presented to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. Rather, embodiments of the present application may be applied to any scenario where applicable.

An embodiment of the present application provides a method for transmitting streaming media data, which is applied to a transmitting end, and as shown in fig. 2, the method includes:

step 201, determining the coding redundancy of the current second packet group to be sent according to the number of streaming media data packets and the number of redundant data packets of the adjacent last sent first packet group fed back by the receiving end.

The sending end sends the data packets of each packet group in sequence according to the playing sequence of the video frames corresponding to each packet group, the data packets of the packet group comprise a streaming media data packet and a redundant data packet, and specifically, the sending end sends the data packets of the packet group to the receiving end. The data packets sent by the sending end arrive at the receiving end through the transmission of the network, and the data packets arriving at the receiving end may be lost due to the uncertainty of the network environment, so that in order to adjust the coding redundancy of the current second packet group to be sent in real time, the receiving end needs to feed back the streaming media data packet number and the redundancy data packet number of the adjacent last sent first packet group to the sending end.

The first packet group is a last transmitted packet group adjacent to the second packet group, that is, the first packet group is a transmitted packet group which is located before the second packet group in the transmission sequence and adjacent to the second packet group, and the second packet group is a packet group to be currently transmitted at the transmitting end.

In specific implementation, a sending end responds to an RTSP (Real Time Streaming Protocol) address access request of a receiving end, starts to collect video data, and performs video coding on the collected frame of video data to obtain video coded data corresponding to the frame of video data when the collection of one frame of video data is finished; according to a preset transmission protocol, packaging video coding data corresponding to the frame of video data into a streaming media data packet; performing FEC redundancy coding on the stream media data packet obtained by encapsulation to obtain a redundancy data packet; and forming a packet group by the streaming media data packet obtained by encapsulation and the redundant data packet obtained by coding, and sending the streaming media data packet and the redundant data packet of the packet group. The receiving end can perform corresponding FEC decoding processing and video decoding processing according to the received streaming media data packet and redundant data packet of the packet group, and play the video obtained by decoding. The preset transmission protocol may be RTP, that is, the streaming media data packet and the redundant data packet in the packet group are RTP data packets.

In specific implementation, the h.264 video coding technology may be used to perform video coding on the collected video data, and other video coding technologies may also be used to perform video coding on the collected video data, which is not limited herein.

Step 202, according to the coding redundancy of the second packet group, FEC redundancy coding is performed on the streaming media data packet of the second packet group to obtain a redundancy data packet of the second packet group.

In specific implementation, for each frame of collected video data, each pair of video coded data corresponding to the frame of video data is encapsulated to obtain a streaming media data packet, and the streaming media data packet is stored to a sending end, and specifically can be stored to a first buffer area of the sending end. In practical applications, the redundancy coding may be performed by using Reed-solomon codes (RS codes) in FEC redundancy coding, or may be performed by using Low Density Parity Check codes (LDPC), LT digital fountain codes, or Raptor codes, which are not limited herein.

Optionally, before FEC redundancy coding is performed on the streaming media data packets of the second packet group, the method further includes: and determining that the number of the stored streaming media data packets of the second packet group is equal to a preset value, where the preset value is the total number of the streaming media data packets included in the second packet group, or the preset value is a value set according to an actual application requirement, and is not limited herein.

Optionally, before FEC redundancy coding is performed on the streaming media data packets of the second packet group, the method further includes: and performing 0 complementing operation on the length of the streaming media data packet with the length smaller than the preset length in the second packet group to enable the length to be equal to the preset length.

Specifically, when the RS code in the FEC redundant coding is used for redundant coding, the lengths of the coded streaming media data packets are required to be the same, so that zero padding operation needs to be performed on the streaming media data packets in the second packet group whose lengths are smaller than the preset length, so as to make the lengths of the streaming media data packets in the second packet group the same. When the streaming media data packet is an RTP data packet, the preset length is the maximum length of the RTP data packet, and the maximum length is 1448 bytes. The zero padding operation is to pad zero after the valid data of the streaming media data packet to obtain a streaming media data packet with a preset length.

Optionally, storing the redundant data packets of the second packet group to a second buffer area of the sending end; and releasing the second cache region to save the cache space of the sending end when the streaming media data packet and the redundant data packet of the second packet group are determined to be sent completely.

Step 203, sending the streaming media data packet and the redundant data packet of the second packet group.

In specific implementation, the streaming media data packets and the redundant data packets of each packet group have a sending sequence, and serial numbers can be set for the streaming media data packets and the redundant data packets according to the sending sequence of the streaming media data packets and the redundant data packets of the packet groups, wherein the sending sequence of the streaming media data packets and the redundant data packets with smaller serial numbers is earlier, and the sending sequence of the streaming media data packets of the packet groups is earlier than that of the redundant data packets.

It should be noted that the first packet group and the second packet group are packet groups in a non-first transmission order, and when the sending end sends the packet groups in the first transmission order, the number of the streaming media data packets and the number of the redundant data packets fed back by the receiving end are not yet received, at this time, the sending end may perform FEC redundant coding on the streaming media data packets of the packet groups in the first transmission order according to a preset coding redundancy, and the preset coding redundancy may be set according to actual application requirements, which is not limited herein.

According to the embodiment of the application, the coding redundancy of the current second packet group to be sent can be determined according to the number of the streaming media data packets and the number of the redundant data packets of the adjacent last sent first packet group fed back by the receiving end, and after the second packet group is subjected to FEC redundant coding, the streaming media data packets and the redundant data packets of the second packet group are sent, so that the coding redundancy of the current packet group to be sent can be dynamically adjusted, the coding redundancy enough for recovering lost streaming media data packets can be provided under the condition of occupying less network bandwidth as much as possible, and the video blocking problem caused by the loss of the streaming media data packets can be avoided.

In the embodiment of the application, if the number of streaming media data packets and the number of redundant data packets fed back by the receiving end are too small, it is indicated that the current network environment is poor, and the determined coding redundancy of the current packet group to be sent is larger; if the number of the streaming media data packets and the number of the redundant data packets fed back by the receiving end are more, the current network environment is better, the determined coding redundancy of the current packet group to be sent is smaller, and the coding redundancy of the current packet group to be sent can be dynamically adjusted according to the number of the streaming media data packets and the number of the redundant data packets fed back by the receiving end, so that the coding redundancy enough for recovering the lost streaming media data packets can be provided under the condition of occupying less network bandwidth as much as possible.

Optionally, according to the content provided in fig. 3, the coding redundancy of the second packet group currently to be transmitted is determined:

step 301, taking the sum of the number of the streaming media data packets and the number of the redundant data packets of the first packet group fed back by the receiving end as a first sum.

In the step, the sum of the number of the streaming media data packets and the number of the redundant data packets of the first packet group fed back by the receiving end is determined, and the sum is used as the first sum, thereby providing a basis for subsequently determining the coding redundancy of the second packet group.

Step 302, using the sum of the number of the stored streaming media data packets of the first packet group and the number of the redundant data packets as a second sum.

In specific implementation, while a sending end sends a packet group, a streaming media data packet and a redundant data packet of the packet group being sent are stored in the sending end, and at this time, the stored media data packet and redundant data packet of the packet group are not lost due to the influence of a network environment. In this step, the sum of the streaming media data packet and the redundant data packet of the first packet group stored at the sending end is determined, and the sum is used as a second sum. If the data packet loss occurs when the sending end sends the first packet group, the first sum value is smaller than the second sum value.

In specific implementation, the execution sequence of step 301 and step 302 is not limited, step 302 may be executed first and then step 301 is executed, or both steps may be executed at the same time.

Step 303, determining a packet loss rate of the first packet group according to the first sum and the second sum.

In specific implementation, the difference between the first sum and the second sum may be calculated, and the result obtained by dividing the absolute value of the difference by the second sum is used as the packet loss rate of the first packet group.

Step 304, determining the product of the number of the streaming media data packets of the second packet group and the packet loss rate of the first packet group.

Step 305, taking the result of rounding up the product as the coding redundancy of the second packet group.

In specific implementation, the number of streaming media data packets of the second packet group is obtained, and the product of the obtained number of streaming media data packets and the packet loss rate of the first packet group is determined. Rounding up the product and taking the result of rounding up the product as the coding redundancy of the second packet group. Alternatively, the result of rounding down the product may be used as the coding redundancy of the second packet group, which is not limited herein.

Optionally, adding an extension packet header to a streaming media data packet and a redundant data packet of a packet group to be sent currently; the extended packet header of the streaming media data packet comprises a serial number of a packet group to which the streaming media data packet belongs, an effective data length of the streaming media data packet, a serial number of the streaming media data packet in the packet group to which the streaming media data packet belongs and a distinguishing identifier of the streaming media data packet; the extension packet header of the redundant data packet comprises the serial number of the packet group to which the redundant data packet belongs, the effective data length of the redundant data packet, the serial number of the redundant data packet in the packet group to which the redundant data packet belongs and the identification mark of the redundant data packet. And the sending end sends the streaming media data packet and the redundant data packet of the packet group after the extension packet header is added. The receiving end can determine the number of the received streaming media data packets and the number of the redundant data packets of the packet group according to the extension packet header. In addition, the receiving end can also carry out FEC decoding processing on the packet group according to the received streaming media data packet of the packet group and the extended packet head in the redundant data packet so as to recover the lost streaming media data packet of the packet group in the transmission process.

Specifically, the format of the data packet (including the streaming media data packet and the redundant data packet) is shown in fig. 4, and the following describes each field in fig. 4:

the Pkt _ Size field is used to fill the effective data length of the data packet, and when the data packet is a data packet after zero padding, the effective load of the data packet before zero padding is performed on the effective data of the data packet, so that a receiving end can recover the effective load of the data packet according to the Pkt _ Size field to avoid using the supplemented zero in the data packet as the effective data, and optionally, the length of the Pkt _ Size field is 2 bytes.

The Blk _ Num field is used to fill a sequence number of a packet group to which the data packet belongs, where values of the Blk _ Num field in a streaming media data packet and a redundant data packet of the same packet group are the same, values of the Blk _ Num field in a streaming media data packet and a redundant data packet of different packet groups are different, and a receiving end may distinguish the packet group to which each received data packet belongs according to the Blk _ Num field, and optionally, the length of the Blk _ Num field is 4 bytes.

The Blk _ Seq _ Num field is used to fill the sequence number of the packet within the packet group to which the packet belongs, wherein, according to the sending sequence of the data packets of the packet group from front to back, the serial number is set for each data packet in the packet group, wherein, the larger the sequence number in the packet group, the later the transmission sequence of the data packet, and the sequence number of the redundant data packet is larger than the sequence number of the streaming media data packet, and the serial numbers of the data packets are different from each other and the serial number of the next data packet is incremented by 1 on the basis of the serial number of the previous data packet, the serial number of the initial data packet can be 0 or 1, here, not limited, at this time, all data packets in the same packet group have unique sequence numbers, and the receiving end can identify the packet group after transmitting to the receiving end according to the Blk _ Seq _ Num field and the Blk _ Num field, the length of the Blk _ Seq _ Num field of the missing data packet within the packet group is optionally 1 byte.

The Pkt _ Redundancy field is used to fill the discrimination identifier of the data packet, when the discrimination identifier filled in the field is a specific value, it indicates that the data packet is a redundant data packet, and when the discrimination identifier filled in the field is an unspecified value, it indicates that the data packet is a streaming media data packet, optionally, the specific value is the coding Redundancy of the packet group to which the data packet belongs, and the specific value may also be other values, such as 0 xff.

The names of the above fields are only exemplary names, and the names of the fields may be set to other names according to the difference of the names of the fields.

The transmitting end architecture provided in the embodiment of the present application is described below with reference to fig. 5.

The architecture of the sending end in the embodiment of the application is shown in fig. 5, where the hardware architecture of the sending end is the Tiny6410 hardware architecture shown in fig. 5, the software architecture of the sending end is composed of the h.264 video coding process, the streaming media sending end process and the Linux kernel shown in fig. 5, and the Linux kernel includes a driver of a corresponding hardware module in the hardware architecture. The hardware architecture and the software architecture of the sending end are matched with each other, so that the steps of the method for transmitting the streaming media data packet by the sending end are realized. Of course, fig. 5 is only an example of an architecture applied to the transmitting end in the embodiment of the present application, and other architectures applied to the embodiment of the present application may also be used to implement the method for transmitting streaming media data by the transmitting end in the embodiment of the present application.

The following briefly introduces each module in the Tiny6410 hardware architecture in the sending end:

the Tiny6410 is a development board customized by FriendlyARM company, an S3C6410 microprocessor based on an ARM11 kernel, an OV9650 image sensor, RT3070USB WIFI, 256M DDR RAM, 256M Nand Flash and the like are integrated on the development board, and various peripheral interfaces (not shown in the figure) such as USB Host, SD/MMC, CameraIF and the like are also provided. An OV9650 image sensor is selected for a video acquisition part, and the OV9650 image sensor supports a maximum 30fps VGA (640x480) YUV mode, so that the requirement of real-time video monitoring can be basically met; because a hardware encoder (MFC) is integrated in the S3C6410 microprocessor, the CPU resource of the S3C6410 microprocessor can not be occupied during hardware encoding, the hardware encoder is selected to perform video hardware encoding on the acquired video data, and particularly, the hardware encoder can perform video hardware encoding in an H.264 Format on the acquired video data; the wireless network transmission part of the sending end selects an RT3070USB WIFI module to build a wireless AP (Access Point), and the transmission rate can reach 150Mbps at most.

The following describes the software architecture in the sender:

the H.264 video coding process responds to an RTSP address access request of a receiving end, and sends a video acquisition and video coding instruction to the H.264 video coding process through Unix Domain Socket inter-process communication; after the h.264 video coding process finishes collecting and coding one frame of video data, sending the video coding data of the frame of video data to the streaming media sending end process, the streaming media sending end process encapsulating the received video coding data into an RTP streaming media data packet, and sending the streaming media data packet by using the method for transmitting the streaming media data packet provided by the embodiment.

The format of the encoded video data packet for communication between the h.264 video encoding process and the streaming media sending end process can be shown in fig. 6, where the Command field is a 1-byte custom Command code, and the Datalen field is a 4-byte integer variable.

In the case that the h.264 video coding process sends the message to the streaming media sending end process, the Command field in fig. 6 is used to fill the message identifier of starting to send the video coding data, or the Command field is used to fill the preparation end identifier, that is, the h.264 video coding process ends sending the message to the streaming media sending end process, the Datalen field is used to fill the effective video coding data length after removing the header in the previous sent message, and the effective load field is used to carry the effective video coding data in the currently sent message.

Under the condition that the stream media sending end process sends the message to the h.264 video coding process, the Command field in fig. 6 is used for filling a successful response identifier, where the identifier is used to indicate that the stream media sending end process receives the message sent by the h.264 video coding process, or the Command field is used to fill a sending failure identifier, where the identifier is used to indicate that the stream media sending end process does not receive the message sent by the h.264 video coding process, and since the message fed back by the stream media sending end process to the h.264 video coding process does not need to carry the length and the payload of the valid video coding data, both the Datalen field and the payload field are empty.

Different identifiers filled in the Command field can be distinguished by different numerical values, and specific numerical values are not limited herein.

The H.264 Video coding process adopts a multi-thread architecture and comprises a Video4Linux2 Video acquisition thread, an H.264 hard Video coding compression thread and a Video data output thread. Data interaction among the threads is realized by adopting a Fifo (first-in first-out) mechanism, and the threads are executed in parallel.

Among them, V4L2 (i.e., Video4Linux2) is the specification of the mainstream Video capture driver in the Linux system, and Video frame data in the OV9650 image sensor can be captured by directly calling the V4L2API interface. In order to simplify the video capture operation, the V4L2 video capture thread encapsulates the V4L2 specification, which is simplified into 4 steps: (1) video acquisition initialization: calling a V4L2 command to complete the work of opening the image sensor in the graph 5, inquiring the equipment attribute of the image sensor, setting the video frame format, requesting the V4L2 drive distribution frame buffer (the frame buffer refers to a block of memory space which is separately opened for the video frame data) integrated in the Linux kernel of the graph 5, driving the frame buffer mapping to the user space (the space for running the H.264 video coding process and the streaming media sending end process in the graph 5) and the like; (2) acquiring a video frame: invoking the V4L2 command to acquire a video frame from the image sensor of fig. 5; (3) video frame enqueuing: calling the V4L2 command to complete the work of frame buffering and returning to the fifo queue; (4) video collection is closed: invoking V4L2 commands the image sensor in fig. 5 to be turned off.

The H.264 hard video coding compression thread adopts a hardware encoder in the S3C6410 microprocessor to carry out H.264 hardware coding compression, thereby reducing the occupation of CPU resources of the microprocessor and improving the video coding speed. The hardware codec provides a plurality of API interfaces for executing H.264 hardware coding compression, and the compression process comprises hardware encoder initialization, execution of H.264 hardware coding compression, generation of video coded data after H.264 hardware coding, hardware encoder release and the like.

The video data output thread is mainly responsible for interacting video coding data messages with the streaming media sending end process, and the thread mainly sends video coding data subjected to video coding to the streaming media sending end process.

The streaming media sending end process can transmit streaming media data in an H.264 coding format based on an open-source LIVE555 multimedia transmission framework. When the method is specifically implemented, a BasicTaskSchedule object and a task debugger can be created in the framework, and a use environment is set; and creating the RTSP service and the session, and adding the session into the RTSP service, thereby realizing the construction of a streaming media data transmission environment. Then, in order to implement that a stream media sending end process transmits a stream media data packet and a redundant data packet, a class (for example, may be named as a pipe stream source class) for acquiring video coded data from a video data output thread may be constructed in the framework, the class forwards the acquired video coded data to an interface (for example, named as a dowgetnextframe () interface) for notifying the class to continue acquiring the video coded data and an error detection component (for example, named as an H264FrameSource component), the error detection component analyzes whether the video coded data forwarded by the class meets an encoding specification of h.264 according to an h.264 stream structure, if so, generates a corresponding SDP description, and if not, discards the acquired video coded data. The method includes the steps that video coding data meeting coding specifications are sent to a packing component (named as an H264RTPSink component, for example), the packing component packs the video coding data into RTP streaming media data packets, the packing component sends the RTP streaming media data packets obtained through packing and conversion to an FEC redundancy coding component (named as an H264FecEncode component, for example), and the FEC redundancy coding component sends the streaming media data packets and redundancy data packets of a packet group by using the method for transmitting the streaming media data packets at a transmitting end provided by the embodiment of the present application.

Wherein, FEC redundancy coding subassembly can self-define and capsulate 3 API functions, include: 1. an API function afeclinit (RS _ K, RS _ R, FecBuf) configured to complete initialization of an FEC redundant coding structure, and allocate a corresponding redundant data packet buffer (corresponding to the second buffer) to an obtained redundant data packet, where RS _ K is a preset value, RS _ R is coding redundancy, and FecBuf is a redundant data packet buffer, where the size of the redundant data packet buffer can be dynamically set according to coding redundancy, and the size of the redundant data packet buffer is specifically a product of coding redundancy and redundant data packet length; 2. an API function AFecEnc (SourceFec, RS _ K × PktSize, PktSize) configured to perform FEC redundant coding on a streaming media data packet of a packet group, and store a generated FEC redundant data packet into an allocated redundant data packet buffer, where PktSize is an effective data length of the streaming media data packet, and RS _ K is a preset value; 3. an API function afectinit (fechandle border) configured to release the redundant data packet buffer when the streaming media data packet and the redundant data packet of a packet group are both sent.

The embodiment of the application can be applied to systems with high requirements on reliability and real-time performance, such as home security, vehicle-mounted monitoring, target tracking and the like.

An embodiment of the present application provides a method for transmitting streaming media data, which is applied to a transmitting end, and as shown in fig. 7, the method includes:

step 701, receiving a streaming media data packet and a redundant data packet of a packet group sent by a sending end, where the redundant data packet is a redundant data packet obtained after FEC redundant coding.

In specific implementation, after a sending end sends a streaming media data packet and a redundant data packet of a packet group, a corresponding receiving end can receive the streaming media data packet and the redundant data packet, and the situation of packet loss may occur in the process of sending the streaming media data packet and the redundant data packet of the packet group by the sending end under the influence of a network environment. And the redundant data packet is obtained by the sending end through FEC redundant coding according to the streaming media data packet and the coding redundancy of the packet group.

Optionally, the receiving end stores the received streaming media data packet and the redundant data packet of the packet group.

Optionally, the packet group sent by the sending end corresponds to the first packet group in step 201.

Step 702, determining the number of received streaming media data packets and the number of received redundant data packets of the packet group.

Step 703, feeding back the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group to the transmitting end.

In specific implementation, the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group may be fed back to the sending end separately, or the number of the streaming media data packets and the number of the redundant data packets of the packet group may be added to an RR (receiver report) packet of an RTCP (Real-time Control Protocol), and the number of the streaming media data packets and the number of the redundant data packets of the packet group may be fed back to the sending end through the RR packet.

Optionally, if the sum of the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group is not less than a preset value, feeding back the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group to the transmitting end, where the preset value is the same as the above-mentioned preset value.

Optionally, according to the content provided in fig. 8, feeding back the determined number of streaming media packets and the determined number of redundant packets of the packet group to the sending end:

step 801, add a first extension field, a second extension field, and a third extension field in the RR packet of RTCP.

Specifically, a first extension field, a second extension field, and a third extension word may be added to the header of the RR packet.

Step 802, writing the sequence number of the packet group into the first extension field, writing the number of the streaming media data packets of the packet group into the second extension field, and writing the number of the redundant data packets of the packet group into the third extension field to obtain a write-in RR packet.

In practical application, the sequence number of the packet group, the number of the streaming media data packets of the packet group, and the number of the redundant data packets of the packet group may be written into three extension fields, for example, the number of the streaming media data packets of the packet group may be written into the first extension field, the sequence number of the packet group may be written into the second extension field, and the number of the redundant data packets of the packet group may be written into the third extension field.

It should be noted that the sending end and the receiving end need to agree in advance on the content written in different extension fields of the RR packet, so that the sending end can accurately analyze the corresponding streaming media data packet number and the redundant data packet number after receiving the RR packet.

And step 803, feeding back the written RR packet to the sending end.

In specific implementation, the written RR packet is fed back to the sending end through the RTCP protocol, so that the sending end analyzes the number of the streaming media data packets and the number of the redundant data packets of the corresponding packet group according to the extension field written in the RR packet, and the analyzed numbers are used as the number of the streaming media data packets and the number of the redundant data packets of the packet group fed back by the receiving end.

In the embodiment of the application, the RR in the RTCP is used for feeding back the number of the streaming media data packets and the number of the redundant data packets of the packet group to the sending end, and the number of the streaming media data packets and the number of the redundant data packets of the packet group do not need to be independently sent back to the sending end, so that network resources are saved to a certain extent.

Optionally, the streaming media data packet includes an extension packet header, where the extension packet header includes a sequence number of a packet group to which the streaming media data packet belongs and a distinguishing identifier of the streaming media data packet, and the redundant data packet includes an extension packet header, where the extension packet header includes a sequence number of a packet group to which the redundant data packet belongs and a distinguishing identifier of the redundant data packet, where meanings of the sequence number of the packet group and the distinguishing identifier in the extension packet header are described above, and are not described herein again.

Alternatively, the streaming media data packets and the redundant data packets of the packet group sent by the sending end may be received according to the content provided in fig. 9:

step 901, in response to the data packet arrival notification, determining whether the data packet is a streaming media data packet or a redundant data packet according to the identification in the extension packet header of the data packet.

In specific implementation, when receiving a data packet arrival notification, the monitoring socket descriptor at the receiving end parses the identification in the extension packet header of the data packet, and executes step 902 if the identification indicates that the data packet is a streaming media data packet, and executes step 903 if the identification indicates that the data packet is a redundant data packet.

More specifically, if the sequence number of the packet group belonging to the extension packet header of the data packet is less than or equal to the sequence number of the designated packet group, the data packet is discarded, and the designated packet group is the sequence number of the packet group recently fed back to the transmitting end by the receiving end, wherein the packet group recently fed back to the transmitting end is the packet group with the largest sequence number in all the packet groups fed back to the transmitting end.

Step 902, store the streaming media data packet to the first linked list buffer.

The stream media data packets of the packet group can be stored in the first linked list buffer area according to the sequence of the stream media data packet from small to large of the sequence number of the packet group to which the stream media data packet belongs.

Step 903, storing the redundant data packet to a second linked list buffer area.

The redundant data packets of the packet group can be stored in the second linked list buffer area according to the sequence of the redundant data packets from small to large in the packet group to which the redundant data packets belong.

The first linked list buffer area and the second linked list buffer area are respectively in a linked list structure and are ordered, received streaming media data packets can be stored to the first linked list buffer area according to the sequence of the serial numbers of the received streaming media data packets in the packet group to which the received streaming media data packets belong from small to large, and received redundant data packets can be stored to the second linked list buffer area according to the sequence of the serial numbers of the received redundant data packets in the packet group to which the received streaming media data packets belong from small to large, so that fast search traversal operation can be carried out in the corresponding linked list buffer areas.

Optionally, determining the number of received streaming media data packets and the number of redundant data packets of the packet group specifically includes: and determining the number of the streaming media data packets of which the extended packet headers contain the serial numbers of the packet groups in the first linked list buffer area, and determining the number of the redundant data packets of which the extended packet headers contain the serial numbers of the packet groups in the second linked list buffer area.

In specific implementation, according to the extension packet header of the data packet, the first linked list buffer area and the second linked list buffer area are traversed respectively, so that the number of the streaming media data packets containing the sequence number of the same packet group in the first linked list buffer area and the number of the redundant data packets containing the sequence number of the same packet group in the second linked list buffer area are counted.

Optionally, if the first linked list buffer does not have an available space, feeding back the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group to the sending end.

In specific implementation, when the first linked list buffer area has no available space, the receiving end cannot continue to store the streaming media data packet, and at this time, in order to ensure that the transmitting end can determine the coding redundancy of the current packet group to be transmitted in real time, the number of the streaming media data packets and the number of the redundant data packets of the packet group can be fed back to the transmitting end.

Optionally, the extension packet header of the streaming media data packet further includes an effective data length of the streaming media data packet and a sequence number of the streaming media data packet in the packet group to which the streaming media data packet belongs, and the extension packet header of the redundant data packet further includes an effective data length of the redundant data packet and a sequence number of the redundant data packet in the packet group to which the redundant data packet belongs, where the sequence number and the effective data length in the packet group to which the extension packet header belongs are described above, and are not described herein again.

As shown in fig. 10, the method for transmitting streaming media data according to the embodiment of the present application further includes:

step 1001, storing the streaming media data packets of the packet group stored in the first linked list buffer area, and storing the redundant data packets of the packet group stored in the second linked list buffer area to a third linked list buffer area.

The third linked list buffer area is a buffer area with a linked list structure and is ordered, and the stream media data packets and the redundant data packets of the packet groups can be stored in the third linked list buffer area according to the sequence of the stream media data packets and the redundant data packets from small to large in the packet group to which the stream media data packets and the redundant data packets belong, so that fast search traversal operation can be performed in the third linked list buffer area, and the stream media data packets and the redundant data packets in the third linked list buffer area are subjected to FEC decoding processing.

Optionally, before step 1001, determining a streaming media data packet whose sequence number of a packet group in the extended packet header in the first linked list buffer is smaller than the sequence number of the packet group in step 1001; and removing the determined extension packet header of the streaming media data packet, and performing video decoding processing on the streaming media data packet after the extension packet header is removed, wherein when the video coding adopts an H.264 video coding algorithm, the video decoding correspondingly adopts an H.264 video decoding algorithm.

Step 1002, removing the extended headers of the streaming media data packets and the redundant data packets in the third linked list buffer area.

In specific implementation, since the extension packet header is an extra added packet header, the extension packet header of the streaming media data packet and the extension packet header of the redundant data packet need to be removed before FEC decoding processing is performed on the streaming media data packet and the redundant data packet.

Step 1003, performing FEC decoding processing on the packet group according to the extension packet header of each streaming media data packet and the extension packet header of the redundant data packet in the third linked list buffer area, and the streaming media data packet and the redundant data packet after removing the extension packet headers.

In specific implementation, aiming at each data packet (the data packet is a streaming media data packet or a redundant data packet) with an extension packet header removed in any packet group, analyzing the extension packet header of the data packet, and determining the packet group to which the data packet belongs according to the serial number of the packet group to which the data packet belongs in the extension packet header; determining whether the data packet is a streaming media data packet or a redundant data packet according to the distinguishing identifier in the extension packet header; determining whether the data packet is subjected to zero padding operation or not according to the effective data length of the data packet and the length of the data packet, and if so, performing zero padding operation on the data packet to obtain an original data packet; recording the sequence number of the data packet in the packet group. And determining whether the data packet is lost or not according to the serial number of each data packet in any packet group, if so, performing FEC decoding processing on any packet group according to the streaming media data packet and the redundant data packet of any packet group, of which the extension packet header is removed, and performing corresponding video decoding on the data subjected to the FEC decoding processing. When the H.264 video coding algorithm is adopted in video coding, the H.264 video decoding algorithm is correspondingly adopted in video decoding.

Optionally, after FEC video decoding processing is performed, deleting the streaming media data packet subjected to FEC decoding processing in the first linked list buffer area, and deleting the redundant data packet subjected to FEC decoding processing in the second linked list buffer area; and emptying the third linked list buffer area. This frees up the buffer space at the receiving end. Optionally, the sizes of the first linked list buffer, the second linked list buffer, and the third linked list buffer are the same, for example, each is 2 times of the product of the preset value and the size of the streaming media data packet. Of course, the sizes of the first linked list buffer, the second linked list buffer and the third linked list buffer may be different, and are not limited herein.

Optionally, under the condition that the first linked list buffer area has no available space, video decoding is performed on the streaming media data packet with the minimum serial number of the packet group to which the packet header belongs in the first linked list buffer area, the streaming media data packet with the minimum serial number of the packet group to which the packet header belongs in the first linked list buffer area is deleted, meanwhile, the corresponding redundant data packet in the second linked list buffer area is deleted, and the corresponding redundant data packet is a redundant data packet belonging to the same packet group as the streaming media data packet deleted in the first linked list buffer area.

When FEC decoding processing is performed in the embodiment of the present application, an ffplay module under an FFmpeg multimedia codec frame may be used as a decoding basis, and specifically, an FEC decoding component may be constructed under the ffplay module, so as to implement the step of transmitting streaming media data in the embodiment of the present application.

An embodiment of the present application further provides a system for transmitting streaming media data, including:

the transmitting end is used for determining the coding redundancy of a second packet group to be transmitted currently according to the streaming media data packet number and the redundancy data packet number of the first packet group fed back by the receiving end; according to the coding redundancy of the second packet group, carrying out FEC redundancy coding on the streaming media data packet of the second packet group to obtain a redundancy data packet of the second packet group; and transmitting the streaming media data packet and the redundant data packet of the second packet group, wherein the first packet group is a last transmitted packet group adjacent to the second packet group.

An embodiment of the present application further provides a device for transmitting streaming media data, which is applied to a transmitting end, as shown in fig. 11, and includes:

a first determining module 1101, configured to determine, according to the number of streaming media data packets and the number of redundant data packets of a first packet group that has been sent next to the first packet group that has been sent, a coding redundancy of a second packet group that is currently to be sent;

an encoding module 1102, configured to perform forward error correction, FEC, redundancy encoding on the streaming media data packets of the second packet group according to the encoding redundancy of the second packet group, so as to obtain redundancy data packets of the second packet group;

a sending module 1103, configured to send the streaming media data packet and the redundant data packet of the second packet group.

Optionally, the first determining module 1101 is specifically configured to:

determining a difference between the first sum and the second sum;

Optionally, the apparatus for transmitting streaming media data provided in this embodiment of the present application further includes:

a second determining module 1104, configured to determine that the number of stored streaming media data packets of the second packet group is equal to a preset value before the coding module performs FEC redundancy coding on the streaming media data packets of the second packet group.

a zero padding module 1105, configured to perform 0 padding on the length of the streaming media data packet in the second packet group that is shorter than the preset length before the coding module performs FEC redundant coding on the streaming media data packet in the second packet group, so that the length of the streaming media data packet is equal to the preset length.

An embodiment of the present application further provides a device for transmitting streaming media data, which is applied to a receiving end, as shown in fig. 12, and includes:

a receiving module 1201, configured to receive a streaming media data packet and a redundant data packet of a packet group sent by a sending end, where the redundant data packet is a redundant data packet obtained after forward error correction FEC redundant coding;

a determining module 1202, configured to determine the number of received streaming media data packets and the number of redundant data packets of the packet group;

a feedback module 1203, configured to feed back the determined number of the streaming media data packets and the number of the redundant data packets of the packet group to the sending end.

Optionally, the feedback module 1203 is specifically configured to:

and feeding the RR packet to the sending end.

Optionally, the streaming media data packet includes an extension packet header, where the extension packet header includes a sequence number of a packet group to which the streaming media data packet belongs and a distinguishing identifier of the streaming media data packet, and the redundant data packet includes an extension packet header, where the extension packet header includes a sequence number of a packet group to which the redundant data packet belongs and a distinguishing identifier of the redundant data packet, then the receiving module 1201 is specifically configured to:

Optionally, the determining module 1202 is specifically configured to:

Optionally, the feedback module 1203 is specifically configured to:

Optionally, the extended packet header of the streaming media data packet further includes an effective data length of the streaming media data packet and a sequence number of the streaming media data packet in the packet group to which the streaming media data packet belongs, and the extended packet header of the redundant data packet further includes an effective data length of the redundant data packet and a sequence number of the redundant data packet in the packet group to which the redundant data packet belongs, so that the apparatus for transmitting streaming media data provided in the embodiment of the present application further includes:

a storage module 1204, configured to store the streaming media data packet of the packet group stored in the first linked list cache region and the redundant data packet of the packet group stored in the second linked list cache region in a third linked list cache region;

a removing module 1205, configured to remove the extended header of the streaming media data packet and the redundant data packet in the third linked list buffer;

the decoding module 1206 is configured to perform FEC decoding processing on the packet group according to the extension packet header of each streaming media data packet and the extension packet header of the redundant data packet in the third linked list buffer area, and the streaming media data packet and the redundant data packet after the extension packet header is removed.

a deleting module 1207, configured to delete the streaming media data packet subjected to the FEC decoding processing in the first linked list buffer, and delete the redundant data packet subjected to the FEC decoding processing in the second linked list buffer; and emptying the third linked list buffer area.

The embodiment of the present application provides a non-volatile computer storage medium, where the computer storage medium stores an executable program, and the executable program is executed by a processor to implement any of the steps of the method for transmitting streaming media data applied to a transmitting end provided in the foregoing embodiment.

The embodiments of the present application further provide a computer device, which includes a memory, a processor, and a computer program stored in the memory, where the processor implements the steps of any method for transmitting streaming media data applied to a transmitting end provided in the foregoing embodiments when executing the program.

The computer device provided in this embodiment of the present application is configured to execute any method applied to transmit streaming media data at a sending end in the foregoing embodiments, as shown in fig. 13, and is a schematic diagram of a hardware structure of the computer device in this embodiment, where the computer device may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, and the like. Specifically, the computer device may include a memory 1301, a processor 1302, and a computer program stored on the memory, where the processor implements the steps of any one of the above embodiments of the method applied to transport stream media data at a transmitting end when executing the program. Memory 1301 may include Read Only Memory (ROM) and Random Access Memory (RAM), among other things, and provides processor 1302 with program instructions and data stored in memory 1301.

Further, the computer device described in the embodiment of the present application may further include an input device 1303, an output device 1304, and the like. The input device 1303 may include a keyboard, a mouse, a touch screen, and the like; the output device 1304 may include a Display device such as a Liquid Crystal Display (LCD), a Cathode Ray Tube (CRT), a touch screen, or the like. The memory 1301, the processor 1302, the input device 1303 and the output device 1304 may be connected by a bus or other means, and fig. 13 illustrates an example of a connection by a bus.

The processor 1302 invokes the program instructions stored in the memory 1301 and executes any one of the methods applied to the transport stream media data at the transmitting end according to the obtained program instructions provided by the above-mentioned embodiments.

The present application provides a non-volatile computer storage medium, which stores an executable program, where the executable program is executed by a processor to implement any of the steps of the method for transmitting streaming media data applied to a receiving end provided in the foregoing embodiments.

The embodiments of the present application further provide a computer device, which includes a memory, a processor, and a computer program stored in the memory, where the processor implements the steps of any method for transmitting streaming media data to a receiving end provided in the foregoing embodiments when executing the program.

The computer device provided in this embodiment of the present application is configured to execute any method for transmitting streaming media data applied to a receiving end in the foregoing embodiments, as shown in fig. 14, and is a schematic diagram of a hardware structure of the computer device in this embodiment, where the computer device may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, and the like. Specifically, the computer device may comprise a memory 1401, a processor 1402 and a computer program stored on the memory, the processor implementing the steps of any of the above embodiments of the method for transmitting streaming media data to a receiving end when executing the program. Memory 1401 may include, among other things, Read Only Memory (ROM) and Random Access Memory (RAM), and provides the processor 1402 with program instructions and data stored in the memory 1401.

Further, the computer device described in the embodiment of the present application may further include an input device 1403, an output device 1404, and the like. Input device 1403 may include a keyboard, mouse, touch screen, etc.; the output device 1404 may include a Display device such as a Liquid Crystal Display (LCD), a Cathode Ray Tube (CRT), a touch screen, or the like. The memory 1401, the processor 1402, the input device 1403, and the output device 1404 may be connected by a bus or other means, and are exemplified by a bus connection in fig. 14.

The processor 1402 calls the program instructions stored in the memory 1401 and executes any method applied to the transport stream media data at the receiving end provided by the above-mentioned embodiments according to the obtained program instructions.

It should be noted that although in the above detailed description several modules of the apparatus are mentioned which transport the streaming media data, this division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the modules described above may be embodied in one module according to embodiments of the application. Conversely, the features and functions of one module described above may be further divided into embodiments by a plurality of modules.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the present application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A method of transmitting streaming media data, comprising:

receiving a write receiver report RR packet fed back by a receiving end through a real-time control protocol RTCP; after receiving a streaming media data packet and a redundant data packet of a first packet group sent by a sending end, the receiving end sends the write RR packet to the sending end; adding a first extension field, a second extension field and a third extension field at the head of an RR packet by the receiving end, writing the sequence number of a received first packet group into the first extension field, writing the number of received streaming media data packets of the first packet group into the second extension field, and writing the number of received redundant data packets of the first packet group into the third extension field to obtain the RR packet;

determining the coding redundancy of a second packet group to be sent currently according to the number of the streaming media data packets and the number of the redundant data packets of the first packet group written in the RR packet fed back by a receiving end, and the number of the streaming media data packets and the number of the redundant data packets of the first packet group stored and sent by the sending end; wherein the first packet group is a last transmitted packet group adjacent to the second packet group;

according to the coding redundancy of the second packet group, carrying out Forward Error Correction (FEC) redundancy coding on the streaming media data packet of the second packet group to obtain a redundancy data packet of the second packet group, and storing the redundancy data packet to a second cache region of a sending end;

transmitting the streaming media data packet and the redundant data packet of the second packet group, wherein the first packet group and the second packet group are packet groups in a non-first transmission sequence;

when the streaming media data packets and the redundant data packets of the second packet group are determined to be sent completely, releasing the second cache region;

the determining the coding redundancy of the current second packet group to be sent specifically includes:

2. The method according to claim 1, wherein the determining the packet loss ratio of the first packet group specifically includes:

determining a difference between the first sum and the second sum;

3. The method of claim 1, wherein before FEC redundancy coding the streaming media data packets of the second packet group, further comprising:

4. The method of claim 1, wherein before FEC redundancy coding the streaming media data packets of the second packet group, further comprising:

5. The method according to any one of claims 1 to 4, wherein the streaming media data packet includes an extension packet header, the extension packet header includes a sequence number of a packet group to which the streaming media data packet belongs, an effective data length of the streaming media data packet, a sequence number of the streaming media data packet in the packet group to which the streaming media data packet belongs, and a distinguishing identifier of the streaming media data packet; the redundant data packet comprises an extension packet header, and the extension packet header comprises a serial number of a packet group to which the redundant data packet belongs, an effective data length of the redundant data packet, a serial number of the redundant data packet in the packet group to which the redundant data packet belongs, and a distinguishing identifier of the redundant data packet.

6. A method of transmitting streaming media data, comprising:

receiving a streaming media data packet and a redundant data packet of a first packet group sent by a sending end, wherein the redundant data packet is obtained by the sending end after Forward Error Correction (FEC) redundant coding is carried out on the streaming media data packet of the first packet group based on the coding redundancy of the first packet group, and is stored in a second cache region of the sending end, and when the streaming media data packet and the redundant data packet are determined to be sent completely, the second cache region is released;

determining the number of received streaming media data packets and the number of received redundant data packets of the first packet group;

adding a first extension field, a second extension field and a third extension field at the head of an RR packet reported by a receiver, writing a sequence number of a received first packet group into the first extension field, writing the determined number of streaming media data packets of the first packet group into the second extension field, and writing the number of redundant data packets of the first packet group into the third extension field to obtain a written RR packet;

feeding back the written RR packet to the sending end through a real-time control protocol RTCP;

if the first packet group is a packet group in a non-first transmission order, wherein the redundancy coding degree of the second packet group to be currently transmitted is: and the sending end performs rounding-up on the product of the determined packet loss rate of the first packet group and the number of the streaming media data packets of the current second packet group to be sent based on the number of the streaming media data packets of the first packet group written in the RR packet and the first sum of the number of the redundant data packets, and the stored second sum of the number of the streaming media data packets of the first packet group and the number of the redundant data packets, wherein the first packet group is the last sent packet group adjacent to the second packet group.

7. The method according to claim 6, wherein the streaming media data packet includes an extension packet header, the extension packet header includes a sequence number of a packet group to which the streaming media data packet belongs and a distinguishing identifier of the streaming media data packet, the redundant data packet includes an extension packet header, the extension packet header includes a sequence number of a packet group to which the redundant data packet belongs and a distinguishing identifier of the redundant data packet, and the receiving end receives the streaming media data packet and the redundant data packet of the packet group sent by the sending end, which specifically includes:

8. The method according to claim 7, wherein the determining the number of received streaming media data packets and the number of received redundant data packets of the packet group specifically comprises:

9. The method according to claim 7, wherein the feeding back the determined number of the streaming media data packets and the determined number of the redundant data packets of the packet group to the transmitting end specifically includes:

10. The method of claim 7, wherein the extended header of the streaming media data packet further includes an effective data length of the streaming media data packet and a sequence number of the streaming media data packet in the packet group to which the streaming media data packet belongs, and wherein the extended header of the redundant data packet further includes an effective data length of the redundant data packet and a sequence number of the redundant data packet in the packet group to which the redundant data packet belongs, the method further comprising:

11. An apparatus for transmitting streaming media data, comprising:

a first determining module, configured to receive a RR packet reported by a write receiver fed back by a receiving end through a real-time control protocol RTCP, where the receiving end sends the write RR packet to a sending end after receiving a streaming media data packet and a redundant data packet of a first packet group sent by the sending end, the receiving end adds a first extension field, a second extension field, and a third extension field to a header of the RR packet, writes a sequence number of the received first packet group in the first extension field, writes the number of the received streaming media data packets of the first packet group in the second extension field, and writes the number of the received redundant data packets of the first packet group in the third extension field to obtain the write RR packet; the first determining module determines the coding redundancy of a second packet group to be sent currently according to the number of streaming media data packets and the number of redundant data packets of a first packet group written in the RR packet, and the number of streaming media data packets and the number of redundant data packets of the first packet group stored and sent by the sending end; wherein the first packet group is a last transmitted packet group adjacent to the second packet group;

the coding module is configured to perform forward error correction FEC redundancy coding on the streaming media data packet of the second packet group according to the coding redundancy of the second packet group to obtain a redundant data packet of the second packet group, and store the redundant data packet in a second buffer of the sending end;

a sending module, configured to send the streaming media data packet and the redundant data packet of the second packet group, where the first packet group and the second packet group are packet groups in a non-first sending order, and release the second buffer when it is determined that the streaming media data packet and the redundant data packet of the second packet group are sent completely;

the first determining module is specifically configured to:

12. An apparatus for transmitting streaming media data, comprising:

a receiving module, configured to receive a streaming media data packet and a redundant data packet of a first packet group sent by a sending end, where the redundant data packet is obtained by performing forward error correction, FEC, redundant coding on the streaming media data packet of the first packet group based on the coding redundancy of the first packet group by the sending end, and is stored in a second cache area of the sending end, and when it is determined that the streaming media data packet and the redundant data packet are sent completely, the second cache area is released;

a determining module, configured to determine the number of received streaming media data packets and the number of received redundant data packets of the first packet group; adding a first extension field, a second extension field and a third extension field at the head of an RR packet reported by a receiver, writing a sequence number of a received first packet group into the first extension field, writing the determined number of streaming media data packets of the first packet group into the second extension field, and writing the number of redundant data packets of the first packet group into the third extension field to obtain a written RR packet;

the feedback module is used for feeding back the written RR packet to the sending end through a real-time control protocol RTCP;

if the first packet group is a packet group in a non-first transmission order, wherein the redundancy coding degree of the second packet group to be currently transmitted is: the sending end performs upward rounding on the basis of the streaming media data packet number of the first packet group and the first sum of the redundant data packet number in the written RR packet fed back by the receiving end, and the stored second sum of the streaming media data packet number of the first packet group and the redundant data packet number, the determined product of the packet loss rate of the first packet group and the streaming media data packet number of the current second packet group to be sent, wherein the first packet group is the last sent packet group adjacent to the second packet group.

13. A system for transmitting streaming media data, comprising:

the receiving end is used for receiving a streaming media data packet and a redundant data packet of a first packet group sent by the sending end, wherein the redundant data packet is obtained after Forward Error Correction (FEC) redundant coding; determining the number of received streaming media data packets and the number of received redundant data packets of the first packet group; adding a first extension field, a second extension field and a third extension field in the head of the RR packet reported by the receiver; writing the received sequence number of the first packet group into the first extension field, writing the received number of the streaming media data packets of the first packet group into the second extension field, and writing the received number of the redundant data packets of the first packet group into the third extension field to obtain a written RR packet, and feeding back the written RR packet to the transmitting end through a real-time control protocol (RTCP);

the transmitting end is configured to determine the coding redundancy of a second packet group to be currently transmitted according to the number of streaming media data packets and the number of redundant data packets of the first packet group written in the RR packet fed back by the receiving end, and the number of streaming media data packets and the number of redundant data packets of the first packet group that are already transmitted and stored by the transmitting end; according to the coding redundancy of the second packet group, carrying out Forward Error Correction (FEC) redundancy coding on the streaming media data packet of the second packet group to obtain a redundancy data packet of the second packet group, and storing the redundancy data packet to a second cache region of a sending end;

transmitting the streaming media data packet and the redundant data packet of the second packet group, wherein the first packet group is a last transmitted packet group adjacent to the second packet group, and the first packet group and the second packet group are packet groups in a non-first transmission order;

releasing the second cache region when the streaming media data packet and the redundant data packet of the second packet group are determined to be sent completely;

14. A non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores an executable program, which when executed by a processor, implements the method of any one of claims 1 to 10.