CN116634187A - Video stream processing method, device, equipment and medium based on confluence sending mechanism - Google Patents

Video stream processing method, device, equipment and medium based on confluence sending mechanism Download PDF

Info

Publication number
CN116634187A
CN116634187A CN202310614174.6A CN202310614174A CN116634187A CN 116634187 A CN116634187 A CN 116634187A CN 202310614174 A CN202310614174 A CN 202310614174A CN 116634187 A CN116634187 A CN 116634187A
Authority
CN
China
Prior art keywords
video stream
confluence
packet
data
data packet
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310614174.6A
Other languages
Chinese (zh)
Inventor
邱硕冰
同鑫
邹智群
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Baiguoyuan Information Technology Co Ltd
Original Assignee
Guangzhou Baiguoyuan Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Baiguoyuan Information Technology Co Ltd filed Critical Guangzhou Baiguoyuan Information Technology Co Ltd
Priority to CN202310614174.6A priority Critical patent/CN116634187A/en
Publication of CN116634187A publication Critical patent/CN116634187A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/2365—Multiplexing of several video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/26616—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel for merging a unicast channel into a multicast channel, e.g. in a VOD application, when a client served by unicast channel catches up a multicast channel to save bandwidth
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4347—Demultiplexing of several video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/637—Control signals issued by the client directed to the server or network components
    • H04N21/6377—Control signals issued by the client directed to the server or network components directed to server
    • H04N21/6379—Control signals issued by the client directed to the server or network components directed to server directed to encoder, e.g. for requesting a lower encoding rate
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

The embodiment of the application provides a video stream processing method, a device, equipment and a medium based on a confluence sending mechanism, wherein the method comprises the following steps: acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams; under the condition that network transmission data meet preset conditions, splitting the converging result data obtained by the converging sending module; determining a target split video stream from the split result; and encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, wherein the original data packet and the redundant data packet are used for carrying out anti-lost recovery processing by a receiving end. According to the scheme, on the premise that multi-channel video stream data are subjected to confluence transmission, the video stream data are subjected to redundancy coding in an FEC coding mode under specific conditions, compatibility of confluence transmission and redundancy coding is achieved, packet loss resistance is improved, network transmission delay is reduced, and user experience is improved.

Description

Video stream processing method, device, equipment and medium based on confluence sending mechanism
Technical Field
The embodiment of the application relates to the technical field of video processing, in particular to a video stream processing method, device, equipment and medium based on a confluence sending mechanism.
Background
With the rapid development of internet technology, live video application technology has been rapidly popularized and popularized. The multi-anchor video house is characterized in that a plurality of anchor persons live broadcast through uploading videos, so that the multi-anchor display effect is brought to audiences. The server needs to send each video stream of the anchor to audience respectively, in this process, in order to avoid occurrence of packet loss event, or adopt coding mechanism of forward error correction, data downlink transmission is carried out by adding redundant coding data packet form, but this will cause increase of data packet, and extra occupation to network bandwidth; or a data retransmission mechanism is adopted, namely, after a packet loss event occurs, the data packet of the key frame is retransmitted based on the feedback of the receiving end, but the round trip delay of the information is introduced; and each path of video stream is independently transmitted, the occupation condition of other video streams on network bandwidth is not considered in the transmission process, so that the downlink delay is high, and the user experience is poor. In this context, there is a study to reduce the transmission delay of the downstream video packet.
In the related art, multiple paths of video streams are uniformly managed and transmitted by using a confluence transmission window, so that the multiple paths of video streams can be taken as a whole in the transmission process, the network bandwidth problem and other resource occupation problems are considered, and the fairness of each path of stream can be ensured.
However, in the related art, due to the fair use of the confluence transmission window by the video stream according to the packet receiving sequence, the FEC (Forward Error Correction ) anti-packet loss strategy based on the redundancy encoding of the single-channel video stream is not compatible, and the receiving end can only rely on the data retransmission mechanism to recover under the packet loss scene, which results in weakening the anti-packet loss capability of the downlink, and increasing the possibility of transmission congestion due to occupation of network bandwidth or other resources by the data packet retransmission, and increases the transmission delay, thereby affecting the viewing experience of the user. Therefore, how to improve the downlink packet loss resistance, reduce delay and jamming, and improve user experience is a problem to be solved in the art.
Disclosure of Invention
The embodiment of the application provides a video stream processing method, device, equipment and medium based on a confluence transmission mechanism, which solve the problems that FEC anti-packet loss strategies are incompatible and the anti-packet loss capacity is weakened due to the fact that data are transmitted through a confluence transmission window in the related technology, so that the transmission delay of video packets is increased, and the watching experience of users is affected. According to the video stream processing method based on the confluence transmission mechanism, on the premise that the multi-path video stream data are subjected to confluence transmission, the video stream data are subjected to redundancy coding in an FEC coding mode under specific conditions, so that compatibility of the confluence transmission and the redundancy coding is achieved, the packet loss resistance is improved, the network transmission delay is reduced, and the user experience is improved.
In a first aspect, an embodiment of the present application provides a video stream processing method based on a confluence sending mechanism, where the method includes:
acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams;
under the condition that the network transmission data meet the preset conditions, splitting the converging result data obtained by the converging sending module;
determining a target split video stream from the split result;
and encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, wherein the original data packet and the redundant data packet are used for carrying out anti-lost recovery processing by a receiving end.
In a second aspect, an embodiment of the present application further provides a video stream processing apparatus based on a confluence sending mechanism, where the apparatus includes:
a network transmission data acquisition unit configured to acquire network transmission data of a downlink transmission link; the converging sending module is used for converging at least two paths of video streams;
the distribution unit is configured to distribute the converging result data obtained by the converging sending module under the condition that the network transmission data meet the preset condition;
An encoding object determining unit configured to determine a target split video stream from the split result;
the encoding unit is configured to encode the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, and the original data packet and the redundant data packet are used for the receiving end to carry out anti-lost recovery processing.
In a third aspect, an embodiment of the present application further provides a video stream processing device based on a confluence sending mechanism, where the device includes:
one or more processors;
storage means for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the video stream processing method based on the confluence sending mechanism according to the embodiment of the present application.
In a fourth aspect, an embodiment of the present application further provides a storage medium storing computer executable instructions, where the computer executable instructions when executed by a computer processor are configured to perform the video stream processing method based on the confluence sending mechanism according to the embodiment of the present application.
In a fifth aspect, the embodiment of the present application further provides a computer program product, where the computer program product includes a computer program, where the computer program is stored in a computer readable storage medium, and where at least one processor of the device reads and executes the computer program from the computer readable storage medium, so that the device performs the video stream processing method based on the confluence sending mechanism according to the embodiment of the present application.
In the embodiment of the application, whether the network transmission data meet the preset condition is identified, and if so, the merging result data obtained by the merging and sending module is subjected to splitting processing, and a target splitting video stream is determined from the splitting result, so that the target splitting video stream is encoded in an FEC encoding mode, and the receiving end is subjected to packet loss prevention reduction processing. Therefore, under the condition that FEC coding is suitable for being adopted, the splitting processing can be carried out on the combined result data, so that redundancy coding can be carried out in an FEC coding mode based on the splitting processing result, and the problem of compatibility is solved. By adding the FEC coding mechanism, the anti-packet loss capability is improved compared with a data retransmission mechanism only. And the FEC coding mode is adopted, so that the network transmission delay can be reduced, and the user experience is improved.
Drawings
Fig. 1 is a schematic flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
fig. 2 is a logic schematic diagram of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
fig. 3 is a schematic flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
FIG. 4 is a logic diagram of splitting the combined result data according to an embodiment of the present application;
fig. 5 is a schematic flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
fig. 6 is a schematic flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
fig. 7 is a schematic flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
fig. 8 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
fig. 9 is a schematic flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
fig. 10 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application;
fig. 11 is a schematic structural diagram of a video stream processing device based on a confluence sending mechanism according to an embodiment of the present application;
fig. 12 is a schematic structural diagram of a video stream processing device based on a confluence sending mechanism according to an embodiment of the present application.
Detailed Description
An embodiment of the present application will be described in detail in one embodiment with reference to the accompanying drawings and examples. It should be understood that the particular embodiments described herein are illustrative only and are not limiting of embodiments of the application. It should be further noted that, for convenience of description, only some, but not all of the structures related to the embodiments of the present application are shown in the drawings.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type, and are not limited to the number of objects, such as the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.
The method, the device, the equipment and the medium for processing the video stream based on the confluence sending mechanism provided by the embodiment of the application are described in detail by an embodiment and an application scene thereof in the following with reference to the accompanying drawings.
Fig. 1 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 1, the method specifically comprises the following steps:
s101, acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams.
Firstly, the usage scenario of the scheme may be that after the service end performs merging processing on multiple paths of video streams, the merging processing result is subjected to splitting processing under the condition that a certain condition is met, and the target splitting video is encoded by using an FEC encoding mode, so as to obtain a scenario of an original data packet and a redundant data packet.
Based on the above usage scenario, it can be understood that the execution subject of the present application may be the server, which is not limited herein.
The server side can be used for receiving the multi-path video stream data, carrying out confluence processing, adopting operations such as shunting and FEC coding and the like under the condition of meeting, and finally transmitting the multi-path video stream to the receiving side.
The downlink transmission link may refer to a network link for transmitting data from a video source or a server to a user or a client, and the same transmission window and the same congestion algorithm instance are multiplexed for multiple video streams in the downlink of the multi-user video room to manage and transmit the video streams, which is called a confluence transmission window.
The network transmission data may include information such as video stream data acquired by the confluence transmitting module, the amount of video data transmitted, the packet loss condition, and the time length for receiving feedback after transmitting the video stream data packet.
A video stream may refer to a continuous stream containing both video data and audio data. It is a multimedia data stream for transmitting and presenting continuous video and audio content. A video stream may consist of a series of video frames and corresponding audio samples that are transmitted consecutively in a particular temporal order. Video data describes successive image frames, while audio data describes successive sound samples. These data are compressed and encoded by video encoding and audio encoding techniques to reduce the amount of data and improve transmission efficiency during transmission and storage.
The confluence sending module can obtain and monitor network transmission data through a network transmission protocol and judge whether a packet loss phenomenon exists or not. And then analyzing the video stream data in the video stream data, and collecting the transmitted video data quantity to judge whether the current transmission is normal or not. Finally, the buffered video data needs to be checked to ensure that the buffered video data does not exceed a set threshold.
The converging sending module can perform converging processing on the received video stream data so as to combine multiple paths of video streams into one path. In one embodiment, this may be implemented by a series of algorithms and techniques, such as real-time transmission protocols, advanced video coding, efficient video coding, and the like. Here, it will be understood that the merging and sending module may store video stream packets uploaded by multiple anchor in the same message queue, and each packet in the resulting message queue is no longer in a continuous form similar to a single video stream packet.
S102, under the condition that the network transmission data meet the preset condition, splitting the converging result data obtained by the converging sending module.
After the network transmission data is obtained, the network performance testing tool can be used for analyzing parameters such as network bandwidth, network delay, network packet loss rate and the like in the network transmission data. When the network bandwidth, the network delay, the network packet loss rate and the like reach a certain threshold, the preset condition is considered to be met, and then the splitting processing is carried out on the combined result data. The split processing at this time is a dummy split, which is a processing method performed when the multiple video streams are merged, or a processing method performed after the first merging. The false splitting may be that before the merged data packet is sent to the receiving end, the splitting operation is performed on the merged result so as to perform anti-packet loss reduction processing on the receiving end, and the merged result is not really split and is sent to the receiving end respectively. In one embodiment, the merged data packet may be buffered in a corresponding buffer area according to different video stream IDs, so that FEC encoding is performed on each splitting result, that is, the obtained queue based on the video stream ID, in the buffer area. Therefore, the data packets of each video stream ID in the buffer area are buffered according to the sequence of receiving each video stream, so that the sequence of the obtained data packets is completely consistent with that of a single video stream during stream receiving, and the video packets can be subjected to redundancy coding to generate FEC redundancy packets.
When the network bandwidth is low, if the FEC coding mechanism is adopted, the amount of data to be transmitted is further increased, which may cause network congestion. At this time, if the FEC coding scheme is adopted, the quality of video transmission may be adversely affected. Therefore, when the network bandwidth is low, the FEC coding mechanism may not be employed. In one embodiment, the bandwidth preset threshold may be set according to the bandwidth situation of the actual network environment, for example, the network bandwidth threshold may be set to 200Mbps, and when the network bandwidth exceeds this threshold, the preset condition is considered to be satisfied.
In this scheme, when the network delay is high, an FEC coding mechanism may be used, because the network delay is high, which may cause an increase in the transmission delay of the video stream, thereby affecting the instantaneity. And when the network delay is low or the delay requirement is not high, a key frame retransmission mechanism can be adopted because the transmission speed of the video stream is high. Therefore, in the case where the network delay is low, the false shunt process may not be performed. In one embodiment, the delay preset threshold may be set according to the delay condition of the actual network environment, for example, the network delay threshold may be set to 150ms, and when the network delay exceeds the threshold, the preset condition is considered to be satisfied.
The network packet loss rate may affect the integrity and definition of the video stream, and if the packet loss rate is too high, problems such as degradation of video quality and jamming may be caused. Therefore, the preset threshold of the packet loss rate may be set according to the packet loss condition of the actual network environment, for example, the threshold of the network packet loss rate may be set to be 2%, and when the network packet loss rate exceeds the threshold, the preset condition is considered to be satisfied.
The merging result data may refer to that at least two paths of video streams are put into the same queue to be sent for unified sending. Network transmission data such as network bandwidth, delay, network packet loss rate and the like can be monitored to judge whether preset conditions are met. In one embodiment, these parameters may be obtained by periodic network measurements, real-time probing and statistics, and the like, and compared to a preset threshold. And if the network transmission data meet the preset conditions, splitting the combined result data. In one embodiment, the splitting may be performed according to the video stream ID from the merging result data. For example, the merging result data is provided to include three video streams, and then three video streams 1001, 1002, and 1003 can be divided according to the video stream IDs.
S103, determining a target split video stream from the split result.
The target split video stream can be one or more sub streams obtained after splitting processing in the splitting sending module, and can be selected according to requirements. For example, the current split result includes three video streams numbered 1001, 1002, 1003, respectively. The target video stream may be a single video stream 1001; two video streams can be selected, for example, 1001 and 1002 video streams can be selected as target split video streams; alternatively, all three video streams can be selected as the target split video stream.
If the target split video stream is 1001, only the video stream will be FEC encoded, only the packets of 1001 will be used to generate FEC redundancy packets, while the packets of 1002 and 1003 will not participate in FEC encoding. For the data packets 1002 and 1003, if the packet loss phenomenon occurs, the receiving end cannot recover through the FEC redundancy packet, and can only recover through retransmission and other modes, so that the audience may encounter more problems of blocking and picture loss when watching the two video streams. For 1001 the video stream, even if a part of the data packets are lost due to FEC encoding, the video stream can be recovered by FEC redundancy packets, so that the viewer may watch the video stream more smoothly.
The 1001 and 1002 may also be FEC encoded separately so that the quality of the 1001 and 1002 video streams seen by the viewer is higher. While 1003 video stream is not FEC encoded, once there is data loss, the viewer may see the phenomenon of blocking.
If all three video streams are selected, each video stream has an independent quality assurance and error correction mechanism, and the audience can watch all three video streams smoothly.
The target streaming video stream may be determined to be encoded according to the viewing needs of the user. For example, for live-like content, if a user needs to pay attention to multiple pictures, then multiple video streams may be selected to be encoded simultaneously; if the user only focuses on one of the live pictures, only one of the video streams may be selected. The target streaming video stream may also be determined for encoding based on network bandwidth, and in one embodiment, the selected video stream may be adaptively adjusted based on network bandwidth conditions using a bandwidth adaptive algorithm. If the bandwidth is insufficient, one path of video stream can be selected for encoding; if the bandwidth is sufficient, then more video streams may be selected for encoding. For example, if the user is interested in all video streams and the bandwidth resources are sufficient, it may be chosen to encode all video streams, thereby improving overall viewing quality.
S104, encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, wherein the original data packet and the redundant data packet are used for carrying out anti-lost recovery processing by a receiving end.
The FEC encoding scheme may be a technique of adding redundant data to a transmitting end to improve data reliability. During transmission, the transmitting end may encode the data using an FEC encoder to generate an original data packet and a redundant data packet. When the receiving end receives the data packet, the FEC decoder can be used for decoding the received data, and the redundant data packet is utilized for restoring the missing data packet, so that the anti-lost data packet is realized. The FEC encoding mode may include Reed-Solomon encoding, raptorQ encoding, and LDPC encoding. The original data packets may refer to data packets after splitting the target split video stream before adopting the FEC encoding mode, and the redundant data packets are data packets for anti-packet loss recovery generated after adding redundant information to the data packets.
At the time of encoding, encoding parameters such as an encoding algorithm, encoding length, redundancy, and the like may be set. These parameters need to be selected according to the specific application scenario and network transmission conditions. If A Reed-Solomon coding method is adopted, there are A total of A datA packets, and the target streaming video stream needs to be divided into N datA packets, there are N-A redundant datA packets, and the redundancy is (N-A)/(N).
Fig. 2 is a logic schematic diagram of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 2, the confluence sending window is the confluence sending module in this embodiment. In this scheme, after the data feedback is acquired by the confluence transmitting window, the data feedback may be transferred to a congestion control module (Congestion Control ), where a congestion algorithm is included in the congestion control module and may be executed, and by executing the congestion algorithm, network parameters may be acquired. For example, parameters such as packet loss rate, delay, and bandwidth are obtained. In one embodiment, the network parameters are passed to a hybrid automatic repeat request module (Hybrid Automatic Repeat Request, HARQ), which may be used to perform on-off control of the FEC module and the ARQ module. For example, the FEC module is set to be ON, the ARQ module is set to be OFF, and then the FEC redundancy coding mechanism is adopted to carry out packet loss prevention treatment; and setting the FEC module to be OFF and setting the ARQ module to be ON, and adopting a key frame retransmission mechanism to perform anti-packet loss processing. It will be appreciated that the control module may be configured to communicate the obtained network parameters to the FEC module for the FEC module to determine the number of redundant data packets, etc.
When the FEC encoding method is used for encoding, a FEC encoding function may be used, where the FEC encoding function is used to perform FEC encoding on a set of original data packets, generate a plurality of redundant data packets, and send the redundant data packets and the original data packets to the receiving end. After receiving these data packets, the receiving end can recover the lost original data packets through a decoding algorithm. In FEC encoding using FEC encoding, some parameters, such as the size of the original data packet, the number of redundant data packets, etc., and an FEC encoder object, need to be specified. Generally, the encoder object will be created at initialization and then passed as a parameter into the FecEncode function at encoding.
After the original data packet and the redundant data packet are obtained, the redundant data packet may be inserted into a queue to be transmitted, for example, into a head of the queue to be transmitted, or into a position of the original data packet where the redundant data packet is generated. Finally, all packets in the FEC group may be marked as "to send", waiting for the sending thread to send them out.
At the receiving end, when the FEC decoder is used for decoding the received data, the received data packets can be combined according to a certain rule to obtain a decoding matrix. The decoding matrix contains the original data packet and the redundant data packet. In the decoding matrix, each packet corresponds to an element in the matrix. When the receiving end receives the data packet, the data packet is filled in the corresponding position in the matrix. When the number of received packets is insufficient, there are empty positions in the decoding matrix, which represent missing packets. The FEC decoder may then decode the decoding matrix using a decoding algorithm, such as gaussian elimination, to recover the missing data packets. In this process, the redundant data packet may provide more redundant information, increasing the likelihood of recovering the missing data packet. And combining the original data packet obtained after decoding with the restored missing data packet to obtain the complete target streaming video stream.
In the embodiment of the application, network transmission data of a downlink transmission link is acquired through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams; under the condition that the network transmission data meet the preset conditions, splitting the converging result data obtained by the converging sending module; determining a target split video stream from the split result; and encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, wherein the original data packet and the redundant data packet are used for carrying out anti-lost recovery processing by a receiving end. According to the video stream processing method based on the confluence transmission mechanism, on the premise that the multi-path video stream data are subjected to confluence transmission, the video stream data are subjected to redundancy coding in an FEC coding mode under specific conditions, compatibility of the confluence transmission and the redundancy coding is achieved, the packet loss resistance is improved, the network transmission delay is reduced, and the user experience is improved.
Fig. 3 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 3, the method specifically comprises the following steps:
s301, acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams.
S302, obtaining video stream IDs of at least two paths of video streams subjected to confluence processing by a confluence sending module.
Fig. 4 is a schematic logic diagram of splitting the splitting result data according to an embodiment of the present application, where, as shown in fig. 4, the video stream ID may refer to an identifier for uniquely identifying a video stream, and in one embodiment, may be represented in a form of numerals, letters, or the like. Different video streams need to be identified using different IDs to facilitate identification and processing during transmission. In this scheme, the video stream ID may be represented in digital form, for example, the ID of the video stream 1 may be 1001, the ID of the video stream 2 may be 1002, the ID of the video stream 3 may be 1003, and the like.
The video stream IDs may be automatically assigned by the system during transmission, and the system may automatically assign a unique ID to each video stream during transmission of the video streams to identify and track the video streams during transmission. In one embodiment, this may include obtaining the video stream ID from the video stream source and sending it to the target device in packets along with the transmitted video data. When the target device receives the video stream data, the system may identify and process the received video stream data according to the video stream ID.
S303, splitting the merging result data obtained by the merging sending module based on the video stream ID.
After the video stream ID is obtained, the parsed video stream information can be matched with the video stream ID to be shunted, and corresponding video stream data can be found. And then separating the matched video stream data from the merging result data to obtain independent video stream data. And finally, assembling the separated video stream data to generate split stream result data.
S304, determining a target split video stream from the split result.
S305, encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, wherein the original data packet and the redundant data packet are used for the receiving end to carry out anti-lost recovery processing.
In this embodiment, by splitting based on the video stream ID, a plurality of video streams may be re-split so that the input sequence of the data packet encoded by FEC is consistent with the sequence in which each video stream data packet is received separately, so that the receiving end may decode directly without any improvement after receiving the original data packet and the redundant data packet, and obtain the lost data packet under the condition of packet loss. The scheme can expand the application range of the application, does not need to improve a receiving end, and improves the packet loss resistance of multi-path video stream transmission.
Fig. 5 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 5, the method specifically comprises the following steps:
s501, acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams.
S502, obtaining video stream IDs of at least two paths of video streams subjected to confluence processing by the confluence sending module.
S503, taking out the original data packets of each path of video stream from a queue to be transmitted of the confluence transmitting module based on the video stream ID.
The queue to be transmitted may be a buffer area in the confluence transmitting module storing data packets to be transmitted, and the data packets may have been processed by encoding, encapsulation, etc. and are ready to be transmitted onto the network transmission link. In the queue to be sent, each data packet may include information such as a sending destination address, a data length, and a data content.
The original data packet can be a data packet of the video stream which is not subjected to any coding processing, and in the video processing process, the original data packet is subjected to compression, coding and other processing so as to meet the transmission and storage requirements.
All data packets in the queue to be sent can be traversed, and the data packets with the same video stream ID as the target video stream ID are searched. And sequencing the data packets with the same ID of all the target video streams according to the sequence of receiving the data packets, namely arranging the data packets according to the sequence of the data packets in a queue to be sent, and then sequentially taking out the original data packets from the sequenced data packets.
S504, the original data packets of each path of video stream are cached to FEC caches corresponding to the IDs of each path of video stream according to the stream receiving sequence.
The FEC Cache may be a buffer area, configured to buffer original packets of each path of video stream arranged according to the receiving sequence. In the confluence sending module, after the original data packets of each path of video stream are taken out from the queue to be sent, the original data packets are cached in FEC caches corresponding to the IDs of each video stream according to the receiving sequence. When performing FEC coding, the original data packets of each path of video stream are acquired from an FEC Cache to be coded so as to generate redundant data packets for anti-lost recovery processing. Therefore, the FEC Cache is one of key components for implementing the anti-packet-loss recovery process.
And the original data packet of the corresponding video stream can be taken out from a queue to be transmitted of the confluence transmitting module according to the video stream ID and stored in a buffer array according to the stream receiving sequence. And then, for each video stream ID, creating an FEC Cache, and storing the data packets in the Cache array in the FEC Cache according to the stream receiving sequence. When a new data packet arrives, it may be stored in the FEC Cache according to the order of the received streams.
S505, determining a target split video stream from the split result.
S506, the target streaming video stream is encoded by adopting an FEC encoding mode, so that an original data packet and a redundant data packet of the target streaming video stream are obtained and are used for the receiving end to carry out anti-lost recovery processing.
In this embodiment, the original data packets of each path of video stream are cached in the FEC caches corresponding to each video stream ID according to the receiving sequence, so that the received video data packets can be cached and ordered according to the receiving sequence before being FEC encoded. This facilitates the generation of error correction redundancy data during FEC encoding, thereby improving the reliability and fault tolerance of the video data. Meanwhile, the buffering and the sequencing are also beneficial to ensuring the sequence of data in the process of sending and receiving the data packets, so that the problem that the video quality is influenced due to the fact that the sequence of the data packets is disordered due to factors such as network transmission and the like is avoided.
Fig. 6 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 6, the method specifically comprises the following steps:
s601, acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams.
S602, under the condition that the network transmission data meet the preset condition, splitting the converging result data obtained by the converging sending module.
And S603, determining one or more paths of the video streams as the target split video stream from the at least two paths of video streams in response to the determination operation of the target split video stream.
One or more of the targeted split video streams may be determined by user input, device settings, or automatic policies of the system. In the case of user input or device settings, the user or device may explicitly specify the ID or other relevant information of the target streaming video stream; in the case of system automation policies, different policies may be selected according to different requirements, e.g., may be determined according to importance of the video stream, bandwidth, network conditions, etc. If video streams with numbers 1001, 1002 and 1003 exist, if the importance degree of each video stream is very high, and the network bandwidth is high and the network condition is good, each video can be selected to be encoded respectively, namely, each video has a corresponding redundant data packet, and then each video is a target split stream; if the bandwidth is low or the network condition is bad, only the video stream with higher importance degree can be selected to be encoded, and then the selected video stream with higher importance degree is taken as the target video stream.
S604, encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, wherein the original data packet and the redundant data packet are used for the receiving end to carry out anti-lost recovery processing.
In this embodiment, the video stream may be flexibly scheduled and managed according to the application scenario and the requirement in one embodiment, so as to improve the quality and efficiency of video transmission, and better satisfy the requirement of the user, and improve the effect and user experience of video transmission.
Fig. 7 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 7, the method specifically comprises the following steps:
s701, acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams.
S702, under the condition that the network transmission data meet the preset condition, splitting the converging result data obtained by the converging sending module.
S703, determining a target split video stream from the split result.
S704, encoding a preset number of original data packets of the target streaming video stream by adopting an FEC encoding mode to obtain redundant data packets of the preset number of original data packets.
The preset number of original data packets of the target split stream may be combined, and before encoding, the FEC encoding parameters may be determined, and may include the number of original data packets, the number of redundant data packets, and the size of each data packet. And then, carrying out Reed-Solomon coding on the combination of the original data packets to obtain a corresponding number of redundant data packets. The FEC encoding parameters can be adjusted in real time according to the current network condition or packet loss rate, etc. to improve the stability of the network and the video quality, which is called Refresh FEC param. In one embodiment, it means to dynamically adjust the number of coded redundant packets according to the network jitter condition and the packet loss rate so as to cope with different network packet loss conditions. For example, if the packet loss rate is high, a relatively large number of redundant packets are encoded; if the packet loss rate is low, fewer redundant packets are encoded. In the case of high network jitter, the packet loss recovery capability needs to be improved by increasing the number of redundant packets, but at the same time, the bandwidth and computational resources required for encoding and transmitting the redundant packets are also considered. If the redundant packets are set too much, network congestion may increase, reducing coding and transmission efficiency. Therefore, in the case of high network jitter, an appropriate number of redundant packets should be selected according to the network environment and the application scenario in one embodiment to achieve the best packet loss recovery effect and transmission efficiency.
S705, based on the first serial number and the second serial number of each original data packet in the preset number of original data packets, generating the first serial number and the second serial number of the redundant data packet for the receiving end to perform anti-lost recovery processing.
The first sequence number may be a media layer packet sequence number, denoted by videoSeq, for identifying the order of each packet in the video stream, unaffected by the stream, and each video stream uses a respective videoSeq. By using videoSeq, it is ensured that no collision of media layer packet numbers occurs at the time of the confluence processing by the confluence transmission module. Meanwhile, when FEC encoding is performed, each of the preset number of original data packets needs to have its own video seq, so that when the receiving end performs anti-packet-loss recovery processing, the sequence of the original data packets can be correctly recovered.
The second sequence number may be a packet sequence number associated with the FEC redundancy packet and is denoted by rtpSeq, and is mainly used for receiving-end FEC packet loss detection and recovery logic, and in the confluence sending window, a manner that rtpSeq is singly incremented every path is still adopted, so that no adaptation is required at the receiving end. In the receiving-end FEC packet loss detection and recovery logic, rtpSeq may also be used to identify the association between the FEC redundancy packet and the original data packet, so as to perform FEC packet loss recovery. For example, if there are an original packet 1001, an original packet 1002, and an original packet 1003, the videoSeq of the original packet 1001 may be 01, and rtpseq may be 01; the videoSeq of original packet 1002 may be 02 and rtpseq may be 02; the videoSeq of the original packet 1003 may be 03 and the rtpseq may be 03. The videoSeq of the redundant packet 1 may be 03 and rtpseq may be 04; the videoSeq of the redundant packet 2 may be 03 and the rtpseq may be 05.
Since the first sequence number is a media layer packet sequence number and is not affected by the stream, the first sequence number of the original packet can be directly used when generating the redundant packet. For the second sequence number, because it is mainly used for the FEC packet loss detection and recovery logic of the receiving end, the manner of single increment of rtpSeq per path is still adopted in the confluence sending window. Thus, in generating redundant data packets, certain logical relationships need to be employed to determine their second sequence numbers. For example, the second sequence number of the original data packet may be incremented one by one to obtain the second sequence number of the redundant data packet.
In this embodiment, the redundant data packet may be generated by performing FEC encoding on a preset number of original data packets, and sent to the receiving end. The receiving end can carry out packet loss recovery according to the serial number information of the original data packet and the redundant data packet and the FEC coding mode, thereby improving the reliability and stability of the video stream and reducing the data loss rate in the video stream transmission process. Meanwhile, the first serial number and the second serial number of the redundant data packets are generated based on the first serial number and the second serial number in the original data packets with preset numbers, so that the receiving end can be helped to more accurately determine which data packets need to be recovered when the packet loss recovery is carried out.
Fig. 8 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 8, the method specifically comprises the following steps:
s801, network transmission data of a downlink transmission link is obtained through a confluence sending module; the converging sending module is used for converging at least two paths of video streams.
S802, under the condition that the network transmission data meet the preset condition, splitting the merging result data obtained by the merging sending module.
S803, determining a target split video stream from the split result.
S804, the target streaming video stream is encoded by adopting an FEC encoding mode, so that an original data packet and a redundant data packet of the target streaming video stream are obtained and are used for the receiving end to carry out anti-lost recovery processing.
S805, a first priority is given to the redundant data packet, and the redundant data packet carries the first priority and enters a queue to be transmitted of the confluence transmitting module; the original data packets in the queue to be sent of the confluence sending module are endowed with a second priority, and the first priority is higher than the second priority.
The first priority may be the FEC redundancy packet generated by the encoding, giving the highest priority to enter the queue to be transmitted. The second priority may be the original data packet, giving the next highest priority to be entered into the queue to be transmitted. Since the FEC redundancy packets serve to protect the original data packets, the FEC redundancy packets are given a higher priority to ensure that they can be sent out together with the protected video packets in time.
The first priority may be given to the redundant data packets by adding a priority identifier to the data packets in the queue to be transmitted, in one embodiment a priority field may be added to each data packet, the priority identifier of the redundant data packets may be set higher than the priority identifier of the original data packet, and then they may be added to the queue to be transmitted to ensure that the redundant data packets can be sent out together with the protected original data packet in time. The corresponding redundant data packet can be received directly after the original data packet is received at the receiving end. For example, the queue to be sent may be set as a priority queue, where each packet has a corresponding priority value. When a new data packet is added to the queue, it is inserted into the appropriate position according to the size of its priority value, thereby ensuring that the data in the queue is arranged from high to low in priority. In such a queue, the priority value of the redundant data packet may be set to be the highest, and the priority value of the original data packet is set to be the next highest, so as to achieve the effect that the redundant data packet is sent out earlier than other original data packets in the queue to be sent out, so as to ensure that the redundant data packet is sent out along with the protected original data packet, so that the receiving end performs packet loss verification and recovery processing after packet loss.
In this embodiment, a higher priority is given to the FEC redundancy packet, so that it can be ensured that the data protected by FEC encoding can be transmitted preferentially, thereby ensuring the success rate of the receiving end in recovering the lost packet data, shortening the acknowledgement time of lost packet recovery, and reducing the recovery delay.
Fig. 9 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 9, the method specifically comprises the following steps:
s901, acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams.
S902, calculating network parameters based on the network transmission data through a congestion control module; wherein the network parameters include at least one of packet loss rate, round trip delay, and estimated network bandwidth.
The packet loss rate (losssrate) may refer to the ratio of the data packet that fails to reach the receiving end in the transmission process, and is an index for reflecting the packet loss condition of the link, where the estimated packet loss rates before and after the confluence represent the overall packet loss condition of the downlink. In one embodiment, the indication may be expressed in terms of a percentage, for example, 20% packet loss.
The ratio of the number of lost data packets to the number of total data packets can be calculated by counting the difference between the transmitted data packets and the received data packets, thereby calculating the packet loss rate.
Round trip delay (rtt) may refer to the time interval from the sending of data from the sender to the receiving of data from the receiver and acknowledgement. In one embodiment, this may be expressed in milliseconds, for example, with a round trip delay of 2ms.
The round trip delay may be calculated by calculating the time required to send a packet from the sender to the receiver, which may include transmission delay, processing delay, queuing delay, etc.
Estimating network bandwidth (bandwidth) may refer to being in the current networkUnder the condition, the maximum available transmission rate from the transmitting end to the receiving end. If at time t1, VS receives two video streams, the VS downlink total sending code rate isThe transmission code rate of the video stream A is +.>The transmission code rate of the video stream B is +.>The estimated bandwidth corresponding to the confluence window is +.>The estimated bandwidth value at time t1 can be calculated using the following formula:
equation 1:
equation 2:
obtained using equation 1 aboveAnd link packet loss rate loss t1 Afterwards, the FEC target redundancy degree can be obtained through table look-up >Since the FEC anti-packet loss function depends on the current downlink bandwidth, FEC can be used only when the sum of the downlink stream transmission rate and the stream FEC redundancy target rate is smaller than the estimated bandwidth.
S903, carrying out shunting processing on the merging result data obtained by the merging sending module under the condition that the network transmission data meet the preset condition.
S904, determining a target split video stream from the split result.
S905, determining the FEC redundancy target code rate of the FEC coding mode according to the network parameters.
The FEC redundancy target code rate may refer to a data transmission rate of adding some redundancy data when transmitting data in order to guarantee the reliability of data transmission. It is determined based on network parameters, and an appropriate redundancy rate is typically selected to ensure that the original data is recovered in the event of a network packet loss. The higher the redundancy rate, the lower the packet loss rate that can be recovered, but the increased bandwidth occupation of the transmission. Therefore, the balance of reliability and bandwidth occupation needs to be comprehensively considered when selecting the FEC redundancy target code rate.
If at time t1, the FEC redundancy transmission code rate of the video stream A isVideo stream A calculated using equation 1 +. >And the link packet loss rate loss at the time t1 t1 FEC target redundancy is obtained through table lookup>The transmission rate of the media packets in the current video stream a should be:
according to the FEC redundancy corresponding to the current video stream AAnd the media packet sending code rate in the current video stream AThe FEC redundancy target code rate is calculated and obtained as follows:
similarly, the FEC redundancy target code rate corresponding to the video stream B can be obtainedAnd finally, adding the FEC redundancy target code rates of the video stream A and the video stream B to obtain a confluent FEC redundancy target code rate:
s906, performing FEC coding on the target streaming video stream according to the FEC redundancy target code rate.
The target streaming video may be a video selected by a user for viewing, for example, if four live broadcast windows exist in a multi-user video room, the user may select a live broadcast window according to preference, and if the user prefers to watch a live broadcast video of a fourth window, the target streaming video is a live broadcast video of the fourth window.
To perform FEC encoding, a target streaming video stream may first be divided into a series of packets, and then the amount of redundancy information that needs to be added to the packets is calculated, typically by calculating the ratio of the target FEC redundancy target code rate to the video data transmission rate. For example, if the FEC redundancy target code rate is 0.2 and the video data transmission rate is 20Mbps, the redundancy information to be added is 4Mbps, and the calculation process is:
20Mbps×0.2＝4Mbps；
And finally, encoding the data packets according to the quantity of redundant information to be added by using an FEC encoding algorithm, wherein the FEC encoding algorithm can generate redundant data packets according to the selected FEC encoding mode.
In this embodiment, by calculating network parameters based on network transmission data, the current network condition can be known in real time, the FEC redundancy target code rate of the FEC encoding mode is determined according to the network parameters, and the number of redundant packets generated can be adjusted according to the network condition, so as to improve the stability of video streaming transmission. By performing FEC coding on the target split stream, redundant information can be added, error correction capability of video stream transmission is improved, packet loss rate is reduced, and user experience is improved.
Fig. 10 is a flow chart of a video stream processing method based on a confluence sending mechanism according to an embodiment of the present application. As shown in fig. 10, the method specifically comprises the following steps:
s1001, acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams.
S1002, determining the type of the packet loss event of the receiving end based on the network transmission data.
The type of packet loss event may include a continuous packet loss event and a burst packet loss event.
The continuous packet loss event may refer to a phenomenon that a data packet received by the receiving end is continuously lost for a plurality of times within a period of time. Such packet loss events are typically caused by network congestion, network errors, or errors in the transmission link.
The burst loss event may refer to a short loss phenomenon or a short loss phenomenon of a plurality of times in the data packet received by the receiving end. Such packet loss events are typically caused by network disruption, network jitter, channel interference, and the like.
The type of the packet loss event of the receiving end can be determined by the following steps:
step 1: obtaining the current link packet Loss rate Loss t1 Judging Loss t1 If so, executing the step 2, otherwise executing the step 6.
Step 2: judging the current Time stamp Time t1 Time is the same as the last packet loss Time stamp last If the TIME INTERVAL exceeds the CONTINUOUS TIME INTERVAL (3 s), step 3 is not exceeded, and step 5 is executed if the TIME INTERVAL exceeds the CONTINUOUS TIME INTERVAL.
Step 3: continuous packet loss statistics LossCount first accumulation
The control_loss_count_add_value (2) then determines whether the lowcount exceeds the control_loss_count_threshold (4), if so, step 4 is executed, otherwise, step 5 is executed.
Step 4: setting continuous packet loss statistics LossCount as
ContinuOUS_LOSS_COUNT_THRESHOLD, while confirming that there is a persistent packet LOSS scene, flag loss Set to True, and execute step 5.
Step 5: last packet loss timestamp Time last Refresh to Time t1 Return Flag loss Is a value of (2).
Step 6: and judging whether the current continuous packet loss statistics LossCount is 0, if so, executing the step 7, otherwise, executing the step 8.
Step 7: flag (Flag) loss Set to False, return Flag loss Is a value of (2).
Step 8: lossCount is set to LossCount-1.
The algorithm pre-judges whether the current packet loss is continuous packet loss or burst packet loss based on the past packet loss distribution of the current link, wherein the burst packet loss is considered as burst packet loss if the packet loss distribution is discrete, and the continuous packet loss is judged if the packet loss distribution is concentrated.
S1003, if it is a continuous packet loss event, executing: and under the condition that the network transmission data meet the preset condition, carrying out shunting processing on the converging result data obtained by the converging sending module.
S1004, determining a target split video stream from the split result.
S1005, the target streaming video stream is encoded by adopting an FEC encoding mode, and an original data packet and a redundant data packet of the target streaming video stream are obtained and are used for the receiving end to carry out anti-lost recovery processing.
S1006, if the packet loss event is a sudden packet loss event, the flow is ended.
If the receiving end is judged to have the sudden packet loss event, the flow can be ended, and the subsequent shunting processing can not be performed. This is because sudden packet loss events generally indicate a higher level of network congestion, a poorer network quality, and continuing the offloading process may result in more packet loss and even data corruption.
In this embodiment, the type of the packet loss event occurring at the receiving end is determined based on the network transmission data, and different processing strategies can be adopted according to the type of the packet loss event, so as to optimize the efficiency and quality of data transmission. For consecutive packet loss events, the loss of data can be reduced by the split processing. For the sudden packet loss event, the direct processing may cause serious data loss due to the large number of packet loss, so that the direct ending process can be selected to avoid data loss in one embodiment. Therefore, the stability of data transmission can be effectively improved, and the user experience is improved.
Fig. 11 is a schematic structural diagram of a video stream processing apparatus based on a confluence transmission mechanism according to an embodiment of the present application. As shown in fig. 11, the method specifically includes the following:
a network transmission data acquisition unit 1101 configured to acquire network transmission data of a downlink; the converging sending module is used for converging at least two paths of video streams;
The splitting unit 1102 is configured to split the merging result data obtained by the merging sending module under the condition that the network transmission data meets a preset condition;
an encoding object determining unit 1103 configured to determine a target split video stream from the split result;
the encoding unit 1104 is configured to encode the target split stream by adopting an FEC encoding manner, so as to obtain an original data packet and a redundant data packet of the target split stream, which are used for the receiving end to perform anti-packet loss recovery processing.
In the embodiment of the application, a network transmission data acquisition unit is used for acquiring network transmission data of a downlink transmission link; the converging sending module is used for converging at least two paths of video streams; the distribution unit is used for distributing the converging result data obtained by the converging sending module under the condition that the network transmission data meet the preset condition; the coding object determining unit is used for determining a target split video stream from the split result; the encoding unit is used for encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, and the original data packet and the redundant data packet are used for the receiving end to carry out anti-lost recovery processing. According to the video stream processing device based on the confluence transmission mechanism, on the premise that the multi-path video stream data are subjected to confluence transmission, the video stream data are subjected to redundancy coding in an FEC coding mode under specific conditions, compatibility of the confluence transmission and the redundancy coding is achieved, the packet loss resistance is improved, the network transmission delay is reduced, and the user experience is improved.
The video stream processing device based on the confluence sending mechanism provided by the embodiment of the present application can implement each process implemented by the method embodiments of fig. 1 to 10, and in order to avoid repetition, a description is omitted here.
Fig. 12 is a schematic structural diagram of a video stream processing device based on a confluence transmission mechanism according to an embodiment of the present application, where, as shown in fig. 12, the device includes a processor 1201, a memory 1202, an input device 1203 and an output device 1204; the number of processors 1201 in the device may be one or more, one processor 1201 being taken as an example in fig. 6; the processor 1201, memory 1202, input device 1203 and output device 1204 in the apparatus may be connected by a bus or other means, for example in fig. 12 by a bus connection. The memory 1202 is a computer readable storage medium, and may be used to store a software program, a computer executable program, and a module, such as program instructions/modules corresponding to a video stream processing method based on a confluence transmission mechanism in an embodiment of the present application. The processor 1201 executes various functional applications of the apparatus and data processing by executing software programs, instructions, and modules stored in the memory 1202, that is, implements the video stream processing method based on the confluence transmission mechanism described above. The input device 1203 may be used to receive entered numeric or character information and to generate key signal inputs related to user settings and function control of the apparatus. The output 1204 may include a display device such as a display screen.
The embodiment of the present application also provides a storage medium containing computer executable instructions, which when executed by a computer processor, are configured to perform a video stream processing method based on a confluence transmission mechanism described in the foregoing embodiment, where the method includes:
acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams;
under the condition that the network transmission data meet the preset conditions, splitting the converging result data obtained by the converging sending module;
determining a target split video stream from the split result;
and encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, wherein the original data packet and the redundant data packet are used for carrying out anti-lost recovery processing by a receiving end.
It should be noted that, in the embodiment of the video stream processing apparatus based on the confluence transmission mechanism, each unit and module included are only divided according to the functional logic, but not limited to the above-mentioned division, so long as the corresponding functions can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the embodiments of the present application.
In some possible embodiments, aspects of the method provided by the present application may also be implemented in the form of a program product, which includes a program code for causing a computer device to perform the steps in the method according to the various exemplary embodiments of the present application described in the present specification, when the program product is run on the computer device, for example, the computer device may perform the video stream processing method based on the confluence transmission mechanism described in the embodiment of the present application. The program product may be implemented using any combination of one or more readable media.

Claims (12)

1. A video stream processing method based on a confluence sending mechanism is characterized in that the method is executed by a server; the method comprises the following steps:
acquiring network transmission data of a downlink transmission link through a confluence transmitting module; the converging sending module is used for converging at least two paths of video streams;
under the condition that the network transmission data meet the preset conditions, splitting the converging result data obtained by the converging sending module;
determining a target split video stream from the split result;
And encoding the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, wherein the original data packet and the redundant data packet are used for carrying out anti-lost recovery processing by a receiving end.
2. The video stream processing method based on the merging and transmitting mechanism according to claim 1, wherein the splitting processing of the merging result data obtained by the merging and transmitting module includes:
acquiring video stream IDs of at least two paths of video streams subjected to confluence processing by a confluence transmitting module;
and carrying out splitting processing on the merging result data obtained by the merging sending module based on the video stream ID.
3. The video stream processing method based on the merging and transmitting mechanism according to claim 2, wherein the splitting processing of the merging result data obtained by the merging and transmitting module based on the video stream ID includes:
taking out original data packets of each path of video stream from a queue to be transmitted of the confluence transmitting module based on the video stream ID;
and caching the original data packets of each path of video stream into the FEC Cache corresponding to each video stream ID according to the stream receiving sequence.
4. The video stream processing method based on the confluence transmission mechanism according to claim 1, wherein determining the target shunting video stream from the shunting result comprises:
And responding to the determining operation of the target split stream, and determining one or more paths of the target split stream from the at least two paths of video streams.
5. The video stream processing method based on the confluence sending mechanism as claimed in claim 1, wherein the encoding the target shunting video stream by adopting an FEC encoding manner to obtain an original data packet and a redundant data packet of the target shunting video stream, for a receiving end to perform anti-packet loss recovery processing, includes:
encoding a preset number of original data packets of the target streaming video stream by adopting an FEC encoding mode to obtain redundant data packets of the preset number of original data packets;
and generating the first serial number and the second serial number of the redundant data packet based on the first serial number and the second serial number of each original data packet in the preset number of original data packets, wherein the first serial number and the second serial number are used for the receiving end to perform anti-lost recovery processing.
6. The video stream processing method based on the confluence transmission mechanism according to claim 1, wherein after obtaining the redundant data packet of the target split video stream, the method further comprises:
a first priority is given to the redundant data packet, and the redundant data packet carries the first priority and enters a queue to be transmitted of the confluence transmitting module; the original data packets in the queue to be sent of the confluence sending module are endowed with a second priority, and the first priority is higher than the second priority.
7. The video stream processing method based on the confluence transmission mechanism according to claim 1, wherein after acquiring network transmission data of a downlink transmission link by the confluence transmission module, the method further comprises:
calculating network parameters based on the network transmission data by a congestion control module; wherein the network parameters include at least one of packet loss rate, round trip delay and estimated network bandwidth;
correspondingly, the method for encoding the target streaming video stream by adopting the FEC encoding mode comprises the following steps:
determining an FEC redundancy target code rate of an FEC coding mode according to the network parameters;
and performing FEC coding on the target streaming video stream according to the FEC redundancy target code rate.
8. The video stream processing method based on the confluence transmission mechanism according to claim 1, wherein after acquiring network transmission data of a downlink transmission link by the confluence transmission module, the method further comprises:
determining the type of the packet loss event of the receiving end based on the network transmission data;
if the event is a continuous packet loss event, executing: under the condition that the network transmission data meet the preset conditions, splitting the converging result data obtained by the converging sending module;
If the event is a sudden packet loss event, ending the flow.
9. The video stream processing device based on the confluence sending mechanism is characterized in that the device is configured at a server; the device comprises:
a network transmission data acquisition unit configured to acquire network transmission data of a downlink transmission link; the converging sending module is used for converging at least two paths of video streams;
the distribution unit is configured to distribute the converging result data obtained by the converging sending module under the condition that the network transmission data meet the preset condition;
an encoding object determining unit configured to determine a target split video stream from the split result;
the encoding unit is configured to encode the target streaming video stream by adopting an FEC encoding mode to obtain an original data packet and a redundant data packet of the target streaming video stream, and the original data packet and the redundant data packet are used for the receiving end to carry out anti-lost recovery processing.
10. A video stream processing apparatus based on a confluence transmission mechanism, the apparatus comprising: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method for video stream processing based on a joint transmission mechanism of any one of claims 1 to 8.
11. A storage medium storing computer executable instructions which, when executed by a computer processor, are for performing the video stream processing method based on a confluence transmission mechanism of any one of claims 1-8.
12. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the video stream processing method based on a confluence transmission mechanism as claimed in any one of claims 1-8.
CN202310614174.6A 2023-05-26 2023-05-26 Video stream processing method, device, equipment and medium based on confluence sending mechanism Pending CN116634187A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310614174.6A CN116634187A (en) 2023-05-26 2023-05-26 Video stream processing method, device, equipment and medium based on confluence sending mechanism

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310614174.6A CN116634187A (en) 2023-05-26 2023-05-26 Video stream processing method, device, equipment and medium based on confluence sending mechanism

Publications (1)

Publication Number Publication Date
CN116634187A true CN116634187A (en) 2023-08-22

Family

ID=87612947

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310614174.6A Pending CN116634187A (en) 2023-05-26 2023-05-26 Video stream processing method, device, equipment and medium based on confluence sending mechanism

Country Status (1)

Country Link
CN (1) CN116634187A (en)

Similar Documents

Publication Publication Date Title
US7957307B2 (en) Reducing effects of packet loss in video transmissions
US7539187B2 (en) System and method for low-latency content-sensitive forward error correction
CN111800218B (en) Data stream transmission method and equipment
US7320099B2 (en) Method and apparatus for generating error correction data, and a computer-readable recording medium recording an error correction data generating program thereon
CN106341738B (en) Bandwidth calculation method, server side and system for streaming media network transmission
US6587985B1 (en) Data transmission method, data transmission apparatus, data receiving apparatus, and packet data structure
WO2017157303A1 (en) Anti-packet loss method, device and system for use in real-time communication
US9565482B1 (en) Adaptive profile switching system and method for media streaming over IP networks
US8005028B2 (en) Data communication system, data transmitting device, data transmitting method, data receiving device, and data receiving method
US8588093B2 (en) Streaming communication device, streaming communication method, and streaming communication system
EP2437421B1 (en) Method, device and communication system for retransmitting based on forward error correction
US9781488B2 (en) Controlled adaptive rate switching system and method for media streaming over IP networks
US9577682B2 (en) Adaptive forward error correction (FEC) system and method
US10230651B2 (en) Effective intra-frame refresh in multimedia communications over packet networks
KR20190027856A (en) Forward error correction of data streaming
CN109688425A (en) Live data plug-flow method
WO2003069787A2 (en) System and method for fault tolerant multimedia communication
JP2005033556A (en) Data transmitter, data transmitting method, data receiver, data receiving method
CN116634187A (en) Video stream processing method, device, equipment and medium based on confluence sending mechanism
JP3927486B2 (en) Streaming distribution apparatus, streaming distribution system, and streaming distribution method
JP5682292B2 (en) Video distribution apparatus and video distribution method
EP4024811B1 (en) Method and system for enhancing quality of experience (qoe) of video reception at receiver
CN113542685B (en) Real-time ultra-high definition video transmission method based on reliable UDP
CN116980635A (en) Live broadcast data processing method, device, computer equipment and storage medium
CN117319684A (en) Data transmission method, sending device, receiving device and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination