CN107231328B - Real-time video transmission method, device, equipment and system - Google Patents
Real-time video transmission method, device, equipment and system Download PDFInfo
- Publication number
- CN107231328B CN107231328B CN201610167888.7A CN201610167888A CN107231328B CN 107231328 B CN107231328 B CN 107231328B CN 201610167888 A CN201610167888 A CN 201610167888A CN 107231328 B CN107231328 B CN 107231328B
- Authority
- CN
- China
- Prior art keywords
- key frame
- frame
- sending
- data
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 70
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000012545 processing Methods 0.000 claims abstract description 149
- 239000000872 buffer Substances 0.000 claims abstract description 114
- 230000008672 reprogramming Effects 0.000 claims abstract description 36
- 230000008569 process Effects 0.000 claims abstract description 23
- 238000009432 framing Methods 0.000 claims description 25
- 238000012958 reprocessing Methods 0.000 claims description 10
- 230000003139 buffering effect Effects 0.000 claims description 8
- 238000010586 diagram Methods 0.000 description 14
- 238000011084 recovery Methods 0.000 description 7
- 230000006835 compression Effects 0.000 description 6
- 238000007906 compression Methods 0.000 description 6
- 230000003068 static effect Effects 0.000 description 6
- 230000009467 reduction Effects 0.000 description 3
- 101000741965 Homo sapiens Inactive tyrosine-protein kinase PRAG1 Proteins 0.000 description 2
- 102100038659 Inactive tyrosine-protein kinase PRAG1 Human genes 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/65—Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
- H04L65/612—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for unicast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23406—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving management of server-side video buffer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44004—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving video buffer management, e.g. video decoder buffer or video display buffer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/637—Control signals issued by the client directed to the server or network components
- H04N21/6375—Control signals issued by the client directed to the server or network components for requesting retransmission, e.g. of data packets lost or corrupted during transmission from server
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
- H04N21/6437—Real-time Transport Protocol [RTP]
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Detection And Prevention Of Errors In Transmission (AREA)
Abstract
The invention belongs to the technical field of streaming media data network transmission, and particularly relates to a real-time video transmission method, device, equipment and system. The real-time video transmission method comprises the following steps: the sending end encodes the video frame data into key frames or non-key frames, and sends RTP data packets to the receiving end; the receiving end processes the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end; after receiving, the sending end reassembles the key frame and modifies the sending time of the key frame and then sends the key frame to the receiving end, or retransmits the missing RTP data in the key frame to the receiving end; and repeating the steps 1-3 until all the video frame data are transmitted. According to the method, the RTP data missing in the retransmission key frame is adopted in data processing, the video quality is guaranteed in a mode of discarding incomplete non-key frame data, meanwhile, double buffer areas are adopted, the data processing efficiency is improved, and the display delay of real-time video data is reduced.
Description
Technical Field
The invention belongs to the technical field of streaming media data network transmission, and particularly relates to a real-time video transmission method, device, equipment and system.
Background
With the continuous development of network technology, people's demand for network communication services such as real-time video services, etc., which occupy a relatively large bandwidth, is continuously strengthened. Currently, a common method for transmitting Real-time video streams is to carry TS (Transport Stream) on RTP (Real-time Transport Protocol), while the bottom layer of RTP uses UDP (User Datagram Protocol) for transmission, and at present, wifi, 3G, 4G and other wireless networks are used to connect with the internet in a large amount.
In order to solve The RTP packet loss problem, The IETF (The Internet Engineering Task Force, Internet Engineering Task Force) respectively formulates two algorithms of FEC (forward error correction, also called forward error correction code) and NACK (negotiation acknowledgement, which plays a role in packet loss retransmission) to solve The problem. However, the use of FEC algorithm increases the redundancy of data and increases the bandwidth occupied by the streaming media data. The NACK algorithm introduces acknowledgment retransmissions, which can result in poor real-time performance.
According to a video compression coding algorithm commonly used in the art, a video picture is coded as an image sequence of intra-coded frames (I-frames, also known as key-frames), a plurality of forward predicted frames (P-frames) and bi-directional interpolated frames (B-frames). The key frame is usually the first frame of each GOP (Group of Pictures), which is moderately compressed and serves as a reference point for random access and can be used as a picture. The P frame is predicted from the P frame and the I frame in front of the P frame, in the process of video data transmission, due to data loss, the video cannot be displayed or displayed in a fuzzy and pause mode, if I frame data are incomplete, decoding failure can be caused, a video picture cannot be displayed, if the P frame is lost, decoding display of the later P frame can be influenced, mosaic appears on the picture, and the more the P frame is lost, the more the mosaic becomes serious. During transmission, the discarding of the B frame data packets has little influence on the decoding and displaying of the whole video.
Chinese patent application No. 201410577593.8 discloses a video transmission method, which searches for corresponding video coding parameters by obtaining current network transmission quality information, removes picture frame information according to the video coding parameters, adjusts the code rate and frame rate of the video, and transmits the video after coding. The method adjusts the coding parameters when the network quality is reduced, reduces the video quality and causes the video picture to be unclear. In addition, under the condition that the network quality changes frequently, the sending end of the method changes the video coding parameters frequently and performs corresponding coding processing, so that the processing efficiency is low, the processing delay is increased, and the user experience of the receiving end is influenced.
Disclosure of Invention
One of the objectives of the present invention is to overcome the above disadvantages, and to achieve the purpose of reducing the use of network bandwidth as much as possible in an environment where the network quality is degraded, and at the same time, to ensure the video transmission quality.
In order to solve the technical problem, the invention provides a real-time video transmission method, which comprises the following steps:
step 1: the sending end encodes the video frame data into key frames or non-key frames, and sends RTP data packets to the receiving end; the sending time of the key frame is determined by the configured interval parameter;
step 2: the receiving end processes the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end;
and step 3: after receiving, the sending end reassembles the key frame and modifies the sending time of the key frame and then sends the key frame to the receiving end, or retransmits the missing RTP data in the key frame to the receiving end;
and repeating the steps 1-3 until all the video frame data are transmitted.
The invention provides a dynamic key frame interval, and under the condition of normal network quality, the number of key frames is reduced and the total data amount of transmission is reduced by setting a larger key frame interval; and under the condition that the data processing fails due to the reduction of the network quality, sending an RTP data packet request missing in the re-programming key frame or the retransmission key frame by the receiving end. For the key frame re-encoding request, the sending end dynamically encodes the key frame and modifies the sending time of the key frame according to the request, thereby reducing the waiting time of video recovery of the receiving end and ensuring the timely recovery of video pictures.
Further, the step 1 further includes:
the sending end puts the sent RTP data packet into a sending data buffer area, and when receiving the missing RTP data request in the retransmission key frame, the sending data buffer area is searched preferentially.
By setting the data transmission buffer area, the processing efficiency of searching data when the transmitting end retransmits missing RTP data in the key frame is improved, and the processing time is shortened.
Further, the receiving end in step 2 processes the RTP data packet, specifically:
receiving RTP data packets to a received data buffer;
judging whether the RTP data packet is missing data in the key frame requested to be retransmitted or not;
if the RTP data which is not missing in the retransmission key frame is not the RTP data, normal data frame processing is carried out;
and if the RTP data is missing in the retransmission key frame, performing key frame reprocessing.
Further, the step 2 further comprises: the receiving end is provided with a waiting buffer area for buffering the RTP data packet received by the retransmission key frame.
The receiving end of the invention adopts a double-buffer mechanism, waits for the buffer to buffer the RTP data packet received by the retransmission key frame, the receiving data buffer can continue to receive the subsequent RTP data packet for framing operation, and can continuously decode and display the retransmission key frame and the subsequent video frame after the incomplete key frame is completely received, thereby reducing the real-time video delay caused by waiting for the retransmission data.
Further, the receiving end reads the RTP data packet from the receiving buffer according to the set jitter delay.
The problem of RTP data packet disorder caused in the network transmission process can be relieved by adding jitter delay.
Further, the normal data frame processing includes the following steps:
framing the RTP packets in the received data buffer area and judging whether the video frames are complete or not;
if the video frame is complete, performing video frame decoding operation, if the decoding is successful, marking the processing result as successful, otherwise, marking the processing result as failed, and sending a re-programming key frame request to the sending end;
if the video frame is not complete, further judging whether the video frame is a key frame, if the video frame is the key frame, writing the received RTP data packet into a waiting buffer area, marking the processing result as failure, sending an RTP data request missing in the retransmission key frame to a sending end, and informing the sending end of retransmitting the RTP data missing in the key frame; if the video frame is a non-key frame, discarding the video frame, judging whether the number of discarded non-key frames between two continuous key frames reaches a threshold value, if so, marking the processing result as failure, and sending a key frame reprogramming request to the sending end, otherwise, marking the processing result as success.
According to the technical scheme, under the condition that the video frame is not complete, different processing can be performed according to the type of the video frame: if the key frame is the key frame, the RTP data packet missing in the key frame is requested to be retransmitted, so that the integrity of the video picture can be ensured; if the video frame is a non-key frame and the accumulated discarded frame number is within the set threshold range, directly discarding the frame and waiting for the next video frame, so that the processing has the advantages of reducing the network overhead of reprogramming the key frame, ensuring the fluency of the video, and having only slight mosaic or small time delay on the influence of the video frame; if the number of accumulated non-key frame discarding frames reaches the threshold value, a key frame re-programming request is sent at the moment, and the definition of the video picture is ensured.
Further, the key frame reprocessing includes the following steps:
writing the RTP data packet of the received retransmission key frame into a waiting buffer area;
framing the RTP packets in the waiting buffer area, and judging whether the retransmission key frame of the waiting buffer area is complete or not;
if the retransmission key frame is complete, performing video frame decoding operation, if the decoding is successful, marking the processing result as successful, otherwise, marking the processing result as failed, and sending a key frame reprogramming request to the sending end;
if the retransmission key frame is not complete, further judging whether the waiting time is overtime or not, if the waiting time is not overtime, continuing waiting, otherwise, emptying the receiving data buffer area and the waiting buffer area, marking the processing result as failure, and sending a key frame reprogramming request to the sending end.
According to the technical scheme, the waiting overtime threshold value is set, so that the receiving end can not wait for receiving the RTP data packet in the key frame requesting for retransmission without limit to cause video picture blockage under the condition that the network quality is reduced.
Correspondingly, the invention also provides a real-time video transmission device, which comprises:
a first sending module, configured to execute step 1: the sending end encodes the video frame data into key frames or non-key frames, and sends RTP data packets to the receiving end; the sending time of the key frame is determined by the configured interval parameter;
a receiving module, configured to perform step 2: the receiving end processes the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end;
a second sending module, configured to execute step 3: after receiving, the sending end reassembles the key frame and modifies the sending time of the key frame and then sends the key frame to the receiving end, or retransmits the missing RTP data in the key frame to the receiving end;
and the circulating module is used for repeating the steps 1-3 until all the video frame data are transmitted.
Further, the receiving module includes:
a receiving unit, configured to receive an RTP packet to a received data buffer;
the judging unit is used for judging whether the RTP data packet is missing data in the key frame requested to be retransmitted or not;
the first processing unit is used for processing normal data frames if RTP data missing in the key frame is not retransmitted;
and the second processing unit is used for carrying out key frame reprocessing if the RTP data missing in the key frame is retransmitted.
Further, the receiving module further includes:
and the waiting buffer unit is used for setting a waiting buffer area at the receiving end and buffering the RTP data packet received by the retransmission key frame.
Further, the first processing unit includes:
the first judging component is used for framing the RTP packets in the received data buffer area and judging whether the video frames are complete or not;
the first decoding component is used for carrying out video frame decoding operation if the video frame is complete, identifying the processing result as successful if the decoding is successful, and otherwise identifying the processing result as failed, and sending a key frame reprogramming request to the sending end;
the first processing component is used for further judging whether the video frame is a key frame or not if the video frame is incomplete, writing the received RTP data packet into a waiting buffer area if the video frame is the key frame, marking the processing result as failure, sending an RTP data request missing in the retransmission key frame to the sending end, and informing the sending end of retransmitting the RTP data missing in the key frame; if the video frame is a non-key frame, discarding the video frame, judging whether the number of discarded non-key frames between two continuous key frames reaches a threshold value, if so, marking the processing result as failure, and sending a key frame reprogramming request to the sending end, otherwise, marking the processing result as success.
Further, the second processing unit includes:
a second receiving means for writing the RTP packet of the received retransmission key frame into a waiting buffer;
the second judgment component is used for framing the RTP packets in the waiting buffer area and judging whether the retransmission key frame of the waiting buffer area is complete or not;
the second decoding component is used for carrying out video frame decoding operation if the retransmission key frame is complete, identifying the processing result as successful if the decoding is successful, and otherwise identifying the processing result as failed, and sending a key frame reprogramming request to the sending end;
and the second processing component is used for further judging whether the waiting time is overtime or not if the retransmission key frame is incomplete, continuing waiting if the waiting time is not overtime, otherwise emptying the received data buffer area and the waiting buffer area, marking the processing result as failure and sending a key frame reprogramming request to the sending end.
Correspondingly, the invention also provides a device for real-time video transmission, wherein the device for real-time video transmission is a sending device or a receiving device,
the transmitting device is used for encoding the video frame data into key frames or non-key frames and transmitting RTP data packets to the receiving device; the sending time of the key frame is determined by the configured interval parameter; the sending equipment is also used for re-encoding the key frame and modifying the sending time of the key frame and then sending the key frame to the receiving equipment, or sending the missing RTP data in the requested retransmission key frame to the receiving equipment;
the receiving device is used for processing the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending device.
Correspondingly, the invention also provides a real-time video transmission system, which comprises a sending end and a receiving end, wherein the sending end is used for encoding the video frame data into key frames or non-key frames and sending RTP data packets to the receiving end; the sending time of the key frame is determined by the configured interval parameter; the sending end is also used for re-encoding the key frame and modifying the sending time of the key frame and then sending the key frame to the receiving end, or sending the missing RTP data in the requested retransmission key frame to the receiving end;
the receiving end is used for processing the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end.
Further, the sending end sets a sending data buffer area for caching the sent RTP data packet, and preferentially searches the sending data buffer area when receiving the RTP data request missing in the retransmission key frame.
Further, the receiving end is provided with a receiving data buffer area and a waiting buffer area, the receiving data buffer area is used for caching the received RTP data packets, and the waiting buffer area is used for caching the RTP data packets received by the retransmission key frame.
In summary, the technical scheme of the invention has the following beneficial effects:
1. the invention provides a dynamic key frame interval, and under the condition of normal network quality, the number of key frames is reduced and the total data amount of transmission is reduced by setting a larger key frame interval; and under the condition that the data processing fails due to the reduction of the network quality, sending an RTP data packet request missing in the re-programming key frame or the retransmission key frame by the receiving end. For the key frame re-encoding request, the sending end dynamically encodes the key frame and modifies the sending opportunity of the key frame according to the request, thereby reducing the waiting time of video recovery of the receiving end and ensuring the timely recovery of video pictures;
2. the sending end sets a data sending buffer area, so that the processing efficiency of searching data when missing RTP data in key frames are retransmitted is improved, and the processing time is shortened. The receiving end is provided with a waiting buffer area and a data receiving buffer area, and can simultaneously and concurrently process RTP data packets, so that the waiting time caused by the missing RTP data in the retransmission key frame is reduced to the minimum;
3. in the case of incomplete video frames, different processing is performed according to the type of the video frame: if the key frame is the key frame, the RTP data missing in the key frame is requested to be retransmitted, so that the integrity of the video picture can be ensured; if the video frame is a non-key frame and the accumulated discarded frame number is within the set threshold range, directly discarding the frame and waiting for the next video frame, so that the network overhead of reprogramming the key frame can be reduced, the fluency of the video is ensured, and the influence on the video picture is only slight mosaic or small delay; if the number of accumulated non-key frame discarding frames between two continuous key frames reaches a threshold value, sending a key frame re-programming request at the moment, and ensuring the definition of a video picture;
4. the receiving end relieves the problem of RTP data packet disorder caused in the network transmission process by adding jitter delay, and simultaneously ensures that the receiving end can not wait for receiving the retransmitted key frame RTP data packet all the time without limit to cause video picture blockage under the condition of network quality reduction by setting a waiting overtime threshold.
Drawings
Fig. 1 is a flowchart illustrating steps of a real-time video transmission method according to an embodiment of the present invention.
Fig. 2 is a flowchart of processing steps performed by a receiving end on an RTP packet according to an embodiment of the present invention.
Fig. 3 is a flow chart of normal data frame processing steps according to an embodiment of the invention.
Fig. 4 is a flowchart of key frame reprocessing steps according to an embodiment of the present invention.
Fig. 5 is a structural frame diagram of a real-time video transmission apparatus according to an embodiment of the invention.
Fig. 6 is a structural frame diagram of a receiving module according to an embodiment of the present invention.
FIG. 7 is a block diagram of a first processing unit according to an embodiment of the present invention.
FIG. 8 is a block diagram of a second exemplary embodiment of a processing unit.
Fig. 9 is a block diagram of a transmitting device for real-time video transmission according to an embodiment of the present invention.
Fig. 10 is a block diagram of another receiving device for real-time video transmission according to an embodiment of the present invention.
Fig. 11 is a structural framework diagram of a real-time video transmission system according to an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a flowchart illustrating steps of a real-time video transmission method according to an embodiment of the present invention, including the following steps:
step 1: the sending end encodes the video frame data into key frames or non-key frames, and sends RTP data packets to the receiving end; the sending time of the key frame is determined by the configured interval parameter;
as is known, video uses the principle of human persistence of vision to make human eyes feel moving by playing a series of pictures, wherein each still picture is called a video frame. In order to facilitate transmission and storage of video files in a network, the video needs to be compressed and encoded, so that the file volume is reduced.
In Video and Audio Compression Coding algorithms such as MPEG-2 (Moving Picture expert group Compression Standard Version 2), h.264/AVC (MPEG-4 advanced Video Coding), Video Codec-1 (Video Codec 1, VC-1), and digital Audio and Video Codec Standard (AVS, Audio Coding Standard), Video pictures are coded into an image sequence consisting of an intra-coded frame (I frame, also called a key frame), a plurality of forward predicted frames (P frames), and a bi-directional interpolated frame (B frame), which are currently common. The I-frame is usually the first frame of each GOP (Group of Pictures), and is moderately compressed and used as a reference point for random access, which can be understood as the complete reservation of the frame, and the decoding can be completed only by the frame data; the P frame represents the difference between the frame and the previous I frame (or P frame), the difference defined by the frame needs to be superposed by the picture cached before in decoding to generate a final picture, and the loss of the previous P frame can influence the decoding display of the next P frame to enable the picture to be mosaic; the B frame records the difference between the current frame and the previous and subsequent frames, in other words, to decode the B frame, not only the previous buffer picture but also the decoded picture are obtained, and the final picture is obtained by superimposing the previous and subsequent pictures on the current frame data.
In the three types of frames, the compression rate of the I frame is the minimum, the compression rate of the P frame is the second, the compression rate of the B frame is the highest, the influence of the I frame on the quality of a video picture is the maximum, and the influence of the P frame is the second, and the influence of the B frame is small. In the technical scheme of the invention, the key frames refer to I frames, the non-key frames comprise P frames and B frames, under different network quality environments, the technical scheme adopts the setting of key frame interval parameters to control the number of the key frames so as to control the data volume of the whole transmission video, the interval parameters can set different values according to the network quality conditions, and larger interval parameters are set as far as possible under the condition that the network quality allows. For example, in an embodiment with a poor network environment, the interval parameter is set to encode and send a key frame data every 5 seconds; in another embodiment where the network environment is better, the interval parameter may be set to 10 seconds.
Step 2: the receiving end processes the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end;
in the process of video transmission, a receiving end is mainly responsible for receiving RTP data packets, framing the RTP data packets, decoding the RTP data packets into static image frames in a corresponding mode according to different types of video frames, and finally displaying the images. During network data transmission, two abnormal situations are usually encountered: firstly, the data packet of the static image frame is lost due to the instability of the network, the receiving end cannot obtain complete image frame data, and at the moment, the receiving end can determine whether to request the transmitting end to retransmit an RTP data packet according to the influence on the image quality; second, the image frame data is complete, but an error occurs in the decoding process, and the receiving end requests the transmitting end to re-encode the key frame. Meanwhile, as the video is composed of a series of ordered image pictures, when the RTP data packet is retransmitted, the subsequent video frames can be framed, but the subsequent video frames can be continuously decoded and displayed only after the retransmitted video frames are completely received, framed, decoded and displayed; when a video frame is requested to be reassembled, the receiving end must wait for the receiving, framing, decoding and displaying of the reassembled data frame, and then sequentially process the subsequent image frames.
And step 3: after receiving, the sending end reassembles the key frame and modifies the sending time of the key frame and then sends the key frame to the receiving end, or retransmits the missing RTP data in the key frame to the receiving end;
although the transmission quantity of video data can be greatly reduced and the fluency of the video can be improved by setting larger key frame interval parameters, the recovery time of a video picture is longer when the decoding of a transmitted data packet fails, in order to solve the problem, the technical scheme of the invention allows a receiving end to transmit a key frame reprogramming request when the decoding of the data packet fails, and a transmitting end immediately codes a image frame required to be reprogrammed into a key frame and then transmits the key frame as a next frame to the receiving end after receiving the key frame reprogramming request, thereby realizing the dynamic adjustment of the transmitting time of the key frame. When the receiving end finds that the data packet is lost, the receiving end sends a request for retransmitting the missing RTP data in the key frame, and after the sending end receives the request, the sending end searches the missing RTP data in the key frame required to be retransmitted and sends the missing RTP data to the receiving end. By combining the two modes of encoding key frames at fixed intervals and encoding key frames according to the dynamic request of a receiving end, the transmission quantity of video data can be reduced and the fluency can be improved on the premise of ensuring the quality of video pictures.
For example, in a specific embodiment, under the condition that the key frame sending frequency is fixed, the key frame interval parameter set by the sending end is 10 seconds, and the key frame is sent every 10 seconds under the condition that the transmission data is normal, preferably, special processing can be performed when the video starts to be transmitted, and the 0 th second, the 2 nd second and the 4 th second of the video data starting to be transmitted at the sending end are encoded into the key frame, so that the problem of data loss at the beginning part caused by inconsistent processing time of the sending end and the receiving end can be solved (because the time interval of the key frame starting to be sent is short, even if the receiving end does not successfully receive the key frame of the previous two frames, the influence on the video reality is not great); meanwhile, if the first 4 seconds have requests for re-encoding key frames, the sending end does not process the key frames in order to avoid frequent sending of the key frames. From 4 seconds later, the sender sends key frames every 10 seconds, i.e., 14 seconds, 24 seconds …, and so on. If a request for re-encoding the key frame by the receiving end is received, the image frame required to be re-encoded is immediately encoded into the key frame and then sent to the receiving end as the next frame, and the next key frame is sent every 10 seconds from the sending of the re-encoded key frame. For example, after the 24 th second encoding key frame of the sending end, the re-encoding key frame request is received in the 28 th second, the image frame requested to be re-encoded is immediately encoded into the key frame and sent to the receiving end; the next key frame transmission is then performed at 38 seconds.
Preferably, the transmitting end in the technical solution of the present invention may further set a transmission data buffer area to buffer the transmitted RTP data packet, and when receiving an RTP data packet request missing from the retransmission key frame, preferentially search the transmission data buffer area. Under normal conditions, the key frames needing to be retransmitted are data which are just sent soon, and by setting a data sending buffer area, the probability of searching data from an original video data file can be reduced, and the searching efficiency is improved. For example, in embodiments where network latency is small, the transmit data buffer size may be set to hold RTP data that has been transmitted for the last 2 seconds; in embodiments where network latency is large, the transmit data buffer size may be set to hold RTP data that has been transmitted for the last 5 seconds.
And repeating the steps 1-3 until all the video frame data are transmitted. The sending end and the receiving end work cooperatively until all image frames contained in the whole video are received, framed, decoded and displayed in sequence.
Fig. 2 is a flowchart of a processing step performed by a receiving end on an RTP packet according to an embodiment of the present invention, which includes the following steps:
step one, a receiving end caches a received RTP data packet to a receiving data buffer area;
the receiving end stores all received data in a received data buffer area, and then performs framing, decoding and display processing on the data, and due to the continuity of video image frames, the next frame of video image can be displayed only after the current video frame is displayed.
In a preferred embodiment, in addition to the data receiving buffer, the receiving end further sets a waiting buffer for buffering received incomplete frame data when the RTP data missing in the key frame is requested to be retransmitted, so that the beneficial effects of the processing are that the data receiving buffer can continue to receive subsequent RTP data packets for framing processing of subsequent video frames, the key frame to be retransmitted can be decoded and displayed immediately after the completion of receiving, framing, decoding and displaying, and the recovery time after the key frame retransmission processing is reduced.
Step two, judging whether the RTP data packet is missing RTP data in the retransmission key frame; the receiving end selects the subsequent processing path according to the type of the received data packet and carries out different processing;
step three, if not retransmitting the missing RTP data in the key frame, processing the normal data frame;
fig. 3 is a flowchart of normal data frame processing steps according to an embodiment of the present invention, which includes the following steps:
framing the RTP packets in the received data buffer area and judging whether the video frames are complete or not;
in the process of video data network transmission, a static video frame data is usually decomposed into a plurality of RTP data packets to be sent, each RTP packet has a network sequence number for identifying the data sequence, and after receiving the RTP data packets, a receiving end needs to combine the plurality of RTP data packets into the original video frame data according to RTP packet header information.
In a preferred embodiment, before framing the RTP data packets, the receiving end reads the RTP data packets from the receiving buffer according to the set jitter delay, and adding the jitter delay can alleviate the problem of the RTP data packets out of order during network transmission.
If the video frame is complete, performing video frame decoding operation, if the decoding is successful, marking the processing result as successful, otherwise, marking the processing result as failed; here, the decoding is performed differently according to the type of the data frame: if the frame is the I frame, the video picture can be restored only by the frame data; if the frame is a P frame, the difference defined by the frame data is needed to be superposed by the picture cached before, and a final picture is generated; in the case of B frames, not only the previous buffered picture but also the decoded picture are obtained, and the final picture is obtained by superimposing the previous and subsequent pictures on the data of the current frame. If the decoding is successful, the video frame is displayed, and if the decoding is failed, a key frame reprogramming request needs to be sent.
And if the video frame is not complete, further judging, if the video frame is a key frame, writing the RTP data packet into a waiting buffer area so as to be framed again after receiving retransmission data subsequently, and marking the processing result as failure. Because the key frame has a large influence on the quality of the picture, under the condition, the receiving end initiates an RTP data request for retransmitting missing data in the key frame;
if the video frame is incomplete and is a non-key frame, because the loss of a small amount of non-key frames has little influence on the quality of the video picture, the technical scheme adopts a mode of directly discarding the video frame, so that the retransmission of data is not needed, and the occupation of network bandwidth is reduced. However, if the number of non-key frames directly discarded between two consecutive key frames reaches a certain number, video picture blocking or mosaic may also be caused, so the technical solution may set an upper threshold value for discarding non-key frames between two consecutive key frames, and if the upper threshold value has been reached, the processing result is marked as failure, and the key frame is requested to be reassembled, otherwise, the processing result is marked as success, and the key frame is not requested to be reassembled. For example, in one embodiment, the upper threshold for allowing discarding of non-key frames between two consecutive key frames is set to 10, the accumulated value of the discarded non-key frames is set to zero every time the receiving end processes a key frame, when the receiving end discards one non-key frame, the accumulated value is added by 1, if the accumulated value is not greater than 10, the subsequent video frames are continuously processed, and if the accumulated value is greater than 10, the sending end is requested to reassemble the key frame, and the accumulated value is reset to zero at the same time.
And step four, if the RTP data missing in the key frame is retransmitted, the key frame is reprocessed.
Fig. 4 is a flowchart of a key frame reprocessing step according to an embodiment of the present invention, including the following steps:
writing the received RTP data packet of the retransmission key frame into a waiting buffer area; since the waiting buffer already buffers the previously received partial data, the retransmitted RTP packet is written into the waiting buffer.
Framing the RTP packets in the waiting buffer area, and judging whether the retransmission key frame data of the waiting buffer area is complete or not; the framing operation herein is consistent with the framing operation in normal data frame processing.
If the retransmission key frame is complete, performing video frame decoding operation, if the decoding is successful, marking the processing result as successful, otherwise, marking the processing result as failed; the decoding processing mode is consistent with the normal frame processing mode, the video frame which is decoded successfully is displayed, and if the decoding fails, a key frame reprogramming request needs to be sent.
If the retransmission key frame is not complete, the judgment is needed according to a set overtime threshold, if the waiting time does not exceed the threshold, the waiting is continued, if the waiting time exceeds the threshold, the receiving data buffer area and the waiting buffer area are emptied at the same time, the processing result is identified as failure, and a key frame reprogramming request is sent.
Fig. 5 is a structural framework diagram of a real-time video transmission apparatus according to an embodiment of the present invention, including:
a first sending module, configured to execute step 1: the sending end encodes the video frame data into key frames or non-key frames, and sends RTP data packets to the receiving end; the sending time of the key frame is determined by the configured interval parameter;
a receiving module, configured to perform step 2: the receiving end processes the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end;
a second sending module, configured to execute step 3: after receiving, the sending end reassembles the key frame and modifies the sending time of the key frame and then sends the key frame to the receiving end, or retransmits the missing RTP data in the key frame to the receiving end;
and the circulating module is used for repeating the steps 1-3 until all the video frame data are transmitted.
Fig. 6 is a structural framework diagram of a receiving module according to an embodiment of the present invention, including:
a receiving unit, configured to receive an RTP packet to a received data buffer;
a waiting buffer unit, which is used for setting a waiting buffer area at a receiving end and buffering the RTP data packet received by the retransmission key frame;
the judging unit is used for judging whether the RTP data packet is missing data in the key frame requested to be retransmitted or not;
the first processing unit is used for processing normal data frames if RTP data missing in the key frame is not retransmitted;
and the second processing unit is used for carrying out key frame reprocessing if the RTP data missing in the key frame is retransmitted.
Fig. 7 is a structural framework diagram of a first processing unit according to an embodiment of the present invention, including:
the first judging component is used for framing the RTP packets in the received data buffer area and judging whether the video frames are complete or not;
the first decoding component is used for carrying out video frame decoding operation if the video frame is complete, identifying the processing result as successful if the decoding is successful, and otherwise identifying the processing result as failed, and sending a key frame reprogramming request to the sending end;
the first processing component is used for further judging whether the video frame is a key frame or not if the video frame is incomplete, writing the received RTP data packet into a waiting buffer area if the video frame is the key frame, marking the processing result as failure, sending an RTP data request missing in the retransmission key frame to the sending end, and informing the sending end of retransmitting the RTP data missing in the key frame; if the video frame is a non-key frame, discarding the video frame, judging whether the number of discarded non-key frames between two continuous key frames reaches a threshold value, if so, marking the processing result as failure, and sending a key frame reprogramming request to the sending end, otherwise, marking the processing result as success.
Fig. 8 is a structural framework diagram of a second processing unit according to an embodiment of the present invention, including:
a second receiving means for writing the RTP packet of the received retransmission key frame into a waiting buffer;
the second judgment component is used for framing the RTP packets in the waiting buffer area and judging whether the retransmission key frame of the waiting buffer area is complete or not;
the second decoding component is used for carrying out video frame decoding operation if the retransmission key frame is complete, identifying the processing result as successful if the decoding is successful, and otherwise identifying the processing result as failed, and sending a key frame reprogramming request to the sending end;
and the second processing component is used for further judging whether the waiting time is overtime or not if the retransmission key frame is incomplete, continuing waiting if the waiting time is not overtime, otherwise emptying the received data buffer area and the waiting buffer area, marking the processing result as failure and sending a key frame reprogramming request to the sending end.
Meanwhile, the invention provides a device for real-time video transmission, which can be a receiving device or a sending device, and specifically can be a mobile phone, a landing call machine, a door phone, an indoor unit and other devices with video transmission functions. The transmitting equipment is used for encoding the video frame data into key frames or non-key frames and transmitting RTP data packets to the receiving equipment; the sending time of the key frame is determined by the configured interval parameter; in addition, the sending device is also used for re-encoding the key frame and modifying the sending time of the key frame and then sending the key frame to the receiving device, or sending the missing RTP data in the requested retransmission key frame to the receiving device;
the receiving device is used for processing the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending device. Fig. 9 is a structural diagram of a sending device for real-time video transmission according to an embodiment of the present invention, and a function of a sending end of real-time video transmission can be realized by including a sending module i, a sending module ii, and a loop module in a real-time video transmission apparatus according to the present invention. The first sending module is used for coding the video frame data into key frames or non-key frames and sending RTP data packets to a receiving end; the sending time of the key frame is determined by the configured interval parameter; a second sending module, configured to send the re-encoded key frame and modify the sending time of the key frame to the receiving end, or retransmit RTP data missing in the key frame to the receiving end; and the circulating module is used for circularly processing the steps of the first sending module and the second sending module until all the video frame data are transmitted.
Fig. 10 is a structural diagram of another receiving device for real-time video transmission according to an embodiment of the present invention, and a receiving module in a real-time video transmission apparatus according to the present invention is included to implement a function of a receiving end of real-time video transmission. The receiving module is used for processing the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end.
In practical application, the function of real-time video transmission is realized through the cooperative work of the sending end equipment and the receiving end equipment. For example, in the process of video transmission of a visitor of the access control system, a sending end is a door phone, and a receiving end is an indoor phone. When the door phone and the indoor phone use the real-time video transmission device of the invention to transmit the real-time video,
the first sending module of the door phone is used for encoding the collected video frame data of the visitor into key frames or non-key frames and splitting the key frames or the non-key frames into RTP data packets to the indoor unit; the sending time of the key frame is determined by the configured interval parameter;
and the receiving module of the indoor unit is used for receiving the RTP data packet sent by the doorway machine, framing the RTP data packet, decoding the RTP data packet into static image frames in a corresponding mode according to different types of video frames, and finally displaying the images on a screen of the indoor unit. In the processing process, if the data of the video frame is incomplete, the indoor unit can determine whether to request the door phone to retransmit the missing data packet according to the influence on the picture quality; if the image frame data is complete but errors occur in the decoding process, the indoor unit will request the gateway unit to reassemble the key frame.
The second sending module of the door phone is used for sending the re-programmed key frames and modifying the sending time of the key frames to the indoor phone after receiving the re-programmed key frame request; or when receiving an RTP data request for retransmitting missing key frames, retransmitting missing RTP data packets to the indoor unit;
and the circulating module of the door phone is used for repeating the steps until all the video frame data are sent to the indoor phone.
For another example, the video data transmission between the camera video acquisition point and the control center in the monitoring system and the video transmission between the server and the mobile phone client in the video-on-demand application can all adopt the device of the invention. In short, the device of the present invention can be applied to more than two devices when real-time video transmission is required.
Fig. 11 is a structural frame diagram of a real-time video transmission system according to an embodiment of the present invention, and includes a sending end and a receiving end, where the sending end and the receiving end are connected via a network, and the sending end is configured to encode video frame data into a key frame or a non-key frame, where the key frame is an I frame, and the non-key frame includes a P frame and a B frame, and split the video frame data into a plurality of RTP packets and send the RTP packets to the receiving end; the sending end determines a key frame according to the configured interval parameters; meanwhile, the sending end is also used for re-encoding the key frame and modifying the sending time of the key frame and then sending the key frame to the receiving end, or retransmitting the RTP data missing in the key frame to the receiving end;
the receiving end is mainly responsible for receiving RTP data packets, framing the RTP data packets, decoding the RTP data packets into static image frames in a corresponding mode according to different types of video frames, and finally displaying the images. Meanwhile, when data packet loss occurs in the sending process of a certain static image frame to cause data incompleteness, a receiving end determines whether to request retransmission of missing RTP data in a key frame or not according to the influence condition on the image quality; or when an error occurs in the decoding process, sending a key frame reprogramming request.
In a preferred embodiment, a sending end of the real-time video transmission system may set a sending data buffer for buffering the sent RTP data packet, and when receiving an RTP data request missing in a retransmission key frame, preferentially search the sending data buffer, thereby increasing the search speed and shortening the processing time.
In another preferred embodiment, a receiving end of the real-time video transmission system may be provided with a receiving data buffer and a waiting buffer, the receiving data buffer is configured to buffer the received RTP data packet, the waiting buffer is configured to buffer the RTP data packet already received by the retransmission key frame, the waiting buffer stores incomplete frame data through a double-buffer mechanism, the receiving data buffer may continue to receive subsequent RTP data packets, perform framing processing on subsequent video frames, and immediately decode and display the subsequent video frames after the retransmission key frame is completely received, framed, decoded and displayed, thereby reducing recovery time after the retransmission key frame processing.
Claims (11)
1. A real-time video transmission method, comprising the steps of:
step 1: the sending end encodes the video frame data into key frames or non-key frames, and sends RTP data packets to the receiving end; the sending time of the key frame is determined by the configured interval parameter;
step 2: the receiving end processes the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end; the receiving end processes the RTP data packet, and specifically comprises the following steps:
receiving RTP data packets to a received data buffer;
judging whether the RTP data packet is missing data in the key frame requested to be retransmitted or not;
if the RTP data which is not missing in the retransmission key frame is not the RTP data, normal data frame processing is carried out; the normal data frame processing comprises the following steps: framing the RTP packets in the received data buffer area and judging whether the video frames are complete or not; if the video frame is complete, performing video frame decoding operation, if the decoding is successful, marking the processing result as successful, otherwise, marking the processing result as failed, and sending a re-programming key frame request to the sending end; if the video frame is not complete, further judging whether the video frame is a key frame, if the video frame is the key frame, writing the received RTP data packet into a waiting buffer area, marking the processing result as failure, sending an RTP data request missing in the retransmission key frame to a sending end, and informing the sending end of retransmitting the RTP data missing in the key frame; if the video frame is a non-key frame, discarding the video frame, judging whether the number of discarded non-key frames between two continuous key frames reaches a threshold value, if so, marking the processing result as failure, and sending a re-programming key frame request to the sending end, otherwise, marking the processing result as success;
if the RTP data missing in the key frame is retransmitted, carrying out key frame reprocessing;
and step 3: after receiving, the sending end reassembles the key frame and modifies the sending time of the key frame and then sends the key frame to the receiving end, or retransmits the missing RTP data in the key frame to the receiving end;
and repeating the steps 1-3 until all the video frame data are transmitted.
2. The real-time video transmission method according to claim 1, wherein the step 1 further comprises:
the sending end puts the sent RTP data packet into a sending data buffer area, and when receiving the missing RTP data request in the retransmission key frame, the sending data buffer area is searched preferentially.
3. The real-time video transmission method according to claim 1, wherein the step 2 further comprises: the receiving end is provided with a waiting buffer area for buffering the RTP data packet received by the retransmission key frame.
4. The real-time video transmission method according to claim 1, wherein the receiving end reads the RTP packet from the receiving buffer according to the set jitter delay.
5. The real-time video transmission method according to claim 3, wherein said performing key frame reprocessing comprises the steps of:
writing the RTP data packet of the received retransmission key frame into a waiting buffer area;
framing the RTP packets in the waiting buffer area, and judging whether the retransmission key frame of the waiting buffer area is complete or not;
if the retransmission key frame is complete, performing video frame decoding operation, if the decoding is successful, marking the processing result as successful, otherwise, marking the processing result as failed, and sending a key frame reprogramming request to the sending end;
if the retransmission key frame is not complete, further judging whether the waiting time is overtime or not, if the waiting time is not overtime, continuing waiting, otherwise, emptying the receiving data buffer area and the waiting buffer area, marking the processing result as failure, and sending a key frame reprogramming request to the sending end.
6. A real-time video transmission apparatus, comprising:
the first sending module is used for coding the video frame data into key frames or non-key frames and sending RTP data packets to the receiving module; the sending time of the key frame is determined by the configured interval parameter;
the receiving module is used for processing the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending module II; the receiving module comprises:
a receiving unit, configured to receive an RTP packet to a received data buffer;
the judging unit is used for judging whether the RTP data packet is missing data in the key frame requested to be retransmitted or not;
the first processing unit is used for processing the normal data frame of the RTP data missing in the non-retransmission key frame; the first processing unit includes:
the first judging component is used for framing the RTP packets in the received data buffer area and judging whether the video frames are complete or not;
the first decoding component is used for carrying out video frame decoding operation when the video frame is complete, identifying the processing result as successful if the decoding is successful, and otherwise identifying the processing result as failed, and sending a key frame reprogramming request to the second sending module;
the first processing component is used for further judging whether the video frame is a key frame or not when the video frame is incomplete, writing the received RTP data packet into a waiting buffer area if the video frame is the key frame, marking the processing result as failure, sending a request for retransmitting missing RTP data in the key frame to the second sending module, and informing the second sending module of retransmitting the missing RTP data in the key frame; if the video frame is a non-key frame, discarding the video frame, judging whether the number of discarded non-key frames between two continuous key frames reaches a threshold value, if so, marking the processing result as failure, and sending a re-programming key frame request to the second sending module, otherwise, marking the processing result as success;
the second processing unit is used for carrying out key frame reprocessing on the RTP data missing in the retransmission key frame;
the second sending module is configured to send the reconstructed key frame and modify the sending timing of the key frame to the receiving module after receiving the reconstructed key frame request sent by the first decoding component or the first processing component, or retransmit RTP data missing in the key frame to the receiving module after receiving a RTP data request missing in a retransmission key frame sent by the first processing component;
and the circulating module is used for circularly calling the first sending module, the second receiving module and the second sending module to process until all video frame data are transmitted.
7. The real-time video transmission apparatus according to claim 6, wherein the receiving module further comprises:
and the waiting buffer unit is used for setting a waiting buffer area and buffering the RTP data packet received by the retransmission key frame.
8. The real-time video transmission apparatus according to claim 7, wherein the second processing unit comprises:
a second receiving means for writing the RTP packet of the received retransmission key frame into a waiting buffer;
the second judgment component is used for framing the RTP packets in the waiting buffer area and judging whether the retransmission key frame of the waiting buffer area is complete or not;
the second decoding component is used for carrying out video frame decoding operation when the retransmission key frame is complete, identifying the processing result as successful if the decoding is successful, and otherwise identifying the processing result as failed, and sending a key frame reprogramming request to the second sending module;
and the second processing component is used for further judging whether the waiting time is overtime or not when the retransmission key frame is incomplete, continuing waiting if the waiting time is not overtime, otherwise emptying the data receiving buffer area and the waiting buffer area, marking the processing result as failure, and sending a key frame reprogramming request to the second sending module.
9. A real-time video transmission system is characterized by comprising a sending end and a receiving end, wherein the sending end is used for encoding video frame data into key frames or non-key frames and sending RTP data packets to the receiving end; the sending time of the key frame is determined by the configured interval parameter; the sending end is also used for re-encoding the key frame and modifying the sending time of the key frame and then sending the key frame to the receiving end, or sending the missing RTP data in the requested retransmission key frame to the receiving end;
the receiving end is used for processing the RTP data packet, if the processing is successful, the video frame is displayed, and if the processing is failed, a request for reprogramming the key frame or retransmitting missing RTP data in the key frame is sent to the sending end; the receiving device processes the RTP data packet, specifically:
receiving RTP data packets to a received data buffer;
judging whether the RTP data packet is missing data in the key frame requested to be retransmitted or not;
if the RTP data which is not missing in the retransmission key frame is not the RTP data, normal data frame processing is carried out; the normal data frame processing comprises the following steps: framing the RTP packets in the received data buffer area and judging whether the video frames are complete or not; if the video frame is complete, performing video frame decoding operation, if the decoding is successful, marking the processing result as successful, otherwise, marking the processing result as failed, and sending a re-programming key frame request to the sending end; if the video frame is not complete, further judging whether the video frame is a key frame, if the video frame is the key frame, writing the received RTP data packet into a waiting buffer area, marking the processing result as failure, sending an RTP data request missing in the retransmission key frame to a sending end, and informing the sending end of retransmitting the RTP data missing in the key frame; if the video frame is a non-key frame, discarding the video frame, judging whether the number of discarded non-key frames between two continuous key frames reaches a threshold value, if so, marking the processing result as failure, and sending a re-programming key frame request to the sending end, otherwise, marking the processing result as success;
and if the RTP data is missing in the retransmission key frame, performing key frame reprocessing.
10. The real-time video transmission system according to claim 9, wherein the sending end sets a sending data buffer for buffering the sent RTP packets, and preferentially searches the sending data buffer when receiving the RTP data request missing in the retransmission key frame.
11. The real-time video transmission system according to claim 9, wherein the receiving end is configured with a receiving data buffer and a waiting buffer, the receiving data buffer is configured to buffer the received RTP packets, and the waiting buffer is configured to buffer the received RTP packets of the retransmission key frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610167888.7A CN107231328B (en) | 2016-03-23 | 2016-03-23 | Real-time video transmission method, device, equipment and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610167888.7A CN107231328B (en) | 2016-03-23 | 2016-03-23 | Real-time video transmission method, device, equipment and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107231328A CN107231328A (en) | 2017-10-03 |
| CN107231328B true CN107231328B (en) | 2020-08-28 |
Family
ID=59931630
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610167888.7A Active CN107231328B (en) | 2016-03-23 | 2016-03-23 | Real-time video transmission method, device, equipment and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107231328B (en) |
Families Citing this family (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107948142A (en) * | 2017-11-15 | 2018-04-20 | 广州杏雨信息科技有限公司 | A kind of point-to-point live transmission method of 3D HD videos and system |
| CN107846475B (en) * | 2017-12-06 | 2020-08-18 | 中国水利水电科学研究院 | Intelligent water conservancy information measurement and control device |
| CN108366306A (en) * | 2017-12-13 | 2018-08-03 | 天津天地伟业机器人技术有限公司 | A kind of control method of the bit stream based on embedded device |
| CN111326176A (en) * | 2018-12-14 | 2020-06-23 | 中移（杭州）信息技术有限公司 | Detection method, device and medium of RTP packet based on OPUS coding |
| CN109361935A (en) * | 2018-12-24 | 2019-02-19 | 广州微算互联信息技术有限公司 | Picture stream video transmission method and system |
| CN109862400B (en) * | 2019-02-18 | 2021-08-31 | 苏州长风航空电子有限公司 | Streaming media transmission method, device and system |
| CN110062003B (en) * | 2019-04-30 | 2022-01-25 | 北京金山云网络技术有限公司 | Video data transmitting method, video data transmitting device, electronic equipment and storage medium |
| CN110225347A (en) * | 2019-06-24 | 2019-09-10 | 北京大米科技有限公司 | Method of transmitting video data, device, electronic equipment and storage medium |
| CN110225348A (en) * | 2019-06-24 | 2019-09-10 | 北京大米科技有限公司 | Restorative procedure, device, electronic equipment and the storage medium of video data |
| CN112449190A (en) * | 2019-09-05 | 2021-03-05 | 曙光网络科技有限公司 | Decoding method of concurrent video session IPB frame image group |
| CN110557677A (en) * | 2019-09-27 | 2019-12-10 | 北京西山居互动娱乐科技有限公司 | Video transmission method and device |
| CN110769380A (en) * | 2019-10-31 | 2020-02-07 | 联想(北京)有限公司 | Video distribution method and device |
| CN111163362B (en) * | 2019-12-30 | 2021-12-24 | 北京佳讯飞鸿电气股份有限公司 | Video receiving method and system capable of self-adapting retransmission waiting time |
| CN113452953B (en) * | 2020-03-26 | 2022-06-14 | 浙江宇视科技有限公司 | Video stream transmission control method, device, equipment and medium |
| CN111953612B (en) * | 2020-07-21 | 2024-02-23 | 西安万像电子科技有限公司 | Data transmission control method and device |
| CN112291523B (en) * | 2020-10-29 | 2023-12-05 | 合肥安迅精密技术有限公司 | Image data receiving system and method of chip mounter equipment |
| CN112689160B (en) * | 2020-11-27 | 2022-12-09 | 烟台艾睿光电科技有限公司 | Video transmission method and device applied to image acquisition equipment |
| CN113612962A (en) * | 2021-07-15 | 2021-11-05 | 深圳市捷视飞通科技股份有限公司 | Video conference processing method, system and device |
| CN115102927B (en) * | 2022-04-29 | 2023-10-27 | 厦门立林科技有限公司 | SIP intercom method, system and storage device for keeping video clear |
| WO2023206910A1 (en) * | 2022-04-29 | 2023-11-02 | 厦门立林科技有限公司 | Sip intercom method and system based on local area network and wide area network, and storage medium |
| CN115208864B (en) * | 2022-08-12 | 2024-02-06 | 阿波罗智联(北京)科技有限公司 | Data transmission method, device, equipment, vehicle and storage medium |
| CN116112697B (en) * | 2022-11-28 | 2023-08-11 | 长沙千视电子科技有限公司 | NDI-based real-time video recording method |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7174561B2 (en) * | 2001-04-13 | 2007-02-06 | Emc Corporation | MPEG dual-channel decoder data and control protocols for real-time video streaming |
| CN100588249C (en) * | 2006-07-27 | 2010-02-03 | 腾讯科技（深圳）有限公司 | Method, system and terminal for adjusting video quality |
| US20090303309A1 (en) * | 2008-06-04 | 2009-12-10 | Pantech Co., Ltd. | Mobile terminal and method for transmitting video data in video telephony system |
| CN101998101A (en) * | 2009-08-31 | 2011-03-30 | 中兴通讯股份有限公司 | Video data receiving and transmitting systems and video data processing method for video telephone |
| CN101909210A (en) * | 2009-12-17 | 2010-12-08 | 新奥特（北京）视频技术有限公司 | Network streaming media server and low-bandwidth high-quality solution thereof |
| CN101860733A (en) * | 2010-06-11 | 2010-10-13 | 深圳市黄河数字技术有限公司 | 3g network video monitoring system and monitoring method |
| CN101990087A (en) * | 2010-09-28 | 2011-03-23 | 深圳中兴力维技术有限公司 | Wireless video monitoring system and method for dynamically regulating code stream according to network state |
| CN103533387B (en) * | 2013-10-21 | 2016-08-17 | 腾讯科技（深圳）有限公司 | A kind of live video control, equipment and system |
| CN104869461A (en) * | 2015-05-22 | 2015-08-26 | 南京创维信息技术研究院有限公司 | Video data processing system and method |
-
2016
- 2016-03-23 CN CN201610167888.7A patent/CN107231328B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107231328A (en) | 2017-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107231328B (en) | Real-time video transmission method, device, equipment and system | |
| CN109729439B (en) | Real-time video transmission method | |
| US8711929B2 (en) | Network-based dynamic encoding | |
| US8472520B2 (en) | Systems and methods for transmitting and receiving data streams with feedback information over a lossy network | |
| US10652580B2 (en) | Video data processing method and apparatus | |
| EP2061174B1 (en) | Data communication system, data transmitting device and method, using probe packets and having a transmission buffer control | |
| KR101242663B1 (en) | Packet transmission apparatus, communication system and computer-readable recording medium | |
| US10944973B2 (en) | Estimation of video quality of experience on media servers | |
| US20110085602A1 (en) | Video Communication System, Device and Method Based on Feedback Reference Frames | |
| JP2001274861A (en) | Method and device for data transmission | |
| US20150103885A1 (en) | Real time ip video transmission with high resilience to network errors | |
| KR20040009928A (en) | Method of generating transmission control parameter and selective retranmission method according to the packet characteristics. | |
| JP2007150916A (en) | Communication system, terminal device and computer program | |
| CN110392284B (en) | Video encoding method, video data processing method, video encoding apparatus, video data processing apparatus, computer device, and storage medium | |
| CN111093083B (en) | Data transmission method and device | |
| JPH10126772A (en) | Dynamic image data transfer system | |
| JP2005033556A (en) | Data transmitter, data transmitting method, data receiver, data receiving method | |
| CN112995214B (en) | Real-time video transmission system, method and computer readable storage medium | |
| CN101645903A (en) | Method and device for transmitting multimedia data | |
| EP2908516A1 (en) | Process for transmitting an ongoing video stream from a publisher to a receiver through a MCU unit during a live session | |
| JP2006279436A (en) | Multimedia communication system and data deleting method for re-transmission | |
| CN113542685B (en) | Real-time ultra-high definition video transmission method based on reliable UDP | |
| JPH10200897A (en) | Moving picture transmission device | |
| Ganguly et al. | Synergized QoE-Centric Streaming for Telerobotics | |
| CN114866523A (en) | UDP-based video rapid transmission method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200722 Address after: Cangshan District of Fuzhou City, Fujian province 350000 Jinshan Road No. 618 juyuanzhou Ruijie Science Park building 19-22 Applicant after: FUJIAN STAR-NET COMMUNICATION Co.,Ltd. Address before: Cangshan District of Fuzhou City, Fujian province 350000 Jinshan Road No. 618 juyuanzhou Ruijie Science Park building 19-22 Applicant before: FUJIAN STAR-NET COMMUNICATION Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |